Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
From: Bruce Lilly <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: [email-masked]
Date: Tue, 31 Aug 2004 19:41:40 -0400
List-Id: <ietf-822.imc.org>

Keith Moore wrote:

> I
> suppose you could argue for a UA that in the presence of list-post, sent
> replies to that address instead - but I think that would be very
> counterintuitive.

I know of at least two MUAs that provide three reply options:
    Kmail 1.7          Ximian Evolution 1.4.6
---------------------------------------------
Reply to Author           Reply to Sender
Reply to All              Reply to All
Reply to Mailing List     Reply to List

In each case IMAP folders can be configured as associated with
a mailing list mailto URI which is used for the Reply to
[Mailing] List function.  No List-Post field required.

Both are of course limited somewhat in terms of platform support.

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: the most obvious failure in To-NoReply
Date: Tue, 31 Aug 2004 15:51:45 -0400
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

>>> I don't want _replies_ to go to [email-masked]. I want _followups_
>>> to go to [email-masked].
>
>> "followup" is a Usenet term.  Usenet has two different kinds of
>> addresses - email addresses and newsgroups.
>
> But mail also has two different kinds of addresses or, to put it 
> another
> way, mailing lists certainly need two kinds of addresses:
>
> 1. From: To: Cc: Bcc: etc.
> 2. List-Post: Mail-Followup-To: (which not all lists, including this 
> one,
> provide :-( )

Having lists mung mail-followup-to is every bit as bad as having lists 
mung reply-to.  actually it's worse, since one of the arguments in 
favor of mail-followup-to is that it hasn't been corrupted like 
reply-to has.

> MUAs then need some form of command for "Reply-To-List", and the 
> obvious
> name for that command is "Followup" (or the single letter 'F').

No, it's not obvious.  Even though you can gateway mail into news and 
vice versa, in practice the differences between the two environments 
affect how people interact with those environments.  A followup is not 
the same thing as reply-to-list (using list-post) or even reply-to-all. 
  If email had a way to distinguish list addresses from personal 
addresses, something analogous to followup might be "reply to just the 
lists".  but even then it would still be different.

In Usenet there is an implicit assumption that everyone participating 
in a conversation has access to at least one of the newsgroups 
involved.  That, and participants can be subscribed to a usenet 
newsgroup without filling up their inboxes with cruft.

in email it is much less reasonable to assume that everyone in a 
conversation can access any mailing list that is copied on the 
conversation - not all mailing lists are public, and people are less 
likely to be willing to subscribe to an email list than to a usenet 
newsgroup.

Also, usenet articles automagically expire after a time, whereas emails 
typically have to be deleted explicitly, and this also affects how 
people interact with each medium.

> And responses to mailing lists should be followups too.
>
>>  Email
>> does not have visibly different kinds of addresses and does make a
>> distinction between replies and followups.
>
> And that is exactly the root of the problem, which needs to be fixed.

somehow I don't view the fact that email isn't like usenet as "a 
problem".  I'd call it "a feature".  email lists certainly are not a 
perfect discussion medium, but usenet is even less so.  the truth is 
that people find both kinds of discussions useful, and they also find 
it useful that they operate differently than one another.  trying to 
make them the same is probably a disservice to both.

>
>>  Email UAs have "reply" and
>> "reply to all" or similar.  For several reasons including both 
>> protocol
>> differences and cultural differences between email and Usenet, email's
>> "reply to all" and Usenet's "followup" are not quite the same thing.
>> And I don't see evidence of any desire among users to make email more
>> like Usenet.  (actually I see plenty of evidence to the contrary.  
>> but I
>> digress.)
>
> On the contrary, in the case of mailing lists I see evidence of this
> desire all the time. It is just not clear whether the "proper" way to
> respond to a mailing list is "Reply" or "Reply-To-All", since both of 
> them
> usually do the Wrong Thing

you and the author of the subject message may not agree on the Right 
Thing.

> Personally, I find it extremely annoying to receive personal replies to
> messages I post to a mailing list to which I am already subscribed.

Personally, I find it extremely valuable to receive personal replies to 
messages I post to a mailing list to which I am already subscribed.   I 
find it annoying when recipients remove me from the recipient list of 
replies.  Personal replies are handled with a different priority than 
ordinary list mail.  I like having personal replies end up in my inbox, 
and also having a complete set of messages to the list end up in my 
folder for that list.  I also like being able to continue conversations 
between list participants quickly even when the list is slow to forward 
messages.

Keith

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Tue, 31 Aug 2004 15:32:54 -0400
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

>>>> (a) Reply-To with some magic address that is valid syntax but a
>>>> no-op. maybe this:
>>>
>>>> Reply-To: nobody :;
>>>
>>> But in the case of mailing lists, which is where this requirement 
>>> most
>>> often arises, that might also prevent replies being sent to the
>>> mailing list (except in MUAs that had an explicit command for
>>> "Reply-To-List").
>
>> not if the list were on the To or CC line and the recipient did a 
>> "reply
>> to all".  (I'm assuming that reply-to: nobody:; would just remove the
>> addresses obtained from the To and CC fields of the subject message).
>
> Yes, but Reply-To-All is not the way I like to reply to mailing lists
> (though it is often the only one that works).

sorry, I think you're stuck with that.  there are way too many problems 
with lists that mung reply-to.  munging from would be even worse.  I 
suppose you could argue for a UA that in the presence of list-post, 
sent replies to that address instead - but I think that would be very 
counterintuitive.

Keith

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Tue, 31 Aug 2004 15:30:03 -0400
To: Simon Josefsson <[email-masked]>
Sender: [email-masked]

>> if you think about it, anytime someone wants to reply to the _author_,
>> they (or their MUA) should use the From field.  it's not reasonable to
>> assume that the Reply-To field points to the author.  it is reasonable
>> to expect authors to set the From address to a valid address that will
>> reach them.
>
> This might be, but at least my experience with deployed MUAs suggest
> this isn't what is implemented today.  MUAs prefer the Reply-To over
> From on "reply author".  If we want to introduce something new, I
> believe it should be designed considering how the majority of MUAs
> behave today, even though it may be argued that they are all broken.

there is no solution to these problems that doesn't involve some 
changes in MUA behavior.  the question is:  which changes give the most 
benefit for the least transition pain?

it might be that something similar to Usenet's mail-copies-to is less 
painful than either reply-to: nobody:; or mail-{followup,reply}-to.  
I'm not sure yet.  I'd like to look at the proposals side-by-side 
first.

Keith

Date: Mon, 30 Aug 2004 21:29:28 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: reply problem list
Sender: [email-masked]

Keith Moore wrote:
> Okay, I dug out the "reply problem list" I had written as part of the
> DRUMS discussions, and edited it slightly.  
> 
> I've tried to define these problems in terms of observed behavior[...]
> 
> Many of these problems are very similar to one another[...]
> I have listed them as separate problems because some proposed
> solutions will solve one problem while failing to solve a similar one.
> By listing so many separate problems, I don't mean to imply that they're
> all of equal severity. 
> 
> I separated the original list into two sections.  List A consists of the
> problems that aren't related to lists munging Reply-To.[...]
> 
> My reason for compiling this is to help clarify arguments of the form
> "proposal X doesn't solve problem Y".   It appears that any solution to
> the _set_ of problems is likely to involve several changes: perhaps a
> new header field or two, along with (perhaps) recommendations to MUA
> authors about their user interfaces, modifications to various kinds of
> software, and maybe some user education.  With this list we can compare
> various _sets_ of solutions to this _set_ of problems.

My comments below are from the perspective of what is available to
message authors and respondents using only standard (RFC 822/2822)
message header fields, with widely-available MUAs. Consequently,
to the extent that the comments represent any sort of a solution
at all, that solution falls primarily into the "some user education"
category, with a smidgen of "recommendations to MUA authors". The
comments specifically do not advocate adding new header fields or
modifications to any specific software or class of software.

As noted, the basis of the discussion of this list is that authors'
header fields (especially Reply-To) are not modified by mailing list
software.

> List A: REPLY problems not involving modification of message headers by
> mailing lists:
> 
> A1.	Given the vast number of MUAs that implement a "reply all" command
> where the Reply-To address only replaces the From address from the
> original message, there's no good way for the author of a message to say
> "please reply only to these addresses", or (almost equivalently) "please
> don't reply to these addresses". The author can use Bcc for all
> recipients who should not receive the reply, but that doesn't inform the
> "non blind" recipients that the other recipients were also sent copies
> of the message.

If the author sets Reply-To to the list of desired response recipient
addresses, leaving out the To and Cc fields (or setting them to empty
groups) and specifying original message recipients via Bcc (which is my
understanding of the first sentence plus the first clause of the second
sentence), then there aren't any "'non-blind' recipients". The Comments
field could be used to indicate the Bcc recipients, who would not receive
responses (without explicit action by a sender). For that matter, a note
could be placed in the message body (presumably for the benefit of users
afflicted with user-hostile MUAs that make it difficult to see message
headers).  As for the authors of such user-hostile MUAs, I would recommend
a one-way trip to beautiful Guantanamo Bay (advice to such authors would
likely fall on deaf ears).

> A2.	If a message contains a Reply-To field, but the person composing the
> reply isn't aware that Reply-To is being used, the reply may go to
> somewhere besides where the reply author thought it should go. The
> result may be surprising or embarrassing (as when a personal reply gets
> sent to a list), or the reply author may think he has sent the message
> to a list when it has really only gone to one particular recipient.

Caveat sender.  Most MUAs display the recipients. I can't say that I
know of any that do not.  If there are any, then caveat emptor.

> A3.	A few MUAs implement only one kind of reply -- which uses Reply-To
> if present and From otherwise. Users of such MUAs have difficulty
> participating in group discussions on mailing lists (unless the list
> mungs Reply-To).

Caveat emptor.  There are plenty of MUAs that do not have that
limitation, many of them available at no cost and for a variety of
platforms (e.g. Mozilla, pine).  Senders could manually ("manually"
including mechanisms as simple as cut-and-paste) include other
recipients on responses -- unless the MUA is so featureless as to
prevent even that.  Such an MUA is likely to be a problem under any
"set of solutions".

> A4.	If a message is sent to a mailing list, and someone does a "reply
> all" to that message, the author often gets two copies - one sent to the
> From address and another sent to the list address (which presumably
> appeared in the To or Cc field of the subject message).

If the author does not wish to receive a separate (duplicate)
response, he can set Reply-To to the mailing list address (as
is the case with this message).

> A5.	If a user is subscribed to multiple mailing lists, and a message is
> sent to more than one of those mailing lists, the user will get multiple
> copies of the message. This might not be so bad, but the same user will
> also get duplicate copies of the message every time someone does "reply
> to all" to the original messages or any of the replies to that message.

Again, the author can set Reply-To to one or more of the mailing list
addresses.  There may still be multiple copies sent to recipient who
subscribe to multiple lists, but as you say, that's not so bad.

> A6.	On many MUAs, if the sender specified Reply-To, the recipient is
> stuck with that - he cannot easily edit it to reply to From or
> From+To+CC (especially if he cannot see the original headers while he's
> editing the reply recipient list). The author of the reply thus
> sometimes has difficulty sending the reply to where it needs to go.

"Stuck" would be appropriate if the MUA provides no ability to edit
the recipient list; I would be amazed if that's the case with any
MUA that is actually in any appreciable use.  That there may be
some minor difficulty in directing a response to a place other than
that indicated by the original author via Reply-To is hardly
surprising -- that is as it should be.

> A7.	If someone responds to a list message and CCs themselves in the
> response, everyone who does a "reply all" to that message will also Cc
> that person...even if he's on one of the lists. Alternatively, if
> multiple recipients (often lists) get CCed on a message, every "reply
> all" goes to all of those recipients, even if the discussion topic
> wanders and it's no longer appropriate for many of those recipients.

Self-copies can be sent via Bcc or saved as a local file copy to
avoid the first situation. Continuing copies via reply-to-all is
an inherent characteristic of Internet mail -- there is no easy
and reliable way for a message recipient to "opt-out" of receiving
some message(s); if there were, spam wouldn't be a problem.  There
have been proposals that might offer a communications system that
does not have that characteristic, but they either introduce serious
problems in their own right or are a fundamentally different type of
system.  In the specific case of multiple lists, see discussion of
A10 below; a responsible author should be able to resolve the
situation.

> A8.	Many people have multiple mailboxes, or regularly send mail from
> some account at which they don't want to receive mail. If they use
> Reply-To to point to their preferred mailbox, it will prevent some
> recipients from doing "reply to all". Also, since there's no indication
> as to why the author used Reply-To (it could be "please don't mail to me
> at my From address" or it could be "please tell my secretary, not me,
> whether you'll be at the meeting"), it's easy for a reply to go to the
> wrong place.

"Prevent" is too strong; there may be some inconvenience.  Many MUAs
provide the ability to set the From field appropriately, in which case
there is no need to use Reply-To for the stated purpose.

> A9.	Mailing list subscribers who don't understand the reply functions of
> their UA or who have UAs with severely limited reply functions often
> send replies to the author of the message when they should go to the
> list. This causes discussions on the list to suffer and perhaps die out.

Again, setting Reply-To to point to the list will result in replies
going to the list, whether individual respondents use reply or
reply-to-all.  If the original message author wishes responses to go
to the list, he should set Reply-To to the list address.

> A10.	Mailing list subscribers who don't understand the reply functions
> of their UA, often send replies to the mailing list when they should go
> to the author of the message. This causes prolonged discussions on a
> list on which such discussions are inappropriate. 

If the original author detects that condition and is responsible, he
will most likely send a message requesting that the discussion be
continued off-list (and he will probably set Reply-To appropriately;
moreover if he wishes to copy the list on that message, he will
probably use Bcc rather than To or Cc so as to prevent reply-to-all
responses from going to the list).

Summary, including some limited discussion of recent proposals:
A1 could possibly be helped by addition of a Cc-noreply (or similar)
field to indicate recipients who should not receive responses. But
that won't work for the user-hostile MUAs, which brings us back to
a note in the message body (to the extent that authors wish to
cater to users of user-hostile MUAs...).  A2 seems to be solvable
by user education, along with advice to MUA authors if there are
any that seem to have missed the boat on display of recipients.
Ditto for A3.  A4, A5, and A9 seem to be easily handled by appropriate
use of Reply-To -- again probably primarily a user-education issue.
A6 seems to be a description of intended operation rather than a
problem.  Part of A7 can be handled via Bcc or local file copies;
the remaining issue is unlikely to be solved without a fundamental
mail system redesign or without introducing a "cure" that is worse
than the symptoms.  A8 seems to be solvable via advice to MUA
authors.  A10 is another user-education issue.

It appears that user-education and MUA author education may be
warranted, and would solve nearly all of these problems without any
changes to message format or the mail system.  The one difficult
issue is the "opt-out" part of A7 in the absence of a responsible
party who is willing and able to redirect future responses.

To: [email-masked]
Xref: clerew local.mime:3467
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Tue, 31 Aug 2004 09:50:38 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> >(a) Reply-To with some magic address that is valid syntax but a
>> >no-op. maybe this:
>> 
>> >Reply-To: nobody :;
>> 
>> But in the case of mailing lists, which is where this requirement most
>> often arises, that might also prevent replies being sent to the
>> mailing list (except in MUAs that had an explicit command for
>> "Reply-To-List").

>not if the list were on the To or CC line and the recipient did a "reply
>to all".  (I'm assuming that reply-to: nobody:; would just remove the
>addresses obtained from the To and CC fields of the subject message).

Yes, but Reply-To-All is not the way I like to reply to mailing lists
(though it is often the only one that works).
-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3466
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: the most obvious failure in To-NoReply
Date: Tue, 31 Aug 2004 09:32:52 GMT
Lines: 61
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>D. J. Bernstein wrote:
>> 
>> I don't want _replies_ to go to [email-masked]. I want _followups_
>> to go to [email-masked].

>"followup" is a Usenet term.  Usenet has two different kinds of 
>addresses - email addresses and newsgroups.

But mail also has two different kinds of addresses or, to put it another
way, mailing lists certainly need two kinds of addresses:

1. From: To: Cc: Bcc: etc.
2. List-Post: Mail-Followup-To: (which not all lists, including this one,
provide :-( )

MUAs then need some form of command for "Reply-To-List", and the obvious
name for that command is "Followup" (or the single letter 'F'). It was
clearly such a facility that Dan was referring to.

>  Responses to the author's 
>email address are replies; responses to newsgroups are followups.

And responses to mailing lists should be followups too.

>  Email 
>does not have visibly different kinds of addresses and does make a 
>distinction between replies and followups.

And that is exactly the root of the problem, which needs to be fixed.

>  Email UAs have "reply" and 
>"reply to all" or similar.  For several reasons including both protocol 
>differences and cultural differences between email and Usenet, email's 
>"reply to all" and Usenet's "followup" are not quite the same thing. 
>And I don't see evidence of any desire among users to make email more 
>like Usenet.  (actually I see plenty of evidence to the contrary.  but I 
>digress.)

On the contrary, in the case of mailing lists I see evidence of this
desire all the time. It is just not clear whether the "proper" way to
respond to a mailing list is "Reply" or "Reply-To-All", since both of them
usually do the Wrong Thing (my usual practice is to do Reply-To-All and
then weed out all the addresses I d not want). The alternative is for list
expanders to force Reply-To: list-address, which I much prefer, but which
usually leads to flame wars when proposed on a list.

Personally, I find it extremely annoying to receive personal replies to
messages I post to a mailing list to which I am already subscribed.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Tue, 31 Aug 2004 10:55:00 +0200
Lines: 37
Cancel-Lock: sha1:Evmrs8nZJvNP7qUzDVOx8qPoHPg=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>>>>>(a) Reply-To with some magic address that is valid syntax but a
>>>>>no-op. maybe this:
>>>>
>>>>>Reply-To: nobody :;
>>>>
>>>>But in the case of mailing lists, which is where this requirement most
>>>>often arises, that might also prevent replies being sent to the
>>>>mailing list (except in MUAs that had an explicit command for
>>>>"Reply-To-List").
>>>
>>>not if the list were on the To or CC line and the recipient did a "reply
>>>to all".  (I'm assuming that reply-to: nobody:; would just remove the
>>>From addresses from the reply recipient list, and have no effect on the
>>>addresses obtained from the To and CC fields of the subject message).
>> 
>
>> The problem is when people want to reply to the author, and press the
>> "reply author" button.  That operation would take the content of From:
>> as the recipient, but Reply-To would override that value with
>> nobody:;, and the reply would go to the bit bucket.
>
> if you think about it, anytime someone wants to reply to the _author_,
> they (or their MUA) should use the From field.  it's not reasonable to
> assume that the Reply-To field points to the author.  it is reasonable
> to expect authors to set the From address to a valid address that will 
> reach them.

This might be, but at least my experience with deployed MUAs suggest
this isn't what is implemented today.  MUAs prefer the Reply-To over
>From on "reply author".  If we want to introduce something new, I
believe it should be designed considering how the majority of MUAs
behave today, even though it may be argued that they are all broken.

Thanks,
Simon

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Tue, 31 Aug 2004 10:50:46 +0200
Lines: 47
Cancel-Lock: sha1:snOT4pw3MJ7qal7bv0rkS8kUsF0=
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> Simon Josefsson wrote:
>> Keith Moore <[email-masked]> writes:
>
>>>not if the list were on the To or CC line and the recipient did a "reply
>>>to all".  (I'm assuming that reply-to: nobody:; would just remove the
>>>From addresses from the reply recipient list, and have no effect on the
>>>addresses obtained from the To and CC fields of the subject message).
>> 
>> 
>> The problem is when people want to reply to the author, and press the
>> "reply author" button.  That operation would take the content of From:
>> as the recipient, but Reply-To would override that value with
>> nobody:;, and the reply would go to the bit bucket.
>
> Presumably if the author has set Reply-To to an ineffective address,
> he explicitly doesn't want personal responses.

Exactly.

Setting Reply-To: nobody:; was proposed as a solution to the problem
avoiding direct _copies_ of discussions on mailing lists.  Reply-To:
achieve this.

But, as you say, it also achieve another thing, that the user won't
get personal response at all.  This is not what people using MFT
wants, hence Reply-To: nobody:; is not a good replacement.

My impression of the people using MFT, and I'm on several mailing
lists where it is used heavily, is that they _want_ personal replies
to go to themselves, but they don't want copies of the mailing list
discussion sent to them personally.

> Somebody who really wants to override the authors' wishes can of
> course still send a message manually.  Of course it takes a small
> amount of effort, but that is again presumably in accordance with
> the authors' wishes in setting up the message to discourage direct
> responses.  So what's the problem?

The small amount of manual work is required to be done manually,
that's the problem.  The buttons "reply all" and "reply author" work
in many MUAs today.  A solution that break "reply author", and require
users to manually change the address manually, is a non-starter IMHO.

Thanks,
Simon

Date: Mon, 30 Aug 2004 18:56:48 -0700 (PDT)
From: [email-masked]
Subject: Re: The job of an MSA
To: Bruce Lilly <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > First, RFC 2746 is quite clear: Port 25 MAY be used for SUBMIT if for whatever
> > reason 587 cannot be used.

> 2476 -- otherwise OK.

> > More generally, what port something is run on is largely irrelevant; the
> > important issues are service and protocol semantics.

> If the protocols are to be treated as fundamentally different -- and 2476
> says that an MSA runs ESMTP with some restrictions (N.B. not extensions) --
> then that raises the question of how a client which connects to a server
> on port 25 is able to determine whether that server is an MSA or an MTA.

It would have to be configured to know. But this is for the most part an
entirely academic point, since most clients are only prepared to perform
submission activities. Clients prepared to do relaying, with all it implies,
are hardly the norm.

> > Of course this doesn't mean someone won't attempt to use SUBMIT-specific
> > extensions in regular SMTP.

> At the moment, as I understand it, there are no such "extensions", there
> are only "restrictions" [RFC 2476 section 3.1].

> > But just because some clueless ISP engages in
> > egregiously bad practice is no reason to break our protocol specifications. I
> > am dubious about the value of such an extension in any case for all the reason
> > I've previsouly given; I would vehemently oppose making it part of regular
> > SMTP.

> It may be the case that presence of one or more hypothetical submission-
> specific extensions might serve to distinguish an MSA from an MTA. That
> would mean that the presence of such an extension would indicate that the
> server in question is an MSA (running ESMTP), not an MTA (running ESMTP).

> Of course if a particular MSA doesn't support any of the hypothetical
> MSA-specific extensions, there is still the issue of how a client is
> supposed to be able to determine whether it has connected to an MTA or
> an MSA.  What do you suggest?

The idea of offering a specific extension saying "this is a submission server"
or "this is a SMTP relay server" was considered and rejected.  I doubt very
much if revisiting this issue at this time will produce a different result. As 
such, should the issue of a client needing to know the different ever come up,
the only way to do it would be with a configuration option.

				Ned

Date: Mon, 30 Aug 2004 19:42:20 -0400
From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

>>>>(a) Reply-To with some magic address that is valid syntax but a
>>>>no-op. maybe this:
>>>
>>>>Reply-To: nobody :;
>>>
>>>But in the case of mailing lists, which is where this requirement most
>>>often arises, that might also prevent replies being sent to the
>>>mailing list (except in MUAs that had an explicit command for
>>>"Reply-To-List").
>>
>>not if the list were on the To or CC line and the recipient did a "reply
>>to all".  (I'm assuming that reply-to: nobody:; would just remove the
>>From addresses from the reply recipient list, and have no effect on the
>>addresses obtained from the To and CC fields of the subject message).
> 

> The problem is when people want to reply to the author, and press the
> "reply author" button.  That operation would take the content of From:
> as the recipient, but Reply-To would override that value with
> nobody:;, and the reply would go to the bit bucket.

if you think about it, anytime someone wants to reply to the _author_,
they (or their MUA) should use the From field.  it's not reasonable to
assume that the Reply-To field points to the author.  it is reasonable
to expect authors to set the From address to a valid address that will 
reach them.

Date: Mon, 30 Aug 2004 19:34:34 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Simon Josefsson <[email-masked]>
CC: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

Simon Josefsson wrote:
> Keith Moore <[email-masked]> writes:

>>not if the list were on the To or CC line and the recipient did a "reply
>>to all".  (I'm assuming that reply-to: nobody:; would just remove the
>>From addresses from the reply recipient list, and have no effect on the
>>addresses obtained from the To and CC fields of the subject message).
> 
> 
> The problem is when people want to reply to the author, and press the
> "reply author" button.  That operation would take the content of From:
> as the recipient, but Reply-To would override that value with
> nobody:;, and the reply would go to the bit bucket.

Presumably if the author has set Reply-To to an ineffective address,
he explicitly doesn't want personal responses.  Somebody who really
wants to override the authors' wishes can of course still send a
message manually.  Of course it takes a small amount of effort, but
that is again presumably in accordance with the authors' wishes in
setting up the message to discourage direct responses.  So what's
the problem?

Date: Mon, 30 Aug 2004 19:28:36 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Arnt Gulbrandsen wrote:
> 
> Nathaniel Borenstein writes:
> 
>> Hmm, now I'm really confused.  If you're about to transmit a set of
>> messages, and what you have is an mbox file whose format you think you
>> understand, why don't you package it up as a proper MIME multipart
>> before transmitting???  -- Nathaniel
> 
> 
> HTTP too uses content-types. What's the proper type for something like
> http://www.imc.org/ietf-822/entire-arch.txt?

Presumably the HTTP server in question can parse the archive in question
("whose format you think you understand"). In that case, it could be
sent as multipart/digest, which is the "proper MIME multipart" for a
collection of messages.

Failing that, application/octet-stream would be a discrete media type
possibility, equivalent to the proposal in the draft w/o the parameters
suggested in discussion here, with the subtype tag "octet-stream" instead
of "mbox" (and lacking any parameters or specified format, the receiver
has to play guessing games with the content with either subtype tag).

Date: Mon, 30 Aug 2004 15:10:14 -0700
From: Bill McQuillan <[email-masked]>
To: RFC822 <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

On Mon, 2004-08-30 at 11:40:17, Nathaniel Borenstein <[email-masked]> wrote:

> Hmm, now I'm really confused.  If you're about to transmit a set of 
> messages, and what you have is an mbox file whose format you think you 
> understand, why don't you package it up as a proper MIME multipart 
> before transmitting???  -- Nathaniel

It seems to me that there is one piece of information that most mbox formats
have that does not have a corresponding place in message/rfc822, and that is the
date that the message was appended to the mbox. This field is used in many MUAs
to order the display in the inbox absent any threading or other sort option. It
is not quite the same as adding a Received: field just after the Return-Path:
that would contain the From_ address.

-- 
Bill McQuillan <[email-masked]>

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Mon, 30 Aug 2004 17:03:19 -0400
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

Oops, you're right, I guess this pesky web thing isn't going away any 
time soon.. :-)  On the other hand at least having it labeled 
"text/plain" isn't going to mislead anyone into misparsing it!  -- 
Nathaniel

On Aug 30, 2004, at 3:05 PM, Arnt Gulbrandsen wrote:

>
> Nathaniel Borenstein writes:
>> Hmm, now I'm really confused.  If you're about to transmit a set of 
>> messages, and what you have is an mbox file whose format you think 
>> you understand, why don't you package it up as a proper MIME 
>> multipart before transmitting???  -- Nathaniel
>
> HTTP too uses content-types. What's the proper type for something like 
> http://www.imc.org/ietf-822/entire-arch.txt?
>
> Arnt
>
>
>

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Mon, 30 Aug 2004 22:49:32 +0200
Lines: 28
Cancel-Lock: sha1:WDEPvO1YArRihnXfDWkC0E/Ipu0=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> >(a) Reply-To with some magic address that is valid syntax but a
>> >no-op. maybe this:
>> 
>> >Reply-To: nobody :;
>> 
>> But in the case of mailing lists, which is where this requirement most
>> often arises, that might also prevent replies being sent to the
>> mailing list (except in MUAs that had an explicit command for
>> "Reply-To-List").
>
> not if the list were on the To or CC line and the recipient did a "reply
> to all".  (I'm assuming that reply-to: nobody:; would just remove the
> From addresses from the reply recipient list, and have no effect on the
> addresses obtained from the To and CC fields of the subject message).

The problem is when people want to reply to the author, and press the
"reply author" button.  That operation would take the content of From:
as the recipient, but Reply-To would override that value with
nobody:;, and the reply would go to the bit bucket.

Yes, users should be able to manually override the address, but
forcing users into manual work like that seems like poor engineering
to me.

Regards,
Simon

Date: Mon, 30 Aug 2004 16:12:18 -0400
From: Keith Moore <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

> 
> Hmm, now I'm really confused.  If you're about to transmit a set of 
> messages, and what you have is an mbox file whose format you think you 
> understand, why don't you package it up as a proper MIME multipart 
> before transmitting???  -- Nathaniel

because for many things, in practice, mbox files work better than
MIME multiparts.  more tools read and write mbox files than read
and write MIME multiparts.  MUAs tend to handle messages within
MIME multiparts poorly - for instance, they often don't let you
reply to them, refile them to other folders, etc.  MIME multipart 
doesn't have the property that you can add messages to it by 
appending to the file.  

Date: Mon, 30 Aug 2004 15:53:50 -0400
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

> 
> Simon Josefsson writes:
> > At this point, perhaps it is useful to enumerate requirements on the
> > solutions.
> 
> You mentioned two requirements:
> 
>    (1) letting the sender exclude the reply address (normally From or
>        Reply-To) from the followup list;
>    (2) letting the sender include a public recipient while excluding
>        that recipient from the followup list.
> 
> Here are two more requirements:
> 
>    (3) letting the sender _include_ the reply address in the followup
>        list---consider non-subscriber messages to mailing lists;
>    (4) letting the sender specify more than one address for
>    followups---
>        consider messages crossposted to two mailing lists.

Aside: I generally find it unhelpful to enumerate "requirements" because
these things have different degrees of importance.  There's nothing
wrong with enumerating "goals" or enumerating "problems" to be solved,
but calling them "requirements" makes it seem that they're
non-negotiable and all of equal importance.  We'd probably be happy with
a partial solution - one that solved the worst of the problems without
introducing harmful unintended effects -, and we probably won't find a
complete solution anyway.  

> To-NoReply flunks #1.

Not so, the sender has the option of saying "Reply-To: nobody:;" 

Keith

Date: Mon, 30 Aug 2004 15:30:36 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

For one-off's like that, it's impossible. You know that as well as I do; 
the mboxrd/mboxo formats are simpler to parse when you're doing an 
extremely simple thing like that.

But it doesn't matter. People usually parse mailboxes to do more complex 
things, and the amount of additional code to handle content-length is 
usually insignificant compared to the overall amount of code.

And yes, this is a subjective measurement, no more or less than the 
original statement.

And this ends my comments on the topic.

	Tony

D. J. Bernstein wrote:

> Tony Hansen writes:
> 
>>Whether parsing mboxcl2 takes "much" more code than parsing mboxrd is a 
>>subjective measurement.
> 
> Here's complete awk code to count the number of messages in an mbox,
> assuming mboxrd (or mboxo) format:
> 
>    awk '/^From / { ++sum } END { print sum }'
> 
> Please show us your code to do the same thing with mboxcl2.
> 
> Remember that, if you don't check Content-Length, you'll sometimes break
> messages at the wrong place. Also remember that Content-Length can be
> capitalized and spaced strangely, maybe even across lines.

Date: Mon, 30 Aug 2004 15:22:39 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: reply problem list
Sender: [email-masked]

Okay, I dug out the "reply problem list" I had written as part of the
DRUMS discussions, and edited it slightly.  

I've tried to define these problems in terms of observed behavior, not
in terms of "we need a header that does X". This was an attempt to avoid
prejudicing readers to favor some anticipated solution.  Also in these
discussions there seems to be a tendency to define problems in terms of
assumed solutions, which makes it difficult to compare one proposed
solution with another one.  ("the problem is that we don't have a FOO
header; so if we had a FOO header the problem would be solved").

Many of these problems are very similar to one another, especially if
you group them in terms of "this problem is solved by adding header
FOO". I have listed them as separate problems because some proposed
solutions will solve one problem while failing to solve a similar one.
By listing so many separate problems, I don't mean to imply that they're
all of equal severity. 

I separated the original list into two sections.  List A consists of the
problems that aren't related to lists munging Reply-To.  List B consists
of the problem related to lists munging Reply-To.  I'm guessing that
hardly anyone on this mailing list still believes its good practice for
lists to mung Reply-To, so for the time being I'm posting just List A
here.  If you want to see both lists, look at
http://www.cs.utk.edu/~moore/opinions/reply-problem-list.html .

My reason for compiling this is to help clarify arguments of the form
"proposal X doesn't solve problem Y".   It appears that any solution to
the _set_ of problems is likely to involve several changes: perhaps a
new header field or two, along with (perhaps) recommendations to MUA
authors about their user interfaces, modifications to various kinds of
software, and maybe some user education.  With this list we can compare
various _sets_ of solutions to this _set_ of problems.

List A: REPLY problems not involving modification of message headers by
mailing lists:

A1.	Given the vast number of MUAs that implement a "reply all" command
where the Reply-To address only replaces the From address from the
original message, there's no good way for the author of a message to say
"please reply only to these addresses", or (almost equivalently) "please
don't reply to these addresses". The author can use Bcc for all
recipients who should not receive the reply, but that doesn't inform the
"non blind" recipients that the other recipients were also sent copies
of the message.

A2.	If a message contains a Reply-To field, but the person composing the
reply isn't aware that Reply-To is being used, the reply may go to
somewhere besides where the reply author thought it should go. The
result may be surprising or embarrassing (as when a personal reply gets
sent to a list), or the reply author may think he has sent the message
to a list when it has really only gone to one particular recipient.

A3.	A few MUAs implement only one kind of reply -- which uses Reply-To
if present and From otherwise. Users of such MUAs have difficulty
participating in group discussions on mailing lists (unless the list
mungs Reply-To).

A4.	If a message is sent to a mailing list, and someone does a "reply
all" to that message, the author often gets two copies - one sent to the
>From address and another sent to the list address (which presumably
appeared in the To or Cc field of the subject message).

A5.	If a user is subscribed to multiple mailing lists, and a message is
sent to more than one of those mailing lists, the user will get multiple
copies of the message. This might not be so bad, but the same user will
also get duplicate copies of the message every time someone does "reply
to all" to the original messages or any of the replies to that message.

A6.	On many MUAs, if the sender specified Reply-To, the recipient is
stuck with that - he cannot easily edit it to reply to From or
From+To+CC (especially if he cannot see the original headers while he's
editing the reply recipient list). The author of the reply thus
sometimes has difficulty sending the reply to where it needs to go.

A7.	If someone responds to a list message and CCs themselves in the
response, everyone who does a "reply all" to that message will also Cc
that person...even if he's on one of the lists. Alternatively, if
multiple recipients (often lists) get CCed on a message, every "reply
all" goes to all of those recipients, even if the discussion topic
wanders and it's no longer appropriate for many of those recipients.

A8.	Many people have multiple mailboxes, or regularly send mail from
some account at which they don't want to receive mail. If they use
Reply-To to point to their preferred mailbox, it will prevent some
recipients from doing "reply to all". Also, since there's no indication
as to why the author used Reply-To (it could be "please don't mail to me
at my From address" or it could be "please tell my secretary, not me,
whether you'll be at the meeting"), it's easy for a reply to go to the
wrong place.

A9.	Mailing list subscribers who don't understand the reply functions of
their UA or who have UAs with severely limited reply functions often
send replies to the author of the message when they should go to the
list. This causes discussions on the list to suffer and perhaps die out.

A10.	Mailing list subscribers who don't understand the reply functions
of their UA, often send replies to the mailing list when they should go
to the author of the message. This causes prolonged discussions on a
list on which such discussions are inappropriate. 

Date: Mon, 30 Aug 2004 21:05:59 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Nathaniel Borenstein writes:
> Hmm, now I'm really confused.  If you're about to transmit a set of 
> messages, and what you have is an mbox file whose format you think 
> you understand, why don't you package it up as a proper MIME 
> multipart before transmitting???  -- Nathaniel

HTTP too uses content-types. What's the proper type for something like 
http://www.imc.org/ietf-822/entire-arch.txt?

Arnt

Date: 30 Aug 2004 18:50:46 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Charles Lindsey writes:
> Which is why the application/mbox type needs parameters so that this
> information can be given

Shall we next add parameters to the Date header field?

   Date; endianness=big: 30 Aug 2004 18:34:40 -0000
   Date; endianness=pdp: 30 Aug 0402 40:34:18 -0000
   Date; endianness=corrupted: 30 Aug 04 18 -00

Supporting mbox formats that lose information is like supporting dates
with the endianness=corrupted parameter.

I'd rather see an application/mboxrd with a careful specification of the
(working!) mboxrd syntax. Anyone who desperately wants to support
Content-Length can do it on the sending end, converting to mboxrd,
rather than demanding that every reader support Content-Length.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Mon, 30 Aug 2004 14:40:17 -0400
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

Hmm, now I'm really confused.  If you're about to transmit a set of 
messages, and what you have is an mbox file whose format you think you 
understand, why don't you package it up as a proper MIME multipart 
before transmitting???  -- Nathaniel

On Aug 30, 2004, at 7:40 AM, Charles Lindsey wrote:

>
> In <[email-masked]> Nathaniel 
> Borenstein <[email-masked]> writes:
>
>> On Aug 24, 2004, at 9:14 AM, Charles Lindsey wrote:
>>>
>>> We are talking about a new application type, which we presume some
>>> people
>>> will write new software to generate/accept (otherwise there is little
>>> point in having the application type).
>
>> Do we really want people to write new software to *generate* mbox
>> format data?  I was thinking of this as primarily a tool for allowing
>> software to label and read data that is only available in mbox format.
>> Is there any circumstance in which we would actually encourage its
>> generation?
>
> I don't think we need new software to generante mboxen, but what I was
> referring to was software/scripts to generate messages with
> application/mbox for transporting mboxen already generated by whatever
> means. Such software/script would naturally incorporate whatever
> parameters has been defined for that application type.
>
> -- 
> Charles H. Lindsey ---------At Home, doing my own 
> thing------------------------
> Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: 
> http://www.cs.man.ac.uk/~chl
> Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 
> 3JU, U.K.
> PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 
> A4 AB A5
>
>
>

Date: 30 Aug 2004 18:34:40 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

Simon Josefsson writes:
> At this point, perhaps it is useful to enumerate requirements on the
> solutions.

You mentioned two requirements:

   (1) letting the sender exclude the reply address (normally From or
       Reply-To) from the followup list;
   (2) letting the sender include a public recipient while excluding
       that recipient from the followup list.

Here are two more requirements:

   (3) letting the sender _include_ the reply address in the followup
       list---consider non-subscriber messages to mailing lists;
   (4) letting the sender specify more than one address for followups---
       consider messages crossposted to two mailing lists.

To-NoReply flunks #1. The notion of having a ``post to list,'' separate
from replies and followups, flunks both #3 and #4.

Mail-Copies-To:nobody flunks #2. As for #1, Mail-Copies-To:nobody might
seem at first glance to be simpler than Mail-Followup-To, but in fact
the code to do

   Mail-Followup-To || (To + Cc + (Mail-Reply-To || Reply-To || From))
   Mail-Reply-To || Reply-To || From

tends to be a trivial adaptation of the code already written to do

   To + Cc + (Reply-To || From)
   Reply-To || From

whereas Mail-Copies-To:nobody breaks the address-list encapsulation and
requires some new low-level code.

Mail-Followup-To supports #1, #2, #3, and #4, and it does so in a way
that's reasonably easy for both ends to automate. It already has heavy
use in the real world. Nobody has suggested a simpler way to meet the
requirements.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Mon, 30 Aug 2004 14:27:27 -0400
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

> http://www.google.com/search?as_q=mail-followup-to

Yes, I had already done that.  The references only prove my point - most
of the "significant" ones are are associated with MUAs that hardly
anybody uses.  Yes, there are 500K+ hits, but many of them are from
archived list messages from people using Exim or Mush or some obscure
MUA.  They're not a random sample of email messages, and they're
not representative of overall user preference.  And 500K uses over
a period of several years is a drop in the bucket.

If you searched net-accessible archives for messages where the list
had sent reply-to to point to the list, you'd find a lot of hits 
for those also.  But that doesn't say much about overall user
preference, and even less about the desirability of lists munging
reply-to.

Keith

Date: Mon, 30 Aug 2004 14:18:07 -0400
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

> >(a) Reply-To with some magic address that is valid syntax but a
> >no-op. maybe this:
> 
> >Reply-To: nobody :;
> 
> But in the case of mailing lists, which is where this requirement most
> often arises, that might also prevent replies being sent to the
> mailing list (except in MUAs that had an explicit command for
> "Reply-To-List").

not if the list were on the To or CC line and the recipient did a "reply
to all".  (I'm assuming that reply-to: nobody:; would just remove the
>From addresses from the reply recipient list, and have no effect on the
addresses obtained from the To and CC fields of the subject message).

Keith

Date: 30 Aug 2004 17:59:44 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

Keith Moore writes:
> an insignificant part of the installed base

http://www.google.com/search?as_q=mail-followup-to

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: [email-masked]
Xref: clerew local.mime:3437
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Mon, 30 Aug 2004 11:25:34 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>Gnus support an author-copies avoiding solution, by obeying the
>`Mail-Copies-To: nobody' news header even for e-mail.  See
><http://www.newsreaders.com/misc/mail-copies-to.html>.

>It might be complicated to specify Mail-Copies-To for e-mail, in a way
>that make sense for all combination of news/e-mail and handover from
>news to mail and vice versa.  But it may be possible.  Perhaps by
>adopting, for e-mail use, only the "nobody" subset of MCT.

MCT does not make much sense in Email, except for mailing lists, where it
should work quite well.

If you wrote
    Mail-Copies-To: poster
in an ordinary email, then it would be of null effect, because that is
where replies go to in any case. But it would do no harm.

But in a MUA with a "Reply-To-List" feature (which few MUAs have, but
which would reply to the List-Post address), then "MCT: poster" would cause
that command to reply to the poster as well as to the list, which is just
fine.

Likewise, "Mail-Copies-To: [email-masked]" would be no different to
Reply-To in an ordinary mail, but would do the Right Thing with a
"Reply-To-List" command, where provided.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3440
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Mon, 30 Aug 2004 11:36:20 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Bill McQuillan <[email-masked]> writes:

>Since the existing To: and CC: fields are treated nearly identically by
the mail system and are basically just used to designate the "primary" and
"secondary" recipients of the message to the human readers, I do not think
that there needs to be yet two more classes of recipients. Instead, I
think that a single "tertiary" recipient class which is intended not to be
bothered by subsequent exchanges can be designated.

>Not altogether facitiously, I propose to extend the sensory deprivation
metaphor of Blind Carbon Copy (BCC:) to Deaf Carbon Copy (DCC:) for this
tertiary class since follow-on messages in the conversation will not be
"heard" by them.  ;-)

Yes, I rather like this, in principle, though a better name for the header
is needed.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3442
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Mon, 30 Aug 2004 11:44:08 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>Charles Lindsey writes:
>> you find it has been corrupted by all the '>' stuffing

>Certainly the old mboxo format should be nuked---but there are other
>mbox variants that don't corrupt messages.

>The mboxrd format inserts ">" in front of any line beginning "From ",
>">From ", ">>From", etc., and then inserts a "From " line at the top.
>This is trivially reversible. Readers recognize "From " as the start of
>a new message; readers strip ">" from ">From ", ">>From ", etc.

Yes, that is fine so long as you know that you are dealing with an mboxrd,
but in general you don't. Which is why the application/mbox type needs
parameters so that this information can be given (in cases where the
sender know it, of course).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3439
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Mon, 30 Aug 2004 11:34:02 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> There is an alternative, and possibly better, way to solve this problem.
>> 
>> Leave the To: and Cc: headers alone, with their present meanings, and let
>> them be included in all messages as at present.
>> 
>> Now invent ONE new header
>> 
>> no-reply-to	= "No-Reply-To" noreplylist CRLF
>> noreplylist	= ( "To" / "Cc" / "From" / "Reply-To" / "List-Post" /
>> 		    "Bcc" / ... / address) [ "," noreplylist ]
>> 
>> which essentially says, "when replying to this message, kindly do not
>> reply to any of the people listed". So you can exclude the whole of the
>> "To:" list, the whole of the "Cc:" list, the whole of the "From:" list
>> (plus any others that might be allowed) and you can alternatively, or
>> additionally, exclude any explicitly listed <addresses>.

>If you'll note in the draft, I considered something similar to this.
>As you point out, it's not compatible with legacy MUA implementations.
>It also would arguably have one header field changing the meaning of
>another header field.  My personal belief is that the "sendmail -t"
>problem is a minor one - since it seems to affect a relatively small
>number of older MUAs. Others might disagree.

I think it is clear now that sendmail -t is widely used, although not
necessarily by conventional MUAs.

>> Also, while you are about it, can we include some mechanism for "reply to
>> list" in mailing lists? 

>how would this be different than List-Post:?

Yes, but few MUAs provide a facility to direct replies to the List-Post
address. But since your draft already contains much non-normative
material, a plug for including such a feature in MUAs might be useful.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3441
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Mon, 30 Aug 2004 11:40:38 GMT
Lines: 31
Sender: [email-masked]

In <[email-masked]> Nathaniel Borenstein <[email-masked]> writes:

>On Aug 24, 2004, at 9:14 AM, Charles Lindsey wrote:
>>
>> We are talking about a new application type, which we presume some 
>> people
>> will write new software to generate/accept (otherwise there is little
>> point in having the application type).

>Do we really want people to write new software to *generate* mbox 
>format data?  I was thinking of this as primarily a tool for allowing 
>software to label and read data that is only available in mbox format.  
>Is there any circumstance in which we would actually encourage its 
>generation?

I don't think we need new software to generante mboxen, but what I was
referring to was software/scripts to generate messages with
application/mbox for transporting mboxen already generated by whatever
means. Such software/script would naturally incorporate whatever
parameters has been defined for that application type.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3438
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Mon, 30 Aug 2004 11:30:33 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>I'd rather avoid having a definition of Mail-Copies-To in email that 
>differed from the definition used in usenet.

If there is something useful that could be changed in the usenet
definition of MCT in order to accomodate its usage in email, then it is
not too late to incorporate same into USEFOR.

>I can imagine two ways to solve the problem:

>(a) Reply-To with some magic address that is valid syntax but a no-op. 
>maybe this:

>Reply-To: nobody :;

But in the case of mailing lists, which is where this requirement most
often arises, that might also prevent replies being sent to the mailing
list (except in MUAs that had an explicit command for "Reply-To-List").

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 30 Aug 2004 10:55:50 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: 822 <[email-masked]>
Organization: Bruce Lilly
To: [email-masked]
CC: 822 <[email-masked]>
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Laird Breyer wrote:
> On Aug 29 2004, Bruce Lilly wrote:
> 
> 
>>Why on Earth any sane person would set From to a non-
>>working address with no Reply-To field is beyond
>>comprehension. (Well, perhaps no *sane* person would...)

[...]
> I
> presume these people will also set their From and Reply-To fields
> accordingly for each email conversation and recipient, otherwise there
> would be no point.
[...]
> So in answer to your question, the sane person might be a customer using
> such a system.

Not likely, since the message in question had *no* Reply-To field.

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: the most obvious failure in To-NoReply
Date: Mon, 30 Aug 2004 16:28:08 +0200
Lines: 55
Cancel-Lock: sha1:Wxf8peFIyM5kbxkoS0Kj2IfMCkI=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> D. J. Bernstein wrote:
>> Keith Moore writes:
>> 
>>>you're free to set Reply-To to [email-masked]
>> I don't want _replies_ to go to [email-masked]. I want _followups_
>> to go to [email-masked].
>
> "followup" is a Usenet term.  Usenet has two different kinds of 
> addresses - email addresses and newsgroups.  Responses to the author's 
> email address are replies; responses to newsgroups are followups.  Email 
> does not have visibly different kinds of addresses and does make a 
> distinction between replies and followups.  Email UAs have "reply" and 
> "reply to all" or similar.  For several reasons including both protocol 
> differences and cultural differences between email and Usenet, email's 
> "reply to all" and Usenet's "followup" are not quite the same thing. 

What's the difference?  I view them as conceptually the same thing.

> And I don't see evidence of any desire among users to make email more 
> like Usenet.  (actually I see plenty of evidence to the contrary.  but I 
> digress.)

Perhaps the users of gmane.org would disagree.  I'm reading this
mailing list via news through their services, for example.

>> An increasing number of MUAs support the sender specifying
>> a followup list (normally with Mail-Followup-To) that doesn't include
>> the reply list; that's what users want.
>
> A few MUAs support MFT.  The ones I've seen that support MFT represent 
> an insignificant part of the installed base, so I don't accept an 
> argument that the level of MUA spport for MFT is an indication of user 
> preference.
>
> For that matter, the most widely used OS by far is rife with security 
> holes and has been for nearly 10 years now, but I don't accept that as 
> an indication of user preference for security holes.
>
> It seems to me that you've had a lot of time to try to get MFT widely 
> accepted in the marketplace.  Your failure to do so doesn't necessarily 
> mean that MFT is bad, and it says nothing about the relative merits of 
> MFT vs. NR.  But it probably does mean that claims that equate market 
> acceptance with rightness are moot in this discussion.

Right.  But as far as standardization goes, having a proposal
implemented and tested in practice is a strong argument.  Running
code, and all that.  The problems with MFT appear to be well
understood, and some communities still chose to use MFT.  So I
believe, regardless of any NR vs MFT discussion, that the needs of the
people using MFT should be taken care of by any proposal.

Thanks,
Simon

Date: Mon, 30 Aug 2004 09:14:02 -0400
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

D. J. Bernstein wrote:
> Keith Moore writes:
> 
>>you're free to set Reply-To to [email-masked]
> 
> 
> I don't want _replies_ to go to [email-masked]. I want _followups_
> to go to [email-masked].

"followup" is a Usenet term.  Usenet has two different kinds of 
addresses - email addresses and newsgroups.  Responses to the author's 
email address are replies; responses to newsgroups are followups.  Email 
does not have visibly different kinds of addresses and does make a 
distinction between replies and followups.  Email UAs have "reply" and 
"reply to all" or similar.  For several reasons including both protocol 
differences and cultural differences between email and Usenet, email's 
"reply to all" and Usenet's "followup" are not quite the same thing. 
And I don't see evidence of any desire among users to make email more 
like Usenet.  (actually I see plenty of evidence to the contrary.  but I 
digress.)

> Every popular MUA has separate reply/followup functions;  that's what
> users want.

Every popular MUA has separate "reply" and "reply all" functions.  I'm 
quite willing to believe that a significant plurality of users want both 
of these functions.  (I'm not sure about "most" as I keep running into 
users who don't seem to understand the difference between "reply" and 
"reply to all".  It may be that most users never learn to choose - they 
just hit one button or another out of habit, or at random.)

> An increasing number of MUAs support the sender specifying
> a followup list (normally with Mail-Followup-To) that doesn't include
> the reply list; that's what users want.

A few MUAs support MFT.  The ones I've seen that support MFT represent 
an insignificant part of the installed base, so I don't accept an 
argument that the level of MUA spport for MFT is an indication of user 
preference.

For that matter, the most widely used OS by far is rife with security 
holes and has been for nearly 10 years now, but I don't accept that as 
an indication of user preference for security holes.

It seems to me that you've had a lot of time to try to get MFT widely 
accepted in the marketplace.  Your failure to do so doesn't necessarily 
mean that MFT is bad, and it says nothing about the relative merits of 
MFT vs. NR.  But it probably does mean that claims that equate market 
acceptance with rightness are moot in this discussion.

Cc: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: [email-masked]
Date: Mon, 30 Aug 2004 11:20:27 +0100
From: Nick Ing-Simmons <[email-masked]>
Reply-To: Nick Ing-Simmons <[email-masked]>
Sender: [email-masked]

D. J. Bernstein <[email-masked]> writes:
>Tony Hansen writes:
>> Whether parsing mboxcl2 takes "much" more code than parsing mboxrd is a 
>> subjective measurement.
>
>Here's complete awk code to count the number of messages in an mbox,
>assuming mboxrd (or mboxo) format:
>
>   awk '/^From / { ++sum } END { print sum }'
>
>Please show us your code to do the same thing with mboxcl2.
>
>Remember that, if you don't check Content-Length, you'll sometimes break
>messages at the wrong place. Also remember that Content-Length can be
>capitalized and spaced strangely, maybe even across lines.

The other problem with Content-Length skipping N bytes is that it is 
broken by CRLF conversion, which makes it harder to read same mbox 
on Win32 and Unix (but not impossible). 
It is also essential to "ignore" Content-Length that may be present in 
the incomming mail and re-compute it as you write it to your local mbox.

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: the most obvious failure in To-NoReply
Date: Mon, 30 Aug 2004 11:51:45 +0200
Lines: 110
Cancel-Lock: sha1:YrA3jvToKGWNAISE0eTBVrZH1Ec=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>>>NR has no way to specify situations where "reply to all" is not sent
>>>to a superset of the recipients who would be sent if "reply to author"
>>>were used instead.
>> Then NR is broken.
>
> Or alternately: NR may not solve the problem you want to solve.

Right.  I hadn't realize NR didn't solve this problem, but now I have
to say that if this mode isn't supported, I believe the benefits of NR
will be marginal compared to the cost of introducing it.

Especially considering that people will have to continue use MFT, or
standardize another header to do the same, to get a behavior they
want.

I believe NR need to address this.

> Depending on how you look at it, NR and MRT are both broken (for 
> different reasons), and the absence of some solution(s) to several 
> problems associated with replies is also broken.  The question is 
> whether, and how, we can get to a significantly less broken state with a 
> minimum of pain.

At this point, perhaps it is useful to enumerate requirements on the
solutions.  After doing so, it is easier to evaluate which solutions
are less broken, or alternatively: solve the problems people want
solved.  Two requirement:

1. Exclude the originator from being added to To/Cc on followups to
   mailing lists, but keeping reply functionality from working.

2. Being able to include recipients on a copy, like BCC, but that
   their names are shown in the message (and displayed by receiving
   MUA), without those recipients receive further copies of the
   discussion.  (NR)

I know that (1) is a real world requirement, and I believe it has been
the driving reason for MFT deployment.  I'm not so sure about 2, but
NR advocate for this scenario (unless I misunderstand it).

We have three approach that are specified, although not all in IETF
drafts: MFT (deployed), NR, and Mail-Copies-To (MCT), the latter is
used for news, but (at least) Gnus adhere to it in mail too, giving
similar functionality.  An evaluation:

      NR        MFT      MCT
1:   No         Yes[1]   Yes[1]
2:   Yes         No[2]    No[2]

Regarding [1]:

MFT achieve [1] by specifying _exactly_ where follow ups should go,
for example:

  Mail-Followup-To: [email-masked]

where MCT would achieve this indirectly, by:

  Mail-Copies-To: nobody

The latter would survive a rename of [email-masked] to [email-masked],
without the need for users to modify their MFT configurations, which I
see as an advantage.  It would also survive gate:ing between news and
mail, and still make sense.

Regarding [2]:

It must be remembered that it is always possible to do:

  Bcc: [email-masked]

and state this inside the message:

  Note!  I've sent [email-masked] a copy of this e-mail.

When would this be insufficient?  I can think of two reasons.

First, internationalization.  If not all recipients understand the
same set of languages.  Mail can be used to transfer other things than
text, after all, so mail should be useful when there is no shared
writing system.

Secondly, it would be insufficient when we'd want programs to act on
the information.  No MUA would be able to display
[email-masked] in the To/Cc line, with some different markup.
Nor would any script be able to extract [email-masked] as a
recipient of the original message, useful for e.g. message
cancellation.

>> When I send a message to [email-masked], I do _not_
>> want followups to be sent to the original reply address. 
>
> Note that even without either MFT or NR, you're free to set Reply-To to 
> [email-masked].  If the recipient really wants to reply to just the 
> author, he can always manually override the To field of the reply.  (and 
> if he can't do that, his MUA is broken :)  That's no different than many 
> other valid uses of Reply-To: e.g. when someone sends a message and 
> wants normal replies to go to his secretary, but a particular recipient 
> of that message really does need to send a reply to the message author.

The difference is the amount of work requirement, and the annoyance
that result when the manual work is not been done.

MFT is widely adopted in some groups because of this, and I believe
any solution in this area should address those needs.

Thanks,
Simon

Date: 30 Aug 2004 06:07:36 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

Keith Moore writes:
> you're free to set Reply-To to [email-masked]

I don't want _replies_ to go to [email-masked]. I want _followups_
to go to [email-masked].

Every popular MUA has separate reply/followup functions; that's what
users want. An increasing number of MUAs support the sender specifying
a followup list (normally with Mail-Followup-To) that doesn't include
the reply list; that's what users want.

Since adopting To-NoReply wouldn't remove the need for Mail-Followup-To,
and since To-NoReply is useless in a Mail-Followup-To world, there's no
reason for anyone to adopt To-NoReply.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sun, 29 Aug 2004 22:02:07 -0400
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

>>NR has no way to specify situations where "reply to all" is not sent
>>to a superset of the recipients who would be sent if "reply to author"
>>were used instead.
> 
> 
> Then NR is broken.

Or alternately: NR may not solve the problem you want to solve.

Depending on how you look at it, NR and MRT are both broken (for 
different reasons), and the absence of some solution(s) to several 
problems associated with replies is also broken.  The question is 
whether, and how, we can get to a significantly less broken state with a 
minimum of pain.

> When I send a message to [email-masked], I do _not_
> want followups to be sent to the original reply address. 

Note that even without either MFT or NR, you're free to set Reply-To to 
[email-masked].  If the recipient really wants to reply to just the 
author, he can always manually override the To field of the reply.  (and 
if he can't do that, his MUA is broken :)  That's no different than many 
other valid uses of Reply-To: e.g. when someone sends a message and 
wants normal replies to go to his secretary, but a particular recipient 
of that message really does need to send a reply to the message author.

Date: 30 Aug 2004 02:01:58 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Tony Hansen writes:
> Whether parsing mboxcl2 takes "much" more code than parsing mboxrd is a 
> subjective measurement.

Here's complete awk code to count the number of messages in an mbox,
assuming mboxrd (or mboxo) format:

   awk '/^From / { ++sum } END { print sum }'

Please show us your code to do the same thing with mboxcl2.

Remember that, if you don't check Content-Length, you'll sometimes break
messages at the wrong place. Also remember that Content-Length can be
capitalized and spaced strangely, maybe even across lines.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Mon, 30 Aug 2004 10:21:12 +1000
From: Laird Breyer <[email-masked]>
To: 822 <[email-masked]>
Subject: Re: draft-moore-mail-nr-fields-00
Reply-To: [email-masked]
Mail-Followup-To: 822 <[email-masked]>
Sender: [email-masked]

On Aug 29 2004, Bruce Lilly wrote:

> Why on Earth any sane person would set From to a non-
> working address with no Reply-To field is beyond
> comprehension. (Well, perhaps no *sane* person would...)

I'm aware of at least one "antispam solution" which works on
this kind of principle, if I understand your meaning.

This was presented at a past MIT spam conference, the commercial
web page is here: http://www.titankey.com/productInfo/howItWorks/default.htm

The idea is that instead of a single mail address, people own an
unlimited supply of disposable addresses, with time limits etc.  I
presume these people will also set their From and Reply-To fields
accordingly for each email conversation and recipient, otherwise there
would be no point. It "stops spam" because you simply turn off the mail
address with the press of a button and the SMTP server will reject
messages at the gate. That's the rough idea I believe.
 
So in answer to your question, the sane person might be a customer using
such a system.

-- 
Laird Breyer.

Date: Sun, 29 Aug 2004 16:49:04 -0700
Subject: Re: internationalization of mail
To: [email-masked]
From: Eric Fischer <[email-masked]>
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Bruce Lilly writes:
>
>> Aside from font issues, Unicode normalization uses huge tables which 
>> might
>> not be practical for some devices which do support email.  Therefore 
>> transcoding a space- and memory-efficient charset into Unicode may 
>> well render the transformed message unusable or unreplyable if 
>> transferred (forwarded, etc.) to some devices [it is not unusual for 
>> mail to be forwarded to PDAs, mobile phones, pagers, etc.].
>
> Some chap at a Unicode conference talked about a trie-based 
> implementation that squeezed the tables into 35k. I believe he was from 
> Psion.
>
> 35k isn't huge, not even on a handheld.

For the Danger Hiptop (T-Mobile Sidekick) we represent all mail text and 
headers on the device as UTF-8 for storage and network transmission and 
UTF-16 for display.  We do not have normalization tables on the device 
and so far have not not missed them.  The tables we do have are for 
character classes (since that data is required to be Java compliant) and 
for sorting.  We cheat a little on the sorting by not having collation 
data for characters that are not in any of our fonts.

The only real downside I've found to not keeping the original encodings 
on the device is that there is no way to reencode text whose charset was 
mislabeled without refetching it over the network.  But in practice I 
see a lot more text that is completely unlabeled or systematically 
mislabeled (claiming to be ISO-8859-1 instead of windows-1252, for 
instance) than that claims to be one thing but is actually something 
substantially different.

Eric

Date: Sun, 29 Aug 2004 19:48:11 -0400
From: Keith Moore <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

Russ Allbery wrote:
> Keith Moore <[email-masked]> writes:
> 
> 
>>here's something new to argue about :)
>>I'm curious to know what people think of this idea.
> 
> 
> Very minor point, but one that's been bugging me...  Existing mail header
> fields are all either single words or use hyphens to separate words.  I'd
> really rather not see ugly StudlyCaps introduced in a new standardized
> field, particularly since fieldnames are case-insensitive and NoReply is
> at least partly reliant on capitalization to be easily understood.
> Cc-NorePly isn't easily parsable by my eye.
> 
> How about CC-No-Reply?  Cc-No-rePly still makes sense.  :)

I've also considered CC-NR and TO-NR.
I'd be happy with any reasonable spelling.

Keith

Date: Sun, 29 Aug 2004 19:45:56 -0400
From: Keith Moore <[email-masked]>
To: Tony Hansen <[email-masked]>
Cc: Nathaniel Borenstein <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, [email-masked], Charles Lindsey <[email-masked]>, Philip Hazel <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

Tony Hansen wrote:
> Nothing is broken. Sendmail can be invoked using "sendmail addresses" 
> just as easily it can be invoked using "sendmail -t". When your program 
> that uses sendmail is upgraded to recognize the NoReply fields, continue 
> to use "sendmail -t" for the To/Cc/Bcc recipients, and then use 
> "sendmail addresses" for sending the message to the NoReply recipients.

I don't get it.  It seems like the only reason to use "-t" is if the 
sending program doesn't want to bother parsing the header.  once you go 
to the trouble of modifying it to parse NoReply fields, why not have it 
parse  all recipient fields and then do "sendmail all-addresses..."?

> I hope "sendmail -t" is NEVER upgraded to support NoReply headers. That 
> would cause the worst of all possible worlds. As a "sendmail -t" user, 
> you'd never know if your invocation of "sendmail -t" included those 
> NoReply headers or not. Say your "sendmail -t" doesn't support NoReply 
> and you write you code to invoke "sendmail addresses" to handle NoReply 
> headers. Then some helpful admin upgrades your "sendmail -t" to support 
> them. Suddenly the NoReply recipients will be receiving TWO copies of 
> the messages.

another reason why I wouldn't rewrite the submission code to send 
NoReply fields separately.

> If sendmail WERE to be upgraded to support NoReply headers, then it 
> should be done using a DIFFERENT option. (For the version of sendmail on 
> my system, it looks like -T might be available.)

makes sense to me ... especially if -T causes an error with existing 
"sendmail" and clones.   (on my system sendmail -T complains about an
invalid date, so maybe it does something undocumented.)

Keith

Date: Mon, 30 Aug 2004 00:47:05 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: internationalization of mail
Sender: [email-masked]

Bruce Lilly writes:
> Aside from font issues, Unicode normalization uses huge tables which might
> not be practical for some devices which do support email.  Therefore 
> transcoding a space- and memory-efficient charset into Unicode may 
> well render the transformed message unusable or unreplyable if 
> transferred (forwarded, etc.) to some devices [it is not unusual for 
> mail to be forwarded to PDAs, mobile phones, pagers, etc.].

Some chap at a Unicode conference talked about a trie-based 
implementation that squeezed the tables into 35k. I believe he was from 
Psion.

35k isn't huge, not even on a handheld.

Arnt

Date: Sun, 29 Aug 2004 17:25:13 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: 822 <[email-masked]>
Organization: Bruce Lilly
To: 822 <[email-masked]>
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------050702080002010609010004
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

The attached is exactly the sort of spam-like cruft
that has been recently mentioned.

Why on Earth any sane person would set From to a non-
working address with no Reply-To field is beyond
comprehension. (Well, perhaps no *sane* person would...)

--------------050702080002010609010004
Content-Type: message/rfc822;
 name*=ISO-8859-1'en-us'failure%20notice
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename*=ISO-8859-1'en-us'failure%20notice

Date: 29 Aug 2004 21:16:14 -0000
From: [email-masked]
To: [email-masked]
Subject: failure notice

Hi. This is the qmail-send program at cr.yp.to.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<djb-internal/[email-masked]>:
See http://cr.yp.to/docs/copies.html.

--- Below this line is a copy of the message.

Date: Sun, 29 Aug 2004 17:15:41 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: the most obvious failure in To-NoReply

D. J. Bernstein wrote:
> Keith Moore writes:
> 
>>NR has no way to specify situations where "reply to all" is not sent
>>to a superset of the recipients who would be sent if "reply to author"
>>were used instead.
> 
> 
> Then NR is broken.

Not necessarily so. Kindly explain under precisely which circumstance(s)
"reply-to-all" should *not* be a superset of "reply-to-author" and why.

> When I send a message to [email-masked], I do _not_
> want followups to be sent to the original reply address. That's why my
> message says Mail-Followup-To: [email-masked].

There is no "mail-followup-to" -- it is defined in no RFC and in
no Internet draft.  If one wants replies sent to a specific list
of addresses, that is precisely what the standard Reply-To header
field is for.

>>as far as I can tell, this is a plus for NR.
> 
> 
> Then you are fundamentally confused about what mail users want to do.

Anybody who apparently thinks that mail users wish to receive strident
spam-like "you didn't honor "mail-followup-to'" cruft is fundamentally
confused about what mail users want and don't want.

--------------050702080002010609010004--

Date: Sun, 29 Aug 2004 17:15:41 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked]
Subject: Re: the most obvious failure in To-NoReply
Sender: [email-masked]

D. J. Bernstein wrote:
> Keith Moore writes:
> 
>>NR has no way to specify situations where "reply to all" is not sent
>>to a superset of the recipients who would be sent if "reply to author"
>>were used instead.
> 
> 
> Then NR is broken.

Not necessarily so. Kindly explain under precisely which circumstance(s)
"reply-to-all" should *not* be a superset of "reply-to-author" and why.

> When I send a message to [email-masked], I do _not_
> want followups to be sent to the original reply address. That's why my
> message says Mail-Followup-To: [email-masked].

There is no "mail-followup-to" -- it is defined in no RFC and in
no Internet draft.  If one wants replies sent to a specific list
of addresses, that is precisely what the standard Reply-To header
field is for.

>>as far as I can tell, this is a plus for NR.
> 
> 
> Then you are fundamentally confused about what mail users want to do.

Anybody who apparently thinks that mail users wish to receive strident
spam-like "you didn't honor "mail-followup-to'" cruft is fundamentally
confused about what mail users want and don't want.

Date: 29 Aug 2004 21:03:42 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Simon Josefsson writes:
> The neat thing with NoReply is that recipients aren't required to be
> aware of the specification

It's certainly a good idea to avoid unnecessary upgrades.

In this case, however, the behavior of ancient MUAs (namely, always
including the reply recipient list as part of the followup recipient
list) is fundamentally incompatible with what mail users want.

This is why I rejected pure sending-side solutions when I was designing
Mail-Followup-To several years ago. This is also why quite a few MUAs
have, in fact, been upgraded to do a better job---in particular, to
support Mail-Followup-To.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sun, 29 Aug 2004 17:03:07 -0400
From: Tony Hansen <[email-masked]>
CC: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Whether parsing mboxcl2 takes "much" more code than parsing mboxrd is a 
subjective measurement. Having written both types of parsers, I don't 
feel the additional code is significant at all.

The last time I wrote an mbox parser, I wrote it to expect both formats. 
If I found a content-length header in a message, I'd skip that many 
bytes. I would then look for the next From_ header. (If there were no 
content-length header, I just looked for the next From_ header.) This 
algorithm seemed to work extremely well.

	Tony Hansen
	[email-masked]

D. J. Bernstein wrote:

> Charles Lindsey writes:
> 
>>you find it has been corrupted by all the '>' stuffing
> 
> 
> Certainly the old mboxo format should be nuked---but there are other
> mbox variants that don't corrupt messages.
> 
> The mboxrd format inserts ">" in front of any line beginning "From ",
> ">From ", ">>From", etc., and then inserts a "From " line at the top.
> This is trivially reversible. Readers recognize "From " as the start of
> a new message; readers strip ">" from ">From ", ">>From ", etc.
> 
> The competition is mboxcl2, which has Content-Length and doesn't have >
> quoting. There are two big disadvantages to mboxcl2. First, parsing
> mboxcl2 takes much more code than parsing mboxrd. Second, if you write
> mboxcl2 and the client expects mboxrd, you're risking having one message
> split into two, which is much worse than the maximum damage from writing
> mboxrd.
> 
> ---D. J. Bernstein, Associate Professor, Department of Mathematics,
> Statistics, and Computer Science, University of Illinois at Chicago
> 

Date: 29 Aug 2004 20:51:57 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: the most obvious failure in To-NoReply
Sender: [email-masked]

Keith Moore writes:
> NR has no way to specify situations where "reply to all" is not sent
> to a superset of the recipients who would be sent if "reply to author"
> were used instead.

Then NR is broken. When I send a message to [email-masked], I do _not_
want followups to be sent to the original reply address. That's why my
message says Mail-Followup-To: [email-masked].

(The Mail-Followup-To is inserted automatically, of course. The only
manual effort was a long time ago, putting [email-masked] into
$HOME/.lists and setting QMAILMFTFILE to $HOME/.lists.)

> as far as I can tell, this is a plus for NR.

Then you are fundamentally confused about what mail users want to do.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 29 Aug 2004 20:38:41 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Charles Lindsey writes:
> you find it has been corrupted by all the '>' stuffing

Certainly the old mboxo format should be nuked---but there are other
mbox variants that don't corrupt messages.

The mboxrd format inserts ">" in front of any line beginning "From ",
">From ", ">>From", etc., and then inserts a "From " line at the top.
This is trivially reversible. Readers recognize "From " as the start of
a new message; readers strip ">" from ">From ", ">>From ", etc.

The competition is mboxcl2, which has Content-Length and doesn't have >
quoting. There are two big disadvantages to mboxcl2. First, parsing
mboxcl2 takes much more code than parsing mboxrd. Second, if you write
mboxcl2 and the client expects mboxrd, you're risking having one message
split into two, which is much worse than the maximum damage from writing
mboxrd.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sun, 29 Aug 2004 16:33:24 -0400
From: Tony Hansen <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Nathaniel Borenstein <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, [email-masked], Charles Lindsey <[email-masked]>, Philip Hazel <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

Nothing is broken. Sendmail can be invoked using "sendmail addresses" 
just as easily it can be invoked using "sendmail -t". When your program 
that uses sendmail is upgraded to recognize the NoReply fields, continue 
to use "sendmail -t" for the To/Cc/Bcc recipients, and then use 
"sendmail addresses" for sending the message to the NoReply recipients.

I hope "sendmail -t" is NEVER upgraded to support NoReply headers. That 
would cause the worst of all possible worlds. As a "sendmail -t" user, 
you'd never know if your invocation of "sendmail -t" included those 
NoReply headers or not. Say your "sendmail -t" doesn't support NoReply 
and you write you code to invoke "sendmail addresses" to handle NoReply 
headers. Then some helpful admin upgrades your "sendmail -t" to support 
them. Suddenly the NoReply recipients will be receiving TWO copies of 
the messages.

If sendmail WERE to be upgraded to support NoReply headers, then it 
should be done using a DIFFERENT option. (For the version of sendmail on 
my system, it looks like -T might be available.)

	Tony Hansen
	[email-masked]

Keith Moore wrote:

> 
> Nathaniel Borenstein wrote:
> 
>> On Aug 27, 2004, at 4:51 AM, Arnt Gulbrandsen wrote:
>>
>>> Philip Hazel writes:
>>>
>>>> I guess I'm an "other". Not only MUAs, but also other software that 
>>>> wants to send email uses "-t". I suspect there are more applications 
>>>> that do this that I/you/we think.
>>>
>>>
>>>
>>> Right. Personally, I've used it more times than I can remember, in 
>>> situations ranging from little throwaway scripts up to parts of 
>>> billing systems. Am I so unusual? It is the easiest way I know to 
>>> generate a valid message with a custom From field on unix.
>>>
>>> It wouldn't surprise me if many thousand automated mail senders use it.
>>
>>
>>
>> I wouldn't be surprised if I had personally written a thousand shell 
>> scripts that use it.  It wouldn't surprise me if quite a few of them 
>> are still in use at sites I know nothing about.  Breaking "sendmail 
>> -t" strikes me as about as good an idea as changing the semantics of 
>> asterisks in shell scripts.  -- Nathaniel
> 
> 
> note that it would only break the programs that somehow started using 
> ReplyTo fields before sendmail got upgraded to support ReplyTo fields.
> 
> 
> 

To: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sun, 29 Aug 2004 13:26:57 -0700
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> here's something new to argue about :)
> I'm curious to know what people think of this idea.

Very minor point, but one that's been bugging me...  Existing mail header
fields are all either single words or use hyphens to separate words.  I'd
really rather not see ugly StudlyCaps introduced in a new standardized
field, particularly since fieldnames are case-insensitive and NoReply is
at least partly reliant on capitalization to be easily understood.
Cc-NorePly isn't easily parsable by my eye.

How about CC-No-Reply?  Cc-No-rePly still makes sense.  :)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sun, 29 Aug 2004 14:42:51 -0400
From: Keith Moore <[email-masked]>
To: Steve Dorner <[email-masked]>
Cc: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Steve Dorner wrote:
> 
> Why define two fields for this?  I don't understand the difference 
> between To-NoReply and Cc-NoReply.    All these recipients seem like 
> Cc-NoReply to me; they're people who get the message but are not 
> expected to be primary participants.

I suspect that just having CC-NoReply would reap about 99% of the value 
of having both To-NoReply and Cc-NoReply.  But as far as I can tell, 
whether you want a recipient to get replied on a message is more-or-less 
orthogonal to whether the recipient is a "primary recipient" (probably 
meaning, someone who is expected to act on the message) or not.

Keith

Date: Sun, 29 Aug 2004 13:13:50 -0400
From: Keith Moore <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, [email-masked], Charles Lindsey <[email-masked]>, Philip Hazel <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

Nathaniel Borenstein wrote:
> On Aug 27, 2004, at 4:51 AM, Arnt Gulbrandsen wrote:
> 
>> Philip Hazel writes:
>>
>>> I guess I'm an "other". Not only MUAs, but also other software that 
>>> wants to send email uses "-t". I suspect there are more applications 
>>> that do this that I/you/we think.
>>
>>
>> Right. Personally, I've used it more times than I can remember, in 
>> situations ranging from little throwaway scripts up to parts of 
>> billing systems. Am I so unusual? It is the easiest way I know to 
>> generate a valid message with a custom From field on unix.
>>
>> It wouldn't surprise me if many thousand automated mail senders use it.
> 
> 
> I wouldn't be surprised if I had personally written a thousand shell 
> scripts that use it.  It wouldn't surprise me if quite a few of them are 
> still in use at sites I know nothing about.  Breaking "sendmail -t" 
> strikes me as about as good an idea as changing the semantics of 
> asterisks in shell scripts.  -- Nathaniel

note that it would only break the programs that somehow started using 
ReplyTo fields before sendmail got upgraded to support ReplyTo fields.

Cc: Keith Moore <[email-masked]>, [email-masked], Charles Lindsey <[email-masked]>, Philip Hazel <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Sat, 28 Aug 2004 12:08:31 -0400
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Aug 27, 2004, at 4:51 AM, Arnt Gulbrandsen wrote:

> Philip Hazel writes:
>> I guess I'm an "other". Not only MUAs, but also other software that 
>> wants to send email uses "-t". I suspect there are more applications 
>> that do this that I/you/we think.
>
> Right. Personally, I've used it more times than I can remember, in 
> situations ranging from little throwaway scripts up to parts of 
> billing systems. Am I so unusual? It is the easiest way I know to 
> generate a valid message with a custom From field on unix.
>
> It wouldn't surprise me if many thousand automated mail senders use it.

I wouldn't be surprised if I had personally written a thousand shell 
scripts that use it.  It wouldn't surprise me if quite a few of them 
are still in use at sites I know nothing about.  Breaking "sendmail -t" 
strikes me as about as good an idea as changing the semantics of 
asterisks in shell scripts.  -- Nathaniel

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Sat, 28 Aug 2004 11:52:12 -0400
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

I can't resist a question and a wisecrack:

On Aug 24, 2004, at 9:14 AM, Charles Lindsey wrote:
>
> We are talking about a new application type, which we presume some 
> people
> will write new software to generate/accept (otherwise there is little
> point in having the application type).

Do we really want people to write new software to *generate* mbox 
format data?  I was thinking of this as primarily a tool for allowing 
software to label and read data that is only available in mbox format.  
Is there any circumstance in which we would actually encourage its 
generation?

On Aug 27, 2004, at 2:47 PM, Keith Moore wrote:

>>> Simple replace any line in the body starting with "From " with
>>> "From=20" and ensure that the CTE is quoted-printable (well, there 
>>> is a
>>> bit more to it than that).
>>
>> This was suggested over a decade ago when MIME was first developed.
>> The popularity of quoted-printable proved to be unsufficient.
>
> I suspect that circumstances are a bit different now.

Oh yes, nowadays everybody *loves* quoted-printable.  :-)  -- Nathaniel

From: Bruce Lilly <[email-masked]>
Subject: internationalization of mail
To: [email-masked]
Cc: Tex Texin <[email-masked]>
Reply-To: [email-masked]
Date: Sat, 28 Aug 2004 13:40:18 -0500
Sender: [email-masked]

Bear in mind RFC 1958 section 3.  In particular, some
handheld devices (such as the one this message is
being composed on) might not have sufficient
resources to support some of the more recent
Unicode implementations.  Aside from font issues,
Unicode normalization uses huge tables which might
not be practical for some devices which do support
email.  Therefore transcoding a space- and
memory-efficient charset into Unicode may well
render the transformed message unusable or
unreplyable if transferred (forwarded, etc.) to
some devices [it is not unusual for mail to be forwarded to PDAs, mobile
 phones, pagers, etc.].

Cc: Steve Dorner <[email-masked]>, [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
To: [email-masked]
Date: Sat, 28 Aug 2004 11:23:58 +0100
From: Nick Ing-Simmons <[email-masked]>
Reply-To: Nick Ing-Simmons <[email-masked]>
Sender: [email-masked]

<[email-masked]> writes:
>> Why define two fields for this?  I don't understand the difference
>> between To-NoReply and Cc-NoReply.    All these recipients seem like
>> Cc-NoReply to me; they're people who get the message but are not
>> expected to be primary participants.
>
>This didn't occur to me when I first read the draft, but now that it has
>been pointed out I have to say I don't see much point in having two
>separate fields.

It smells of 
FYI: 
to me ;-)

Date: Fri, 27 Aug 2004 22:53:22 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Keith Moore wrote:

> The biggest potential for unmet expectations that I see is for users of
> legacy MUAs that (a) use "sendmail -t" and (b) allow message authors to 
> specify arbitrary headers.  The potential exists for those users to
> specify NoReply recipients who will not receive the message.  The 
> document doesn't currently recommend that "sendmail" and similar 
> programs be upgraded to support NoReply fields, but that is probably
> what should happen.  Assuming that all "sendmail" and similar programs
> would eventually be upgraded, this would further minimize the potential
> for unmet expectations.

I feel that worries about MUAs using "sendmail -t" by existing MUAs is a 
red herring.

If those MUAs were to be enhanced to support sending messages with 
To/CC-No-Reply headers, they would be able to send messages to those 
individuals by invoking sendmail a second time and providing those 
addresses on the sendmail command line. No change to sendmail is 
required -- just changes to the MUA.

	Tony Hansen
	[email-masked]

Date: Sat, 28 Aug 2004 09:55:18 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: internationalization of mail
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 27 2004, Arnt Gulbrandsen wrote:

> http://odur.let.rug.nl/~vannoord/TextCat/ is probably what you're 
> thinking of. At least I've never seen anything better.

This looks like it, although I had remembered it as a C library, not Perl.
But I must have been mistaken. The above web page also has a link to 
competitors, so it's a good place to refer. 

> Asmus Freytag spoke about it at IUC14, as I recall. He used character 
> pair frequencies and wrote code that was fairly reliable after about 18 
> characters. His code lost accuracy again after about 44 characters, for 

There's a common phenomenon with machine learning algorithms which
states that as you learn more and more data, first generalization
accuracy improves but if you keep learning forever, then
generalization accuracy worsens again. 

(An analogy would be learning about cats: first you learn that they
have four legs and so you are able to distinguish cats from
birds. Success! Then you learn that cats have a long tail, and you can
distinguish them from pigs. Success! Then you learn that they have
fur, and when somebody shows you a hairless cat, you fail to recognize
it).

> Yes, I did, because that's where I've run into problems. Identifying the 
> language's not so hard, but pretty soon a name like Danuta Hübner will 
> crop up and what do you do then. (She's a Polish politican, and ü is 
> not a letter one often sees in Polish.)

Good example. 

-- 
Laird Breyer.

Date: Fri, 27 Aug 2004 14:14:34 -0700 (PDT)
From: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
To: Steve Dorner <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Why define two fields for this?  I don't understand the difference
> between To-NoReply and Cc-NoReply.    All these recipients seem like
> Cc-NoReply to me; they're people who get the message but are not
> expected to be primary participants.

This didn't occur to me when I first read the draft, but now that it has
been pointed out I have to say I don't see much point in having two
separate fields.

				Ned

Date: Fri, 27 Aug 2004 14:02:02 -0700
To: [email-masked]
From: Steve Dorner <[email-masked]>
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Why define two fields for this?  I don't understand the difference 
between To-NoReply and Cc-NoReply.    All these recipients seem like 
Cc-NoReply to me; they're people who get the message but are not 
expected to be primary participants.

To: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 27 Aug 2004 13:55:09 -0700
Sender: [email-masked]

Arnt Gulbrandsen <[email-masked]> writes:

> Right. Personally, I've used it more times than I can remember, in
> situations ranging from little throwaway scripts up to parts of billing
> systems. Am I so unusual? It is the easiest way I know to generate a
> valid message with a custom From field on unix.

> It wouldn't surprise me if many thousand automated mail senders use it.

You're not unusual at all.  We probably have at least one hundred
different processes that use it in one place or another, many hidden deep
inside other code.  It is the standard way to send mail for any Unix
process and is supported by essentially every Unix MTA.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 27 Aug 2004 14:47:35 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

> > Simple replace any line in the body starting with "From " with
> > "From=20" and ensure that the CTE is quoted-printable (well, there is a
> > bit more to it than that).
> 
> This was suggested over a decade ago when MIME was first developed.
> The popularity of quoted-printable proved to be unsufficient.

I suspect that circumstances are a bit different now.

Keith

Date: Fri, 27 Aug 2004 11:26:35 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> Keith Moore <[email-masked]> writes:

> >now if I were to write a recommendation for how to write mbox files,
> >I'd say that all body parts MUST be encoded if necessary to prevent
> >the characters "From" from appearing at the beginning of a line.
> >I'd also say that Content-Length MUST NOT be written, that From_ dates
> >have to be in exactly such-and-such a format, that Return-Path fields
> >are REQUIRED when storing a message received by SMTP, even if they're
> >redundant with From_ lines, etc.  but that's not the problem we're
> >currently trying to solve.

> Which suggests an alternative to the usual, irreversible, >-stuffing
> convention.

Actually, it reverses pretty well if you do it correctly.

> Simple replace any line in the body starting with "From " with
> "From=20" and ensure that the CTE is quoted-printable (well, there is a
> bit more to it than that).

This was suggested over a decade ago when MIME was first developed.
The popularity of quoted-printable proved to be unsufficient.

				Ned

Date: Fri, 27 Aug 2004 14:01:17 -0400
From: Keith Moore <[email-masked]>
To: Steve Dorner <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

> I'm having trouble seeing significant benefits.  What problem would 
> this proposal solve?  The justifications in the draft seem to me to 
> be a little ... esoteric.  I really wonder if defining new header 
> fields and MTA behaviors is justified.

it's a valid question.

> The primary desire is to reduce the number of messages some 
> recipients receive.  

not only to reduce network bandwidth consumption and the number
of messages in those people's inboxes, but (more broadly) to 
make email a more effective tool for discussions with more than 
two participants.  the more people who appear to be involved
in a conversation, the harder it is to get closure.

> The situations given seem mostly to be about informing some folks 
> that the discussion is going to continue amongst other folks, and 
> that the folks of the first part needn't worry their pretty heads 
> about the matter anymore.
> 
> This could be accomplished by simply forwarding a copy of the reply 
> to the folks who need to be informed.

True.  Though by a similar argument, we don't need the ability to 
have multiple recipients in the To or Cc fields either.  In fact,
authors find it useful to be able to send only a single message,
and recipients find it useful to know who else has received the
same message.

> I'd need to be made to understand many more benefits to this proposal 
> before I'd consider adding support for it.  I'd be surprised if other 
> MUA authors didn't feel the same way.

it's a reasonable request, and I'll work on coming up with more 
justification for the next revision.

thanks,

Keith

Date: Fri, 27 Aug 2004 10:25:17 -0700 (PDT)
From: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
To: Keith Moore <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> > > So with Reply-To as it is generally implemented, there's no way for
> > > an author to include a recipient in a To or CC field without that
> > > recipient getting replies to the message.

> > Sure there is; *only* "reply-to-all" goes to original message
> > To and Cc recipients -- normal replies only go to the address(es)
> > specified by the original message author in Reply-To.

> for many people, "reply to all" is "normal".  "reply to author" is the
> special case.  at any rate, this is a recipient decision, not one made
> by a message author.

That's certainly the case for me; reply-to-all is what I typically do.

> > > The biggest potential for unmet expectations that I see is for users of
> > > legacy MUAs that (a) use "sendmail -t" and (b) allow message authors to
> > > specify arbitrary headers.  The potential exists for those users to
> > > specify NoReply recipients who will not receive the message.  The
> > > document doesn't currently recommend that "sendmail" and similar
> > > programs be upgraded to support NoReply fields, but that is probably
> > > what should happen.  Assuming that all "sendmail" and similar programs
> > > would eventually be upgraded, this would further minimize the potential
> > > for unmet expectations.

> > Such "upgrading" should probably involve a separate argument to handle
> > the NoReply fields, one that is currently unused,

> well, the legacy submission program would have to fail if it were given
> the currently unused argument, so that the new MUA could test to see
> whether that feature were supported by the submission program.  given
> that this kind of submission interface isn't standardized, and that
> there are multiple programs which implement something like this
> interface, I think it would be difficult to rely on such a test.  also,
> if the MUA is going to be modified to perform such a test, it seems like
> it would be almost as easy, and much more reliable, to modify the MUA to
> just extract the recipient list and supply that on the command line.
> (granted, it would have to do Bcc processing also).

Capabilities negotiation is the killer in this situations. When ESMTP was firt
proposed there was a tremendous amount of angst about how things like
per-recipient parameters could be dealt with in legacy submission interfaces.
It was eventually decided that having one place to do this was both necessary
and sufficient and that the upgrade path would be to abandon less capable
interfaces and move to SMTP. This eventually became one of the reasons SMTP
SUBMIT was developed.

				Ned

Date: Fri, 27 Aug 2004 10:08:10 -0700
To: John C Klensin <[email-masked]>
From: Steve Dorner <[email-masked]>
Subject: Re: draft-moore-mail-nr-fields-00
Cc: [email-masked]
Sender: [email-masked]

At 3:55 PM -0400 8/26/04, John C Klensin wrote:
>unless it is really clear that there would be significant benefits.

I'm having trouble seeing significant benefits.  What problem would 
this proposal solve?  The justifications in the draft seem to me to 
be a little ... esoteric.  I really wonder if defining new header 
fields and MTA behaviors is justified.

The primary desire is to reduce the number of messages some 
recipients receive.  In order to accomplish this, the sender of a 
message puts some of those recipients in special header fields.

The situations given seem mostly to be about informing some folks 
that the discussion is going to continue amongst other folks, and 
that the folks of the first part needn't worry their pretty heads 
about the matter anymore.

This could be accomplished by simply forwarding a copy of the reply 
to the folks who need to be informed.  Yeah, this is probably a tiny 
bit more work for the sender than putting their addresses in a 
separate field.  But it's really not so hard and it doesn't require 
MUA's to add special support for generation.

I'd need to be made to understand many more benefits to this proposal 
before I'd consider adding support for it.  I'd be surprised if other 
MUA authors didn't feel the same way.

Date: Fri, 27 Aug 2004 11:56:09 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

>>So with Reply-To as it is generally implemented, there's no way for
>>an author to include a recipient in a To or CC field without that
>>recipient getting replies to the message.
> 
> 
> Sure there is; *only* "reply-to-all" goes to original message
> To and Cc recipients -- normal replies only go to the address(es)
> specified by the original message author in Reply-To.

for many people, "reply to all" is "normal".  "reply to author" is the 
special case.  at any rate, this is a recipient decision, not one made 
by a message author.

>>The biggest potential for unmet expectations that I see is for users of
>>legacy MUAs that (a) use "sendmail -t" and (b) allow message authors to 
>>specify arbitrary headers.  The potential exists for those users to
>>specify NoReply recipients who will not receive the message.  The 
>>document doesn't currently recommend that "sendmail" and similar 
>>programs be upgraded to support NoReply fields, but that is probably
>>what should happen.  Assuming that all "sendmail" and similar programs
>>would eventually be upgraded, this would further minimize the potential
>>for unmet expectations.
> 
> 
> Such "upgrading" should probably involve a separate argument to handle
> the NoReply fields, one that is currently unused,

well, the legacy submission program would have to fail if it were given
the currently unused argument, so that the new MUA could test to see 
whether that feature were supported by the submission program.  given 
that this kind of submission interface isn't standardized, and that 
there are multiple programs which implement something like this 
interface, I think it would be difficult to rely on such a test.  also, 
if the MUA is going to be modified to perform such a test, it seems like 
it would be almost as easy, and much more reliable, to modify the MUA to 
just extract the recipient list and supply that on the command line. 
(granted, it would have to do Bcc processing also).

> I think John is mostly right about existing use of Reply-To/To/Cc.
> Certainly the two examples in the draft's introduction can be handled
> readily.  In the case of the first example, the "handshake" message
> would have
>   To: C
>   From: B
>   Cc: A
>   Reply-To: A
> when delegating followup to C. C's followup goes to A alone and
> any further correspondence is between A  and C unless one or the other
> chooses to manually include B or other recipients.

that's a good point - if we can solve the problem with existing fields
then the example is not a justification for new fields.  so I'll need to 
come up with a better example :)

(many years ago I wrote down a list of every problem I could identify 
with "reply" given the current header definitions.  I haven't yet taken 
the trouble to look at that list in the context of the current 
discussion, but it will be interesting to re-examine that list with an 
eye to how NR fields might or might not help with those problems)

> The second example more vividly demonstrates problems with the NoReply
> field in the scenario posited.  With existing standard fields, at some
> point either D or E realizes that they have gone off track and simply
> drops Ccs to the list (or if neither does so, the list may be annoyed,
> with or without NoReply fields).  However, use of NoReply fields
> requires either E or D to possess prescience about drifting off-topic;
> while that may be predictable in the case of some specific instances
> of a D or E, it is in general unrealistic to expect such predictability.

I don't think that's quite right.  All that's really needed is for the 
author of a message to realize that certain recipients probably don't 
need to receive replies to the message currently being composed.

It may be that the most likely cases are

a) for an author to realize that he doesn't want to receive replies to 
the message currently being composed.

(which is solved with "Reply-To: nobody:;")

and

b) a recipient to realize that he doesn't want to receive any more 
copies of messages to that thread

(a problem which NR fields do not address)

> I believe that there may be an omission -- there should probably be
> Resent- versions of the NoReply fields corresponding to the Resent-
> versions of the existing standard recipient fields.

interesting idea.

in general, thanks for the careful review and comments.

Keith

Date: Fri, 27 Aug 2004 11:42:16 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Keith Moore wrote:

> For better or worse, Reply-To has been nearly universally implemented
> by MUAs as:
> 
> When composing a reply to a message for which the Reply-To field
> exists, use the addresses in Reply-To as the addresses of the To
> field of the reply.  If the reply is a "reply to all", To and Cc 
> addresses from the subject message are also Cc'ed on the reply.
> 
> So with Reply-To as it is generally implemented, there's no way for
> an author to include a recipient in a To or CC field without that
> recipient getting replies to the message.

Sure there is; *only* "reply-to-all" goes to original message
To and Cc recipients -- normal replies only go to the address(es)
specified by the original message author in Reply-To.

> I thought this was clear from what I wrote, but if not, I can certainly
> add clarifying text.

I think removing obfuscating text is the answer; see below.

> The biggest potential for unmet expectations that I see is for users of
> legacy MUAs that (a) use "sendmail -t" and (b) allow message authors to 
> specify arbitrary headers.  The potential exists for those users to
> specify NoReply recipients who will not receive the message.  The 
> document doesn't currently recommend that "sendmail" and similar 
> programs be upgraded to support NoReply fields, but that is probably
> what should happen.  Assuming that all "sendmail" and similar programs
> would eventually be upgraded, this would further minimize the potential
> for unmet expectations.

Such "upgrading" should probably involve a separate argument to handle
the NoReply fields, one that is currently unused, so that an MUA can
determine whether gathering recipient addresses from NoReply fields
(for the envelope) is supported by a particular instance of such a
program.  Without at least some primitive method for "negotiating"
capabilities, there is potential for such issues of unmet
expectations.  Note that it is possible to specify additional
recipient addresses when invoking sendmail (with or w/o -t).

Comments on the draft:

I think John is mostly right about existing use of Reply-To/To/Cc.
Certainly the two examples in the draft's introduction can be handled
readily.  In the case of the first example, the "handshake" message
would have
  To: C
  From: B
  Cc: A
  Reply-To: A
when delegating followup to C. C's followup goes to A alone and
any further correspondence is between A  and C unless one or the other
chooses to manually include B or other recipients.

The second example more vividly demonstrates problems with the NoReply
field in the scenario posited.  With existing standard fields, at some
point either D or E realizes that they have gone off track and simply
drops Ccs to the list (or if neither does so, the list may be annoyed,
with or without NoReply fields).  However, use of NoReply fields
requires either E or D to possess prescience about drifting off-topic;
while that may be predictable in the case of some specific instances
of a D or E, it is in general unrealistic to expect such predictability.

Perhaps the author can come up with an example that illustrates
some capability that cannot be handled by the existing standard
fields.

I believe that there may be an omission -- there should probably be
Resent- versions of the NoReply fields corresponding to the Resent-
versions of the existing standard recipient fields.

Section 3.1 makes a statement about how fields are presented by
(all) legacy MUAs, however at least some present all header fields
unless directed to hide particular field names, so NoReply fields
would be presented by default.

As an aside, RFC 2047's successor should probably clarify handling
of encoded-words in unrecognized extension fields (now that RFC
2822 has removed the distinction between extension and user-defined
fields).

Minor nits (typos, usage, spelling, etc.):

General comment: both "[Aa]n MUA" and "a MUA" appear in the draft.
I would prefer consistent use of "an MUA"; I don't feel strongly
about "a" vs. "an", but consistency would be an improvement.

2.2.5:

IMO eliding the comma and remainder of the first sentence would
improve clarity.

3.3:

explicit -> explicitly
may not -> might not

3.4:

elide the first "will"

3.5:

address headers -> address header fields (or simply address fields)

5.1 last paragraph, first line:

NoReply field do -> NoReply fields do

Date: Fri, 27 Aug 2004 16:22:06 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: internationalization of mail
Sender: [email-masked]

Tex Texin writes:
> The thrust of my questions is to find out the percentage of mail 
> messages is incorrectly labeled or not labeled with an encoding, and 
> may therefore be risky to transcode.

I've never seen good statistics on anything like this, and I've read 
these lists for more than a decade. I assume the reason is that most 
mail is private, and users keep it private.

Anecotally, it seems that unlabelled mail is seen now and then. 
Mislabelled is unusual.

> Secondarily there are many products that support Unicode but not 
> necessarily the full (or nearly full) character repertoire.
> I need to understand whether moving to Unicode means some mail clients 
> won't support some languages.

It won't make much of a difference. Enough messages have been sent using 
unicode recently that most readers have been upgraded to accept it.

Don't worry so much about the character repertoire. That you can't use 
all of that repertoire doesn't matter, as long as the product can show 
that which the user can (and wants to) read.

Arnt

Date: Fri, 27 Aug 2004 09:04:34 -0400
From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

Simon Josefsson wrote:
> Keith Moore <[email-masked]> writes:
> 
> 
>>>If nobody can find client behaviors that MFT/MRT can achieve, that NR
>>>together with (a) cannot, especially wrt news (MFT/MRT is used in news
>>>too, IIRC), I'd like to see NR move forward.
>>
>>there are client behaviors that MFT/MRT can produce that NR cannot 
>>produce.  in particular, NR has no way to specify situations where
>>"reply to all" is not sent to a superset of the recipients who would be 
>>sent if "reply to author" were used instead.
>>
>>as far as I can tell, this is a plus for NR.
> 
> 
> Yes.  I should have added a "useful" before "client behaviors".  Since
> it is possible to generate some things with MFT/MRT that NR can't do,
> it isn't obvious that all of those MFT/MRT things are useless,
> although it appears so.  That need to be verified through more review.

I did a case analysis of NR vs. MFT.  The cases looked like this:

class 	field	reply to author		reply to all	superset
F1	From	n			n		y
F2	From	n			y		y
F3	From	y			n		n
F4	From	y			y		y
T1	To	n			n		y
T2	To	n			y		y
T3	To	y			n		n
T4	To	y			y		y
C1	Cc	n			n		y
C2	Cc	n			y		y
C3	Cc	y			n		n
C4	Cc	y			y		y

R2	-	n			y		y
R3	-	y			n		n
R4	-	y			y		y

each class except the R classes is a class of recipients.  The R classes 
are not recipients of the original message, but are intended recipients 
of replies.  the superset column is whether the reply to all recipients
are a superset of reply to author recipients.

I believe this is is a complete set of cases.

say you want to send a message to zero or more of each class of 
recipients.  here's how you'd do it:

NR:

From: F1, F2, F3 (1), F4
Reply-To: (2) F4, T4, C4, R2(1), R3 (1), R4
To: T2, T3 (1), T4
To-NoReply: T1
Cc: F2, C2, C3 (1), C4
Cc-NoReply: C1

MFT:
From: F1, F2, F3, F4
To: T1, T2, T3, T4
Cc: C1, C2, C3, C4
Mail-Followup-To: F2, F4, T2, T4, C2, C4, R2, R4
Mail-Reply-To: F3, F4, T3, T4, C3, C4, R3, R4

Notes:
1. classes that can't be dealt with properly using NR are F3, T3, C3, 
R2, and R3. The *3 classes can't be dealt with because NR fields only 
specify replyability in general, they don't specify replyability 
differently for reply to all and reply to author.  In the NR example 
above,  each of the *3 classes is treated as if it were *4. The R2 class 
cannot be dealt with because Reply-To (which is the only way to include 
recipients on replies who were not recipients of the original message) 
doesn't distinguish between reply to author and reply to all.
2. if all classes in the reply-to field are empty, specify this as 
Reply-To: nobody:;

MFT seems more straightforward when viewed from the point-of-view
of this case analysis, but this is partially because the cases were 
organized in such a way as to cover the union of what MFT and NR can do.

I see two questions that determine the relative merit of NR vs MFT:

1. how useful are cases F3, T3, C3, R2, and R3?

IMHO, R2 might be useful.  The others seem more likely to cause 
confusion than to provide behavior that an author finds useful.

2. how much potential is there for intended "NoReply" recipients to fail 
to recieve mail because "sendmail -t" doesn't recognize the new fields?

Keep in mind that this is only a problem when the new fields are used.
Presumably existing mail robots that generate messages and send them to
"sendmail -t" won't automagically start generating NoReply fields.
But for user agents that allow users to specify arbitrary header fields
and pass them off to "sendmail -t", this is a problem.

Also, I don't see that there's already a significant deployment of MFT, 
so I don't think that's much of a factor.

Keith

Date: Fri, 27 Aug 2004 14:24:49 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: internationalization of mail
Sender: [email-masked]

Laird Breyer writes:
> I understand your point, and I agree with you. What I did have in mind 
> was a system which uses sequences of several bytes to try to 
> characterize languages. (eg the sequence "qu" is more common in 
> French than in English). I once saw such a library on the web, but I 
> can't remember what it was called, or I would have linked to it :-(.

http://odur.let.rug.nl/~vannoord/TextCat/ is probably what you're 
thinking of. At least I've never seen anything better.

> From memory, the idea was to scan a few lines of text trying to match 
> those sequences, with language dependent weightings. This would give 
> the "most probable" language, etc.

Right. That page seems to link to various papers on the subject. Also, 
Asmus Freytag spoke about it at IUC14, as I recall. He used character 
pair frequencies and wrote code that was fairly reliable after about 18 
characters. His code lost accuracy again after about 44 characters, for 
some reason. (I think I have a copy of the talk at home, can look at 
it.)

> Possibly you know all this already, but your post seemed to me to 
> concentrate on single character issues.

Yes, I did, because that's where I've run into problems. Identifying the 
language's not so hard, but pretty soon a name like Danuta Hübner will 
crop up and what do you do then. (She's a Polish politican, and ü is 
not a letter one often sees in Polish.)

Arnt

Date: Fri, 27 Aug 2004 21:40:09 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: internationalization of mail
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 27 2004, Arnt Gulbrandsen wrote:

> There are. I've worked on two, myself. The problem with them all is that 
> many character sets are so similar. The values that are lower-case in 
> one is often lowercase in another, the ones that are illegal in one is 
> illegal in another.

 <snip>

I understand your point, and I agree with you. What I did have in mind
was a system which uses sequences of several bytes to try to
characterize languages. (eg the sequence "qu" is more common in French
than in English). I once saw such a library on the web, but I can't
remember what it was called, or I would have linked to it :-(.  From
memory, the idea was to scan a few lines of text trying to match those
sequences, with language dependent weightings. This would give the
"most probable" language, etc.

Possibly you know all this already, but your post seemed to me to
concentrate on single character issues.

But you are right overall, getting perfect results won't work in this
way, and in fact if the input text is smaller than a few lines, there
may not be enough data to run meaningful statistics at all.

-- 
Laird Breyer.

Date: Fri, 27 Aug 2004 12:57:27 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: internationalization of mail
Sender: [email-masked]

Laird Breyer writes, answering Tex Texin:
> Have you considered using a statistical character set detector? A 
> quick google search reveals http://trific.ath.cx/software/enca/, but 
> if the license isn't acceptable, there are probably others out there.

There are. I've worked on two, myself. The problem with them all is that 
many character sets are so similar. The values that are lower-case in 
one is often lowercase in another, the ones that are illegal in one is 
illegal in another.

Look at this:

$ grep 0xC4 8859-{?,??}.TXT
8859-1.TXT:0xC4 0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-2.TXT:0xC4 0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-3.TXT:0xC4 0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-4.TXT:0xC4 0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-5.TXT:0xC4 0x0424  #       CYRILLIC CAPITAL LETTER EF
8859-6.TXT:0xC4 0x0624  #       ARABIC LETTER WAW WITH HAMZA ABOVE
8859-7.TXT:0xC4 0x0394  #       GREEK CAPITAL LETTER DELTA
8859-9.TXT:0xC4 0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-10.TXT:0xC4        0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-11.TXT:0xC4        0x0E24  #       THAI CHARACTER RU
8859-13.TXT:0xC4        0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-14.TXT:0xC4        0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-15.TXT:0xC4        0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS
8859-16.TXT:0xC4        0x00C4  #       LATIN CAPITAL LETTER A WITH DIAERESIS

(I picked C4 at random.)

One hand, it means that if you choose the wrong Latin-* encoding, the 
error often doesn't matter, because C4 is the same letter in all the 
Latin-* encodings. On the other, it means that if the text contains a 
character which differs among the sibling encodings, the text is 
subtly, almost undetectably, altered.

$ grep 0xF9 8859-{?,??}.TXT
8859-1.TXT:0xF9 0x00F9  #       LATIN SMALL LETTER U WITH GRAVE
8859-2.TXT:0xF9 0x016F  #       LATIN SMALL LETTER U WITH RING ABOVE
8859-3.TXT:0xF9 0x00F9  #       LATIN SMALL LETTER U WITH GRAVE
8859-4.TXT:0xF9 0x0173  #       LATIN SMALL LETTER U WITH OGONEK
8859-5.TXT:0xF9 0x0459  #       CYRILLIC SMALL LETTER LJE
8859-7.TXT:0xF9 0x03C9  #       GREEK SMALL LETTER OMEGA
8859-8.TXT:0xF9 0x05E9  #       HEBREW LETTER SHIN
8859-9.TXT:0xF9 0x00F9  #       LATIN SMALL LETTER U WITH GRAVE
8859-10.TXT:0xF9        0x0173  #       LATIN SMALL LETTER U WITH OGONEK
8859-11.TXT:0xF9        0x0E59  #       THAI DIGIT NINE
8859-13.TXT:0xF9        0x0142  #       LATIN SMALL LETTER L WITH STROKE
8859-14.TXT:0xF9        0x00F9  #       LATIN SMALL LETTER U WITH GRAVE
8859-15.TXT:0xF9        0x00F9  #       LATIN SMALL LETTER U WITH GRAVE
8859-16.TXT:0xF9        0x00F9  #       LATIN SMALL LETTER U WITH GRAVE

So for latin text, U-grave can be substituted for U-stroke. Such changes 
have a good chance of passing a first examination undetected and being 
archived as correct. Rather complicates searching, of course.

In some cases, an error which is clearly obvious at once is preferable 
to one which may sneak by undetected. I don't know whether this applies 
to Tex's problem.

> This would permit high confidence decisions about when to convert an 
> incoming message, although after conversion the problems you 
> described would still exist, if the detection was incorrect.

Indeed. There might be fewer errors remaining (depending on this and 
that), but they would be harder to find and more difficult to handle.

Arnt

Date: Fri, 27 Aug 2004 20:13:29 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: internationalization of mail
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 27 2004, Tex Texin wrote:

> If a Thai font is used, expecting data in a Thai encoding (TIS
> 620-2533 for example) it may not matter that the data is mislabeled
> as long as it is unchanged it will display the right character for
> each byte. But if the Thai data has gone thru a conversion from
> 8859-1 to utf-8, the bytes will no longer show the correct Thai
> characters, so for this situation the solution of simply changing
> fonts no longer works. 

Have you considered using a statistical character set detector?
A quick google search reveals http://trific.ath.cx/software/enca/,
but if the license isn't acceptable, there are probably others out there.

This would permit high confidence decisions about when to convert an
incoming message, although after conversion the problems you described
would still exist, if the detection was incorrect.

-- 
Laird Breyer.

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Fri, 27 Aug 2004 11:52:52 +0200
Lines: 30
Cancel-Lock: sha1:dzTR8qWtMmSpulC2gj0U9n16TKE=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> If nobody can find client behaviors that MFT/MRT can achieve, that NR
>> together with (a) cannot, especially wrt news (MFT/MRT is used in news
>> too, IIRC), I'd like to see NR move forward.
>
> there are client behaviors that MFT/MRT can produce that NR cannot 
> produce.  in particular, NR has no way to specify situations where
> "reply to all" is not sent to a superset of the recipients who would be 
> sent if "reply to author" were used instead.
>
> as far as I can tell, this is a plus for NR.

Yes.  I should have added a "useful" before "client behaviors".  Since
it is possible to generate some things with MFT/MRT that NR can't do,
it isn't obvious that all of those MFT/MRT things are useless,
although it appears so.  That need to be verified through more review.

But I didn't realize receivers needed user interface changes, that
seem like a disadvantage.  Then perhaps the cost benefit of NR over
MFT/MRT is more questionable.  And having two deployed solutions for a
similar thing will be somewhat confusing, although I don't see any
conflicts in supporting both ideas in the same agent.  OTOH, if
receivers aren't changed, those additional copies will look as BCC's,
which the user should be aware of anyway.

Still, I'd like to see _some_ solution to this problem move forward.

Thanks,
Simon

Date: Fri, 27 Aug 2004 02:46:24 -0700
From: Katsuhiko Momoi <[email-masked]>
To: [email-masked]
CC: Tex Texin <[email-masked]>
Subject: Re: internationalization of mail
Sender: [email-masked]

Hi, Tex,

This issue has may dimensions. In this message, I want to address just 
one of the questions you asked.

Tex Texin wrote:

>Kat,
>
>thanks. I agree with you about using Unicode internally. Where the encoding of
>the message is correctly labeled so it can be accurately transcoded it is a
>good solution. The thrust of my questions is to find out the percentage of mail
>messages is incorrectly labeled or not labeled with an encoding, and may
>therefore be risky to transcode.
>
>Secondarily there are many products that support Unicode but not necessarily
>the full (or nearly full) character repertoire.
>I need to understand whether moving to Unicode means some mail clients won't
>support some languages.
>As you know many products today don't fully support a variety of languages
>properly, but users can still use them by using a native font and ignoring
>encoding declarations. For example, displaying Thai with a Thai font even
>though the product thinks it is using ISO 8859-1.
>
>However, if the text is transcoded to UTF-8 based on an incorrect encoding,
>then it won't be displayable by treating the utf-8 bytes as Thai characters.
>
>So, can I take it as true that for NS mail, the transcoding to Unicode didn't
>introduce significant problems and most mail was rendered effectively?
>  
>
As far as I know, we haven't heard too many complaints from straight 
Mail users even from those using Simplified Chinese or Russian.  Most 
complaints are from news readers. They complain that languages like 
Russian and Chinese are often mislabeled with incorrect charset names. 
This is presumably because in news, there are many small/personal 
programs for news reading. For this sort of case, we devised a 
per-folder charset override control. In Mozilla, you can set the default 
encoding for each mail folder separately as one of the folder 
properties. You can also in addition select an option to override all 
MIME labeling with this default charset for that folder. I understand 
that this works well in this type of cases in newsgroups.

In the latest version of Mozilla/Netscape7.2, you can also set the 
mail-send encoding to override the original encoding of the message 
you're replying to. So for example, (assuming that you have set both the 
folder override and mail-send charset override),

1) You receive a msg in GB2312 but with an incorrect charset label of 
ISO-8859-1.
2) But the folder override is on and this means regardless of MIME 
charset info in the msg,  this msg will be displayed as GB2312 in this 
particular folder. So far, so good.
3) Now you want to send a reply to this msg. Instead of honoring the 
original charset (ISO-8859-1), selecting an override option for the send 
charset I mentioned above will reply to this in GB2312 thereby 
correcting the incorrect encoding.

These 2 overriding mechanisms help a lot. I think  this type of idea is 
extensible to other mail systems. A bit more complicated case is where 
news articles are sent as mail msg to mailboxes.

- Kat

>tex
>
>
>Katsuhiko Momoi wrote:
>  
>
>>Tex Texin wrote:
>>
>>    
>>
>>>Hi Kat!
>>>It's been a while...
>>>
>>>
>>>      
>>>
>>Hi, Tex,
>>
>>Good to see you here.
>>
>>    
>>
>>>thanks for the mail, it was helpful.
>>>
>>>Correct me if I am wrong, but in the Mozilla environment the mail is treated as
>>>a bucket of bytes that just gets passed around.
>>>So for example if one subject hdr is in iso 2022-jp and another is in
>>>shift-jis, you can change the display to show and interpret one or the other
>>>correctly, but not both at the same time. For purposes of search and filtering,
>>>I was thinking of converting to unicode, and then all subjects would display
>>>correctly, as well as be searchable.
>>>
>>>
>>>      
>>>
>>Not really. The received msgs are stored as is but as each msg is sent
>>to rendering, it is converted to utf-8 before it gets to rendering. So
>>in Mozilla mail, one can display headers of all supported encodings at
>>once in the header viewing pane. I had a demo I used at one of the
>>Unicode conferences which showed mail headers in 15 or so languages. But
>>the requirement for this is that the each header must be marked with
>>proper MIME encoding info. This means Mozilla can display shift_jis and
>>iso-2022-jp headers at the same time as long they use MIME encoding. If
>>a header of a msg does not use MIME encoding, then Mozilla will use the
>>user's default msg display encoding.
>>
>>Sorting and filtering for Mozilla are all done in Unicode as well. For
>>more info, you can look at our Unicode presentation (by Naoki Hotta and
>>I) here:
>>
>>http://wp.netscape.com/eng/intl/docs/iuc17/mail/iuc17mail.html
>>
>>and in particular at this diagram showing the data flow for MIME header
>>and body with associated MIME encoding.
>>
>>http://wp.netscape.com/eng/intl/docs/iuc17/mail/slide05.html
>>
>>If I may add my 2 cents,  if one is designing a webmail system, why not
>>use Unicode for all internal processing? (while keeping the original
>>data as is in case you need to refer to them).  Most search data are
>>handled in Unicode anyway now and this will increase compatibility with
>>the search functions you might want to offer in a webmail service. I
>>would even go as far as to say that we may want to unify send encoding
>>of webmail systems to just UTF-8 unless only ASCII data is being sent
>>out. I think Gmail is currently doing just that. In the absence of
>>country specific standards, this is one way to improve the situation.
>>Most popular mail programs can handle UTF-8. If some mailers cannot,
>>then I think it is time to update the code! I think this standardization
>>can happen first in webmail -- where simplification is acceptable --
>>ahead of standalone mailers, which may face harder hurdles in unifying
>>to UTF-8.
>>
>>- Kat
>>
>>    
>>
>>>But it may not be worthwhile if many of the mails are incorrectly identified so
>>>transcoding to utf-8 generates muck, or errors out.
>>>
>>>tex
>>>
>>>Katsuhiko Momoi wrote:
>>>
>>>
>>>      
>>>
>>>>Tex Texin wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>Hi,
>>>>>
>>>>>(snipped)
>>>>>
>>>>>2) Although I know the character encodings used in many regions, I note that
>>>>>mail clients sometimes prefer a different encoding than what other applications
>>>>>traditionally use. For example, ISO 2022-jp is used more often for mail in
>>>>>Japan than it is used precentage wise for other kinds of Japanese software (I
>>>>>believe).
>>>>>
>>>>>What is the preferred encoding(s) for mail in each country market? (Although we
>>>>>might prefer to recommend utf-8, I am looking to understand what the market
>>>>>practice actually is.)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>Tex,
>>>>
>>>>It would have been great if we had such a list when we worked on the
>>>>Mozilla mail. We tried to restrict the number of outgoing msg encodings
>>>>to the ones we can recommend (ISO types mostly unless we heard loud user
>>>>complaints).  The list was based on known RFC's and our knowledge of the
>>>>market since 1995. The results of this rather tortured attempt can be
>>>>found in the Mozilla menu:
>>>>
>>>>Edit > Preferences > Mail & Newsgroups > Composition > Character Encoding
>>>>
>>>>For reading msgs, we did not impose such a restriction and so the user
>>>>can choose to correct a msg encoding with all the encodings available
>>>>for browsing. For sending msgs, however, the user will have to customize
>>>>the above limited list by specifically opening the Customize .. menu --
>>>>and then the user can add any encoding he/she chooses.
>>>>
>>>>As far as we knew at the time (a few years back), some RFC's defined
>>>>encodings  for for certain languages (e.g. Japanese) but most languages
>>>>had no declaration from any authoritative body on this topic. This meant
>>>>for us the use of the greatest common denominator encoding(s) for each
>>>>language/language group (mostly ISO types) unless users of some
>>>>language/language groups complained -- Hebrew and Russian come to mind.
>>>>
>>>>If this situation has changed, I would love to know also.
>>>>
>>>>- Kat
>>>>
>>>>--
>>>>Katsuhiko Momoi
>>>>e-mail: [email-masked]
>>>>
>>>>
>>>>        
>>>>
>
>  
>

-- 
Katsuhiko Momoi
e-mail: [email-masked]

Date: Fri, 27 Aug 2004 01:54:32 -0700
From: Bill McQuillan <[email-masked]>
To: RFC822 <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

On Wed, 2004-08-25 at 18:35:07, Keith Moore <[email-masked]> wrote:
> here's something new to argue about :)

> I'm curious to know what people think of this idea.

While I am a little doubtful about the cost-benefit ratio of this proposal, I still have a suggestion.  :-)

Since the existing To: and CC: fields are treated nearly identically by the mail system and are basically just used to designate the "primary" and "secondary" recipients of the message to the human readers, I do not think that there needs to be yet two more classes of recipients. Instead, I think that a single "tertiary" recipient class which is intended not to be bothered by subsequent exchanges can be designated.

Not altogether facitiously, I propose to extend the sensory deprivation metaphor of Blind Carbon Copy (BCC:) to Deaf Carbon Copy (DCC:) for this tertiary class since follow-on messages in the conversation will not be "heard" by them.  ;-)

This would not affect the rest of the design in the NR draft except that now there would only be one new header field for MUAs to extract from, display or ignore.

-- 
Bill McQuillan <[email-masked]>

Date: Fri, 27 Aug 2004 10:51:37 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Philip Hazel <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Cc: Keith Moore <[email-masked]>, [email-masked], Charles Lindsey <[email-masked]>
Sender: [email-masked]

Philip Hazel writes:
> I guess I'm an "other". Not only MUAs, but also other software that 
> wants to send email uses "-t". I suspect there are more applications 
> that do this that I/you/we think.

Right. Personally, I've used it more times than I can remember, in 
situations ranging from little throwaway scripts up to parts of billing 
systems. Am I so unusual? It is the easiest way I know to generate a 
valid message with a custom From field on unix.

It wouldn't surprise me if many thousand automated mail senders use it.

Arnt

Date: Fri, 27 Aug 2004 01:51:29 -0700
From: Tex Texin <[email-masked]>
Organization: http://www.XenCraft.com
To: Nick Ing-Simmons <[email-masked]>
CC: [email-masked]
Subject: Re: internationalization of mail
Sender: [email-masked]

Nick,
thanks.

On #3 I guess I wasn't clear.
If a Thai font is used, expecting data in a Thai encoding (TIS 620-2533 for
example) it may not matter that the data is mislabeled as long as it is
unchanged it will display the right character for each byte. But if the Thai
data has gone thru a conversion from 8859-1 to utf-8, the bytes will no longer
show the correct Thai characters, so for this situation the solution of simply
changing fonts no longer works. The data needs to be transcoded back to
(620-2533 or whatever) for the user to be able to display it.

tex

Nick Ing-Simmons wrote:
> >3) How well supported is utf-8? I know Eudora does not. Pretty much
> >everything else does, right? Or at least supports some subset of Unicode. I am
> >a little concerned that if send out utf-8, then perhaps a Thai user, for
> >example, will find
> >his client doesnt support  Unicode Thai characters, but would have let him
> >display a native encoding...
> 
> That is possible, but typically UTF-8 aware applications "have" to fall
> back to native encoded fonts (good Unicode fonts are still rare) so
> in practice this will work.

-- 
-------------------------------------------------------------
Tex Texin   cell: +1 781 789 1898   mailto:[email-masked]
Xen Master                          http://www.i18nGuy.com
                         
XenCraft		            http://www.XenCraft.com
Making e-Business Work Around the World
-------------------------------------------------------------

Cc: [email-masked]
Subject: Re: internationalization of mail
To: [email-masked]
Date: Fri, 27 Aug 2004 09:25:58 +0100
From: Nick Ing-Simmons <[email-masked]>
Reply-To: Nick Ing-Simmons <[email-masked]>
Sender: [email-masked]

Tex Texin <[email-masked]> writes:
>
>Question one is trying to get at the risk of attempting to transcode every mail
>to utf-8 for storage, search and forwarding to recepients. If the mail is
>mislabeled then transcoding may corrupt the message. Since not all clients
>support utf-8, I may need to transcode the utf-8 to another charset so they can
>accept and render it. Hence question two.

My home-grown perl/Tk MUA does that.

>
>1) Are there statistics anywhere for the number of e-mails or message parts
>that have their charset mislabeled?
>(ie incorrectly identified as some encoding and the content is actually a
>different character encoding)

There is a of mail that is mis-labeled - typically claiming 
iso-8859-1 when it is really windows cp1252
You also get things labelled as ASCII but which have high bit chars.
However most of this junk is SPAM (but then so is most email these days).
I have a hacky workround for the cp1252 case.

>
>3) How well supported is utf-8? I know Eudora does not. Pretty much
>everything else does, right? Or at least supports some subset of Unicode. I am
>a little concerned that if send out utf-8, then perhaps a Thai user, for
>example, will find
>his client doesnt support  Unicode Thai characters, but would have let him
>display a native encoding...

That is possible, but typically UTF-8 aware applications "have" to fall
back to native encoded fonts (good Unicode fonts are still rare) so 
in practice this will work.

Date: Fri, 27 Aug 2004 09:13:39 +0100 (BST)
From: Philip Hazel <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

On Thu, 26 Aug 2004, Keith Moore wrote:

> My personal belief is that the "sendmail -t" problem is a minor one -
> since it seems to affect a relatively small number of older MUAs.
> Others might disagree.

I guess I'm an "other". Not only MUAs, but also other software that
wants to send email uses "-t". I suspect there are more applications 
that do this that I/you/we think.

However, I agree that it affects only a smallish number of MTAs 
(Sendmail, and those that try to mimic its interface).

Regards,
Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Thu, 26 Aug 2004 23:23:23 -0700
From: Tex Texin <[email-masked]>
Organization: http://www.XenCraft.com
To: Katsuhiko Momoi <[email-masked]>
CC: [email-masked]
Subject: Re: internationalization of mail
Sender: [email-masked]

Kat,

thanks. I agree with you about using Unicode internally. Where the encoding of
the message is correctly labeled so it can be accurately transcoded it is a
good solution. The thrust of my questions is to find out the percentage of mail
messages is incorrectly labeled or not labeled with an encoding, and may
therefore be risky to transcode.

Secondarily there are many products that support Unicode but not necessarily
the full (or nearly full) character repertoire.
I need to understand whether moving to Unicode means some mail clients won't
support some languages.
As you know many products today don't fully support a variety of languages
properly, but users can still use them by using a native font and ignoring
encoding declarations. For example, displaying Thai with a Thai font even
though the product thinks it is using ISO 8859-1.

However, if the text is transcoded to UTF-8 based on an incorrect encoding,
then it won't be displayable by treating the utf-8 bytes as Thai characters.

So, can I take it as true that for NS mail, the transcoding to Unicode didn't
introduce significant problems and most mail was rendered effectively?

tex

Katsuhiko Momoi wrote:
> 
> Tex Texin wrote:
> 
> >Hi Kat!
> >It's been a while...
> >
> >
> Hi, Tex,
> 
> Good to see you here.
> 
> >thanks for the mail, it was helpful.
> >
> >Correct me if I am wrong, but in the Mozilla environment the mail is treated as
> >a bucket of bytes that just gets passed around.
> >So for example if one subject hdr is in iso 2022-jp and another is in
> >shift-jis, you can change the display to show and interpret one or the other
> >correctly, but not both at the same time. For purposes of search and filtering,
> >I was thinking of converting to unicode, and then all subjects would display
> >correctly, as well as be searchable.
> >
> >
> 
> Not really. The received msgs are stored as is but as each msg is sent
> to rendering, it is converted to utf-8 before it gets to rendering. So
> in Mozilla mail, one can display headers of all supported encodings at
> once in the header viewing pane. I had a demo I used at one of the
> Unicode conferences which showed mail headers in 15 or so languages. But
> the requirement for this is that the each header must be marked with
> proper MIME encoding info. This means Mozilla can display shift_jis and
> iso-2022-jp headers at the same time as long they use MIME encoding. If
> a header of a msg does not use MIME encoding, then Mozilla will use the
> user's default msg display encoding.
> 
> Sorting and filtering for Mozilla are all done in Unicode as well. For
> more info, you can look at our Unicode presentation (by Naoki Hotta and
> I) here:
> 
> http://wp.netscape.com/eng/intl/docs/iuc17/mail/iuc17mail.html
> 
> and in particular at this diagram showing the data flow for MIME header
> and body with associated MIME encoding.
> 
> http://wp.netscape.com/eng/intl/docs/iuc17/mail/slide05.html
> 
> If I may add my 2 cents,  if one is designing a webmail system, why not
> use Unicode for all internal processing? (while keeping the original
> data as is in case you need to refer to them).  Most search data are
> handled in Unicode anyway now and this will increase compatibility with
> the search functions you might want to offer in a webmail service. I
> would even go as far as to say that we may want to unify send encoding
> of webmail systems to just UTF-8 unless only ASCII data is being sent
> out. I think Gmail is currently doing just that. In the absence of
> country specific standards, this is one way to improve the situation.
> Most popular mail programs can handle UTF-8. If some mailers cannot,
> then I think it is time to update the code! I think this standardization
> can happen first in webmail -- where simplification is acceptable --
> ahead of standalone mailers, which may face harder hurdles in unifying
> to UTF-8.
> 
> - Kat
> 
> >But it may not be worthwhile if many of the mails are incorrectly identified so
> >transcoding to utf-8 generates muck, or errors out.
> >
> >tex
> >
> >Katsuhiko Momoi wrote:
> >
> >
> >>Tex Texin wrote:
> >>
> >>
> >>
> >>>Hi,
> >>>
> >>>(snipped)
> >>>
> >>>2) Although I know the character encodings used in many regions, I note that
> >>>mail clients sometimes prefer a different encoding than what other applications
> >>>traditionally use. For example, ISO 2022-jp is used more often for mail in
> >>>Japan than it is used precentage wise for other kinds of Japanese software (I
> >>>believe).
> >>>
> >>>What is the preferred encoding(s) for mail in each country market? (Although we
> >>>might prefer to recommend utf-8, I am looking to understand what the market
> >>>practice actually is.)
> >>>
> >>>
> >>>
> >>>
> >>Tex,
> >>
> >>It would have been great if we had such a list when we worked on the
> >>Mozilla mail. We tried to restrict the number of outgoing msg encodings
> >>to the ones we can recommend (ISO types mostly unless we heard loud user
> >>complaints).  The list was based on known RFC's and our knowledge of the
> >>market since 1995. The results of this rather tortured attempt can be
> >>found in the Mozilla menu:
> >>
> >>Edit > Preferences > Mail & Newsgroups > Composition > Character Encoding
> >>
> >>For reading msgs, we did not impose such a restriction and so the user
> >>can choose to correct a msg encoding with all the encodings available
> >>for browsing. For sending msgs, however, the user will have to customize
> >>the above limited list by specifically opening the Customize .. menu --
> >>and then the user can add any encoding he/she chooses.
> >>
> >>As far as we knew at the time (a few years back), some RFC's defined
> >>encodings  for for certain languages (e.g. Japanese) but most languages
> >>had no declaration from any authoritative body on this topic. This meant
> >>for us the use of the greatest common denominator encoding(s) for each
> >>language/language group (mostly ISO types) unless users of some
> >>language/language groups complained -- Hebrew and Russian come to mind.
> >>
> >>If this situation has changed, I would love to know also.
> >>
> >>- Kat
> >>
> >>--
> >>Katsuhiko Momoi
> >>e-mail: [email-masked]
> >>
> >>

-- 
-------------------------------------------------------------
Tex Texin   cell: +1 781 789 1898   mailto:[email-masked]
Xen Master                          http://www.i18nGuy.com
                         
XenCraft		            http://www.XenCraft.com
Making e-Business Work Around the World
-------------------------------------------------------------

Date: Thu, 26 Aug 2004 23:18:51 -0400
From: Keith Moore <[email-masked]>
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

> There is an alternative, and possibly better, way to solve this problem.
> 
> Leave the To: and Cc: headers alone, with their present meanings, and let
> them be included in all messages as at present.
> 
> Now invent ONE new header
> 
> no-reply-to	= "No-Reply-To" noreplylist CRLF
> noreplylist	= ( "To" / "Cc" / "From" / "Reply-To" / "List-Post" /
> 		    "Bcc" / ... / address) [ "," noreplylist ]
> 
> which essentially says, "when replying to this message, kindly do not
> reply to any of the people listed". So you can exclude the whole of the
> "To:" list, the whole of the "Cc:" list, the whole of the "From:" list
> (plus any others that might be allowed) and you can alternatively, or
> additionally, exclude any explicitly listed <addresses>.

If you'll note in the draft, I considered something similar to this.
As you point out, it's not compatible with legacy MUA implementations.
It also would arguably have one header field changing the meaning of
another header field.  My personal belief is that the "sendmail -t"
problem is a minor one - since it seems to affect a relatively small
number of older MUAs. Others might disagree.

> Also, while you are about it, can we include some mechanism for "reply to
> list" in mailing lists? 

how would this be different than List-Post:?

Keith

To: [email-masked]
Xref: clerew local.mime:3370
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Thu, 26 Aug 2004 14:43:17 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>here's something new to argue about :)

>I'm curious to know what people think of this idea.

There is an alternative, and possibly better, way to solve this problem.

Leave the To: and Cc: headers alone, with their present meanings, and let
them be included in all messages as at present.

Now invent ONE new header

no-reply-to	= "No-Reply-To" noreplylist CRLF
noreplylist	= ( "To" / "Cc" / "From" / "Reply-To" / "List-Post" /
		    "Bcc" / ... / address) [ "," noreplylist ]

which essentially says, "when replying to this message, kindly do not
reply to any of the people listed". So you can exclude the whole of the
"To:" list, the whole of the "Cc:" list, the whole of the "From:" list
(plus any others that might be allowed) and you can alternatively, or
additionally, exclude any explicitly listed <addresses>.

This will admittedly behave worse on legacy implemenations, but I think it
will otherwise be more easily implemented (for example, 'sendmail -t'
would not need any change), and it does provide a means to exclude the
From-address (which would be particularly useful in mailing lists).

Looking around at other solutions to this problem, there is a fairly
widely used Mail-Copies-To header used in Netnews, but that is limited to
encouraging/discouraging email replies to the poster during followups to
News. It might work quite well for mailing lists as well, but there is no
obvious extension of it to cover these new cases.

Also, while you are about it, can we include some mechanism for "reply to
list" in mailing lists? Perhaps some conbination of Reply-To and
No_Reply-To would do it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3369
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Thu, 26 Aug 2004 14:21:43 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>now if I were to write a recommendation for how to write mbox files,
>I'd say that all body parts MUST be encoded if necessary to prevent
>the characters "From" from appearing at the beginning of a line.
>I'd also say that Content-Length MUST NOT be written, that From_ dates 
>have to be in exactly such-and-such a format, that Return-Path fields 
>are REQUIRED when storing a message received by SMTP, even if they're 
>redundant with From_ lines, etc.  but that's not the problem we're 
>currently trying to solve.

Which suggests an alternative to the usual, irreversible, >-stuffing
convention.

Simple replace any line in the body starting with "From " with
"From=20" and ensure that the CTE is quoted-printable (well, there is a
bit more to it than that).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 26 Aug 2004 19:34:27 -0700
From: Katsuhiko Momoi <[email-masked]>
To: [email-masked]
CC: Tex Texin <[email-masked]>
Subject: Re: internationalization of mail
Sender: [email-masked]

Tex Texin wrote:

>Hi Kat!
>It's been a while...
>  
>
Hi, Tex,

Good to see you here.

>thanks for the mail, it was helpful.
>
>Correct me if I am wrong, but in the Mozilla environment the mail is treated as
>a bucket of bytes that just gets passed around.
>So for example if one subject hdr is in iso 2022-jp and another is in
>shift-jis, you can change the display to show and interpret one or the other
>correctly, but not both at the same time. For purposes of search and filtering,
>I was thinking of converting to unicode, and then all subjects would display
>correctly, as well as be searchable.
>  
>

Not really. The received msgs are stored as is but as each msg is sent 
to rendering, it is converted to utf-8 before it gets to rendering. So 
in Mozilla mail, one can display headers of all supported encodings at 
once in the header viewing pane. I had a demo I used at one of the 
Unicode conferences which showed mail headers in 15 or so languages. But 
the requirement for this is that the each header must be marked with 
proper MIME encoding info. This means Mozilla can display shift_jis and 
iso-2022-jp headers at the same time as long they use MIME encoding. If 
a header of a msg does not use MIME encoding, then Mozilla will use the 
user's default msg display encoding.

Sorting and filtering for Mozilla are all done in Unicode as well. For 
more info, you can look at our Unicode presentation (by Naoki Hotta and 
I) here:

http://wp.netscape.com/eng/intl/docs/iuc17/mail/iuc17mail.html

and in particular at this diagram showing the data flow for MIME header 
and body with associated MIME encoding.

http://wp.netscape.com/eng/intl/docs/iuc17/mail/slide05.html

If I may add my 2 cents,  if one is designing a webmail system, why not 
use Unicode for all internal processing? (while keeping the original 
data as is in case you need to refer to them).  Most search data are 
handled in Unicode anyway now and this will increase compatibility with 
the search functions you might want to offer in a webmail service. I 
would even go as far as to say that we may want to unify send encoding 
of webmail systems to just UTF-8 unless only ASCII data is being sent 
out. I think Gmail is currently doing just that. In the absence of 
country specific standards, this is one way to improve the situation. 
Most popular mail programs can handle UTF-8. If some mailers cannot, 
then I think it is time to update the code! I think this standardization 
can happen first in webmail -- where simplification is acceptable -- 
ahead of standalone mailers, which may face harder hurdles in unifying 
to UTF-8.

- Kat

>But it may not be worthwhile if many of the mails are incorrectly identified so
>transcoding to utf-8 generates muck, or errors out.
>
>tex
>
>Katsuhiko Momoi wrote:
>  
>
>>Tex Texin wrote:
>>
>>    
>>
>>>Hi,
>>>
>>>(snipped)
>>>
>>>2) Although I know the character encodings used in many regions, I note that
>>>mail clients sometimes prefer a different encoding than what other applications
>>>traditionally use. For example, ISO 2022-jp is used more often for mail in
>>>Japan than it is used precentage wise for other kinds of Japanese software (I
>>>believe).
>>>
>>>What is the preferred encoding(s) for mail in each country market? (Although we
>>>might prefer to recommend utf-8, I am looking to understand what the market
>>>practice actually is.)
>>>
>>>
>>>      
>>>
>>Tex,
>>
>>It would have been great if we had such a list when we worked on the
>>Mozilla mail. We tried to restrict the number of outgoing msg encodings
>>to the ones we can recommend (ISO types mostly unless we heard loud user
>>complaints).  The list was based on known RFC's and our knowledge of the
>>market since 1995. The results of this rather tortured attempt can be
>>found in the Mozilla menu:
>>
>>Edit > Preferences > Mail & Newsgroups > Composition > Character Encoding
>>
>>For reading msgs, we did not impose such a restriction and so the user
>>can choose to correct a msg encoding with all the encodings available
>>for browsing. For sending msgs, however, the user will have to customize
>>the above limited list by specifically opening the Customize .. menu --
>>and then the user can add any encoding he/she chooses.
>>
>>As far as we knew at the time (a few years back), some RFC's defined
>>encodings  for for certain languages (e.g. Japanese) but most languages
>>had no declaration from any authoritative body on this topic. This meant
>>for us the use of the greatest common denominator encoding(s) for each
>>language/language group (mostly ISO types) unless users of some
>>language/language groups complained -- Hebrew and Russian come to mind.
>>
>>If this situation has changed, I would love to know also.
>>
>>- Kat
>>
>>--
>>Katsuhiko Momoi
>>e-mail: [email-masked]
>>    
>>

Date: Thu, 26 Aug 2004 19:01:10 -0700
From: Tex Texin <[email-masked]>
Organization: http://www.XenCraft.com
To: Katsuhiko Momoi <[email-masked]>
CC: [email-masked]
Subject: Re: internationalization of mail
Sender: [email-masked]

Hi Kat!
It's been a while...

thanks for the mail, it was helpful.

Correct me if I am wrong, but in the Mozilla environment the mail is treated as
a bucket of bytes that just gets passed around.
So for example if one subject hdr is in iso 2022-jp and another is in
shift-jis, you can change the display to show and interpret one or the other
correctly, but not both at the same time. For purposes of search and filtering,
I was thinking of converting to unicode, and then all subjects would display
correctly, as well as be searchable.

But it may not be worthwhile if many of the mails are incorrectly identified so
transcoding to utf-8 generates muck, or errors out.

tex

Katsuhiko Momoi wrote:
> 
> Tex Texin wrote:
> 
> >Hi,
> >
> >(snipped)
> >
> >2) Although I know the character encodings used in many regions, I note that
> >mail clients sometimes prefer a different encoding than what other applications
> >traditionally use. For example, ISO 2022-jp is used more often for mail in
> >Japan than it is used precentage wise for other kinds of Japanese software (I
> >believe).
> >
> >What is the preferred encoding(s) for mail in each country market? (Although we
> >might prefer to recommend utf-8, I am looking to understand what the market
> >practice actually is.)
> >
> >
> Tex,
> 
> It would have been great if we had such a list when we worked on the
> Mozilla mail. We tried to restrict the number of outgoing msg encodings
> to the ones we can recommend (ISO types mostly unless we heard loud user
> complaints).  The list was based on known RFC's and our knowledge of the
> market since 1995. The results of this rather tortured attempt can be
> found in the Mozilla menu:
> 
> Edit > Preferences > Mail & Newsgroups > Composition > Character Encoding
> 
> For reading msgs, we did not impose such a restriction and so the user
> can choose to correct a msg encoding with all the encodings available
> for browsing. For sending msgs, however, the user will have to customize
> the above limited list by specifically opening the Customize .. menu --
> and then the user can add any encoding he/she chooses.
> 
> As far as we knew at the time (a few years back), some RFC's defined
> encodings  for for certain languages (e.g. Japanese) but most languages
> had no declaration from any authoritative body on this topic. This meant
> for us the use of the greatest common denominator encoding(s) for each
> language/language group (mostly ISO types) unless users of some
> language/language groups complained -- Hebrew and Russian come to mind.
> 
> If this situation has changed, I would love to know also.
> 
> - Kat
> 
> --
> Katsuhiko Momoi
> e-mail: [email-masked]

-- 
-------------------------------------------------------------
Tex Texin   cell: +1 781 789 1898   mailto:[email-masked]
Xen Master                          http://www.i18nGuy.com
                         
XenCraft		            http://www.XenCraft.com
Making e-Business Work Around the World
-------------------------------------------------------------

Date: Thu, 26 Aug 2004 18:47:55 -0700
From: Katsuhiko Momoi <[email-masked]>
To: [email-masked]
CC: Tex Texin <[email-masked]>
Subject: Re: internationalization of mail
Sender: [email-masked]

Tex Texin wrote:

>Hi,
>
>(snipped)
>
>2) Although I know the character encodings used in many regions, I note that
>mail clients sometimes prefer a different encoding than what other applications
>traditionally use. For example, ISO 2022-jp is used more often for mail in
>Japan than it is used precentage wise for other kinds of Japanese software (I
>believe).
>
>What is the preferred encoding(s) for mail in each country market? (Although we
>might prefer to recommend utf-8, I am looking to understand what the market
>practice actually is.)
>  
>
Tex,

It would have been great if we had such a list when we worked on the 
Mozilla mail. We tried to restrict the number of outgoing msg encodings 
to the ones we can recommend (ISO types mostly unless we heard loud user 
complaints).  The list was based on known RFC's and our knowledge of the 
market since 1995. The results of this rather tortured attempt can be 
found in the Mozilla menu:

Edit > Preferences > Mail & Newsgroups > Composition > Character Encoding

For reading msgs, we did not impose such a restriction and so the user 
can choose to correct a msg encoding with all the encodings available 
for browsing. For sending msgs, however, the user will have to customize 
the above limited list by specifically opening the Customize .. menu -- 
and then the user can add any encoding he/she chooses.

As far as we knew at the time (a few years back), some RFC's defined 
encodings  for for certain languages (e.g. Japanese) but most languages 
had no declaration from any authoritative body on this topic. This meant 
for us the use of the greatest common denominator encoding(s) for each 
language/language group (mostly ISO types) unless users of some 
language/language groups complained -- Hebrew and Russian come to mind.

If this situation has changed, I would love to know also.

- Kat

-- 
Katsuhiko Momoi
e-mail: [email-masked]

Date: Thu, 26 Aug 2004 18:19:38 -0700
From: Tex Texin <[email-masked]>
Organization: http://www.XenCraft.com
To: [email-masked]
Subject: internationalization of mail
Sender: [email-masked]

Hi,

I have a few questions. The questions are partially technical partially market
research oriented.
Fundamentally I am trying to establish the risks and benefits of using utf-8
for mail clients and servers.

Question one is trying to get at the risk of attempting to transcode every mail
to utf-8 for storage, search and forwarding to recepients. If the mail is
mislabeled then transcoding may corrupt the message. Since not all clients
support utf-8, I may need to transcode the utf-8 to another charset so they can
accept and render it. Hence question two.

1) Are there statistics anywhere for the number of e-mails or message parts
that have their charset mislabeled?
(ie incorrectly identified as some encoding and the content is actually a
different character encoding)

2) Although I know the character encodings used in many regions, I note that
mail clients sometimes prefer a different encoding than what other applications
traditionally use. For example, ISO 2022-jp is used more often for mail in
Japan than it is used precentage wise for other kinds of Japanese software (I
believe).

What is the preferred encoding(s) for mail in each country market? (Although we
might prefer to recommend utf-8, I am looking to understand what the market
practice actually is.)

3) How well supported is utf-8? I know Eudora does not. Pretty much
everything else does, right? Or at least supports some subset of Unicode. I am
a little concerned that if send out utf-8, then perhaps a Thai user, for
example, will find
his client doesnt support  Unicode Thai characters, but would have let him
display a native encoding...

4) I will want to do some testing of a mail product and I would like to know
which clients are
popular in each region. I know there are some local mail readers that I should
QA with, but I would like to get an expanded list.

======
I am new to this list and it was recommended to me as the place to ask
questions like this. If you are aware of any other forums for discussion i18n
and mail, I would appreciate knowing of them.

thanks
tex

Date: Thu, 26 Aug 2004 20:20:58 -0400
From: Keith Moore <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: Steve Dorner <[email-masked]>, [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

>> Note that the mail clients used by a (large?  overwhelming?)
>> majority of folks will NOT do this.  Most mass-market MUA's
>> hide all but a select (and small) set of headers.
> 
> 
> And, to add to Steve's comment, most of them have decided to do so in 
> the light of overwhelming messages from customers that they don't want 
> to see a lot of clutter.   Few of those MUAs even manage to display 
> Reply-to without being specially stimulated, so the expectation of their 
> showing a new, and (at least initially) little-used, header is, I would 
> think, not large.

as the document points out, there are other ways to display the noreply 
recipients than to display them as separate header fields.  details of 
presentation are, as usual, left to the MUA.

Keith

Date: Thu, 26 Aug 2004 20:18:56 -0400
From: Keith Moore <[email-masked]>
To: Steve Dorner <[email-masked]>
Cc: John C Klensin <[email-masked]>, [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Steve Dorner wrote:
> At 4:45 PM -0400 8/26/04, Keith Moore wrote:
> 
>> For a receiving MUA to "honor" the -NoReply fields merely requires
>> that they be displayed.
> 
> 
> Note that the mail clients used by a (large?  overwhelming?) majority of 
> folks will NOT do this.  Most mass-market MUA's hide all but a select 
> (and small) set of headers.

yes, that's understood.   a few MUAs can be configured as to which 
fields they display.  for the rest, "show all headers" (or "blah blah 
blah" :) is the best they can do.

of course, this is no different than any other new field.

Date: Thu, 26 Aug 2004 20:03:38 -0400
From: John C Klensin <[email-masked]>
To: Steve Dorner <[email-masked]>, Keith Moore <[email-masked]>
cc: [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

--On Thursday, August 26, 2004 4:55 PM -0700 Steve Dorner 
<[email-masked]> wrote:

> At 4:45 PM -0400 8/26/04, Keith Moore wrote:
>> For a receiving MUA to "honor" the -NoReply fields merely
>> requires that they be displayed.
>
> Note that the mail clients used by a (large?  overwhelming?)
> majority of folks will NOT do this.  Most mass-market MUA's
> hide all but a select (and small) set of headers.

And, to add to Steve's comment, most of them have decided to do 
so in the light of overwhelming messages from customers that 
they don't want to see a lot of clutter.   Few of those MUAs 
even manage to display Reply-to without being specially 
stimulated, so the expectation of their showing a new, and (at 
least initially) little-used, header is, I would think, not 
large.

     john

Date: Thu, 26 Aug 2004 16:55:40 -0700
To: Keith Moore <[email-masked]>
From: Steve Dorner <[email-masked]>
Subject: Re: draft-moore-mail-nr-fields-00
Cc: John C Klensin <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

At 4:45 PM -0400 8/26/04, Keith Moore wrote:
>For a receiving MUA to "honor" the -NoReply fields merely requires
>that they be displayed.

Note that the mail clients used by a (large?  overwhelming?) majority 
of folks will NOT do this.  Most mass-market MUA's hide all but a 
select (and small) set of headers.

Date: Thu, 26 Aug 2004 19:31:39 -0400
From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

> If nobody can find client behaviors that MFT/MRT can achieve, that NR
> together with (a) cannot, especially wrt news (MFT/MRT is used in news
> too, IIRC), I'd like to see NR move forward.

there are client behaviors that MFT/MRT can produce that NR cannot 
produce.  in particular, NR has no way to specify situations where
"reply to all" is not sent to a superset of the recipients who would be 
sent if "reply to author" were used instead.

as far as I can tell, this is a plus for NR.

Keith

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Fri, 27 Aug 2004 01:08:28 +0200
Lines: 31
Cancel-Lock: sha1:bOsuVBaxmYkzdPH0ZUHSUMNZtgs=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> I'd rather avoid having a definition of Mail-Copies-To in email that 
> differed from the definition used in usenet.

What if it was possible to use the same definition on both email and
usenet?

> I can imagine two ways to solve the problem:
>
> (a) Reply-To with some magic address that is valid syntax but a no-op. 
> maybe this:
>
> Reply-To: nobody :;

This appear to work.

> (b) a new No-Replies-To-Author field
>
> offhand, I'd prefer (a) on the assumption that it is more likely to do
> the right thing with existing MUAs.  it also avoids having one field
> change the meaning of another field.

Right.  And it would be in the same style as the rest of NR.

If nobody can find client behaviors that MFT/MRT can achieve, that NR
together with (a) cannot, especially wrt news (MFT/MRT is used in news
too, IIRC), I'd like to see NR move forward.

Thanks,
Simon

Date: Thu, 26 Aug 2004 18:42:43 -0400
From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

I'd rather avoid having a definition of Mail-Copies-To in email that 
differed from the definition used in usenet.

I can imagine two ways to solve the problem:

(a) Reply-To with some magic address that is valid syntax but a no-op. 
maybe this:

Reply-To: nobody :;

(b) a new No-Replies-To-Author field

offhand, I'd prefer (a) on the assumption that it is more likely to do
the right thing with existing MUAs.  it also avoids having one field
change the meaning of another field.

Keith

> My conceptual view of things: NoReply replace _all_ functionality in
> MFT/MRT, with better behavior with legacy clients, EXCEPT for the
> feature of avoiding copies to the originator.  To implement the final
> feature, it appears impossible to avoid the same bad legacy behavior
> that MFT/MRT have.  The conclusion is that NoReply, plus an
> author-copies avoiding fix, solve the same problem as MFT/MRT, but
> with better legacy behavior on average.
> 
> Gnus support an author-copies avoiding solution, by obeying the
> `Mail-Copies-To: nobody' news header even for e-mail.  See
> <http://www.newsreaders.com/misc/mail-copies-to.html>.
> 
> It might be complicated to specify Mail-Copies-To for e-mail, in a way
> that make sense for all combination of news/e-mail and handover from
> news to mail and vice versa.  But it may be possible.  Perhaps by
> adopting, for e-mail use, only the "nobody" subset of MCT.
> 
> It is easier to criticize something concrete.  So consider if NR
> included the text below.  How would this interact with other e-mail
> headers, and with the full MCT header used in news?
> 
> 2.3.  The Mail-Copies-To field
> 
>    The Mail-Copies-To header indicates whether or not the originator
>    (i.e., the From: mailbox) wishes to receive replies to the e-mail.
> 
>    The content syntax makes use of syntax defined in [RFC 2822].
> 
>       mail-copies-to    = "Mail-Copies-To:" "nobody" CRLF
> 
>    A minimum of 0 and a maximum of 1 of each of these fields is
>    permitted in a message header.
> 
>    The keyword "nobody" indicates that the sender do not wish to be
>    included in any follow ups to the message.
> 
> 2.3.1. Composition of Replies to messages containing Mail-Copies-To: nobody
> 
>    The automatic actions of a followup agent, when a "Mail-Copies-To:"
>    header is present with the "nobody" value, are as follows.  The
>    followup agent MUST NOT, by default, add the mailbox in From: to
>    the recipient list.  A MUA SHOULD provide a means for a user to
>    override the default address list used in a reply.
> 
> Thanks,
> Simon
> 

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Fri, 27 Aug 2004 00:24:29 +0200
Lines: 61
Cancel-Lock: sha1:hI8msrepEs3Aaf5eeO0c6EtEQg4=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> anyway, the premise behind NoReply fields is that legacy MUAs can
> already do the right thing with them.  I looked at defining other
> kinds of fields that would cancel out the "reply- ability" of other
> addresses, and decided that this wasn't the way to go.
>
> OTOH if we wanted to standardize a convention like the above to allow an
> author to specify "don't send replies to me", I wouldn't have a problem
> with adding a section to the NoReply document that describes that 
> convention.

I believe this would be useful.  I understand it wouldn't be as neat
as the rest of NoReply.

My conceptual view of things: NoReply replace _all_ functionality in
MFT/MRT, with better behavior with legacy clients, EXCEPT for the
feature of avoiding copies to the originator.  To implement the final
feature, it appears impossible to avoid the same bad legacy behavior
that MFT/MRT have.  The conclusion is that NoReply, plus an
author-copies avoiding fix, solve the same problem as MFT/MRT, but
with better legacy behavior on average.

Gnus support an author-copies avoiding solution, by obeying the
`Mail-Copies-To: nobody' news header even for e-mail.  See
<http://www.newsreaders.com/misc/mail-copies-to.html>.

It might be complicated to specify Mail-Copies-To for e-mail, in a way
that make sense for all combination of news/e-mail and handover from
news to mail and vice versa.  But it may be possible.  Perhaps by
adopting, for e-mail use, only the "nobody" subset of MCT.

It is easier to criticize something concrete.  So consider if NR
included the text below.  How would this interact with other e-mail
headers, and with the full MCT header used in news?

2.3.  The Mail-Copies-To field

   The Mail-Copies-To header indicates whether or not the originator
   (i.e., the From: mailbox) wishes to receive replies to the e-mail.

   The content syntax makes use of syntax defined in [RFC 2822].

      mail-copies-to    = "Mail-Copies-To:" "nobody" CRLF

   A minimum of 0 and a maximum of 1 of each of these fields is
   permitted in a message header.

   The keyword "nobody" indicates that the sender do not wish to be
   included in any follow ups to the message.

2.3.1. Composition of Replies to messages containing Mail-Copies-To: nobody

   The automatic actions of a followup agent, when a "Mail-Copies-To:"
   header is present with the "nobody" value, are as follows.  The
   followup agent MUST NOT, by default, add the mailbox in From: to
   the recipient list.  A MUA SHOULD provide a means for a user to
   override the default address list used in a reply.

Thanks,
Simon

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Fri, 27 Aug 2004 00:16:13 +0200
Lines: 23
Cancel-Lock: sha1:L+74qPLm2BY/AazXJgdBxW3HbvI=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

[I find myself agreeing with the rest of what you said.]

>> Further, either the proposal or I am missing something: How can the
>> sender of a messages indicate that she _herself_ doesn't want to
>> receive any further replies?
>
> Reply-To: nobody@[]
>
> (yes, I know it's ugly.)

Not only ugly, anyone responding to a message containing that will
likely receive a bounce because [] doesn't resolve.  Here's what my
sendmail respond:

i7QLVOlD007422: to=<nobody@[]>, delay=00:00:01, xdelay=00:00:00, mailer=esmtp, pri=120331, relay=[], dsn=5.1.2, stat=Host unknown (Name server: []: no data known)
i7QLVOlD007422: i7QLVQlC007432: DSN: Host unknown (Name server: []: no data known)

I don't think this idea should be recommended.

Thanks,
Simon

Date: Thu, 26 Aug 2004 16:45:19 -0400
From: Keith Moore <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: draft-moore-mail-nr-fields-00
Sender: [email-masked]

> 
> Keith,
> 
> Regardless of the merits of this idea has it been introduced as 
> part of the original email model, I see several problems.  At 
> minimum, I believe a future version should respond to them.  At 
> worst, it turns this, on balance, into a Bad Idea:
> 
> (1) Most of the functionality, for most cases, appears to be 
> addressed by
>    Reply-to: address1, address2, address3,...
> Now, while that has rarely been well-implemented, it has been in 
> the standard since 822 or earlier.  It is hard for me to see an 
> argument that a newer set of headers would work significantly 
> better.
> 
> (2) Your legacy discussion fails to discuss the interactions 
> among a Reply-to field that is present and these new fields if 
> they are also present.  It seems to me that could get 
> complicated.
>
> (3) To the extent to which this and Reply-to overlap, it gives 
> us two different ways to do essentially the same thing.  That is 
> rarely, if ever, a good idea.

For better or worse, Reply-To has been nearly universally implemented
by MUAs as:

When composing a reply to a message for which the Reply-To field
exists, use the addresses in Reply-To as the addresses of the To
field of the reply.  If the reply is a "reply to all", To and Cc 
addresses from the subject message are also Cc'ed on the reply.

So with Reply-To as it is generally implemented, there's no way for
an author to include a recipient in a To or CC field without that
recipient getting replies to the message.

Since the -NoReply fields do not affect replies at all, I do not
see how they could interfere with Reply-To.  In particular if a
recipient appears in both a Reply-To field and a -NoReply field,
a reply to that message would be  exactly the same as if the 
recipient appeared only in the Reply-To field.  

I thought this was clear from what I wrote, but if not, I can certainly
add clarifying text.

> (4) If these are sent out from a system that has no idea whether 
> the recipient will honor or ignore them, it seems to me that we 
> introduce the potential for huge levels of confusion and unmet 
> expectations.   That isn't a showstopper, but argues against 
> this, IMO, unless it is really clear that there would be 
> significant benefits.

For a receiving MUA to "honor" the -NoReply fields merely requires
that they be displayed.  They do not affect how replies are composed. 
The reason that I designed the fields to work in this way (in contrast
to some other proposals) is to minimize the potential for confusion and
unmet expectations.  

The biggest potential for unmet expectations that I see is for users of
legacy MUAs that (a) use "sendmail -t" and (b) allow message authors to 
specify arbitrary headers.  The potential exists for those users to
specify NoReply recipients who will not receive the message.  The 
document doesn't currently recommend that "sendmail" and similar 
programs be upgraded to support NoReply fields, but that is probably
what should happen.  Assuming that all "sendmail" and similar programs
would eventually be upgraded, this would further minimize the potential
for unmet expectations.

Keith

Date: Thu, 26 Aug 2004 16:43:48 -0400
From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

> 
> Keith Moore <[email-masked]> writes:
> 
> > I'm curious to know what people think of this idea.
> 
> I like the way it behave on legacy clients.  In comparison,
> Mail-Followup-To and Mail-Reply-To doesn't work if a client isn't
> aware of those headers.
> 
> I dislike that it require modification to the "submission" phase.  I
> think one could argue that it is _more_ difficult to change submission
> agents ("sendmail -t") AND all mail clients that do SMTP internally,
> than it is to change only mail clients to support the MFT/MRT
> approach.

Assuming

a) there are many more MUAs than MTAs that implement a "sendmail -t"
interface, and

b) the actual code changes are trivial, and the real barrier to 
upgrading existing MUAs and MTAs is not in the coding or testing
but in pushing the new bits out to hosts

I think it's about the same amount of _effort_ either way.  However,
Mail-{Followup,Reply}-To (MFT for short) appears to have a more 
difficult/confusing transition phase.  Most sending MUAs will make it
difficult for authors to generate NoReply fields until they're 
upgraded to support them.  And receiving MUAs will do more-or-less
the right thing with them anyway.

MFT also allows the author to specify bizarre behavior - so "reply to
all" can be specified so that only the author gets a reply, whereas
"reply to author" can be specified so that everyone gets a reply. With
NoReply fields, the reply-to-author recipients are always a subset of
the reply-to-all recipients.  To me that seems like a desirable
property.

> Further, either the proposal or I am missing something: How can the
> sender of a messages indicate that she _herself_ doesn't want to
> receive any further replies?

Reply-To: nobody@[]

(yes, I know it's ugly.)  anyway, the premise behind NoReply fields is
that legacy MUAs can already do the right thing with them.  I looked
at defining other kinds of fields that would cancel out the "reply-
ability" of other addresses, and decided that this wasn't the way to go.

OTOH if we wanted to standardize a convention like the above to allow an
author to specify "don't send replies to me", I wouldn't have a problem
with adding a section to the NoReply document that describes that 
convention.

Keith

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: draft-moore-mail-nr-fields-00
Date: Thu, 26 Aug 2004 22:20:24 +0200
Lines: 52
Cancel-Lock: sha1:bzzmMgCqKGKVpZYcRzYAxyXz6O0=
Sender: [email-masked]

John C Klensin <[email-masked]> writes:

> (1) Most of the functionality, for most cases, appears to be 
> addressed by
>   Reply-to: address1, address2, address3,...
> Now, while that has rarely been well-implemented, it has been in 
> the standard since 822 or earlier.  It is hard for me to see an 
> argument that a newer set of headers would work significantly 
> better.

I'm not sure Reply-To fully addresses the same issue that MFT/MRT and
NoReply address.  Some of the problems with Reply-To in general are
mentioned in:

"Reply-To" Munging Considered Harmful
http://www.unicom.com/pw/reply-to-harmful.html

I'm aware that it primarily wanted to address only mailing list
software Reply-To munging.

> (2) Your legacy discussion fails to discuss the interactions 
> among a Reply-to field that is present and these new fields if 
> they are also present.  It seems to me that could get 
> complicated.

With my understanding of the proposal, I don't see this problem.  The
neat thing with NoReply is that recipients aren't required to be aware
of the specification, existing conforming clients will do the right
thing.  All processing happen on the submitter side.  I only read the
proposal briefly, so I may be mistaken though.

See section 2.2.5 on what recipients should do.  They aren't required
to do anything legacy clients don't already do, AFAICT.

> (3) To the extent to which this and Reply-to overlap, it gives 
> us two different ways to do essentially the same thing.  That is 
> rarely, if ever, a good idea.

I agree.  Sadly, the functionality appear to be needed, given the
adoption of MFT/MRT.

> (4) If these are sent out from a system that has no idea whether 
> the recipient will honor or ignore them, it seems to me that we 
> introduce the potential for huge levels of confusion and unmet 
> expectations.   That isn't a showstopper, but argues against 
> this, IMO, unless it is really clear that there would be 
> significant benefits.

Again, as I understood it, this isn't a problem.

Regards,
Simon

Date: Thu, 26 Aug 2004 15:55:51 -0400
From: John C Klensin <[email-masked]>
To: [email-masked]
Subject: draft-moore-mail-nr-fields-00
Sender: [email-masked]

Keith,

Regardless of the merits of this idea has it been introduced as 
part of the original email model, I see several problems.  At 
minimum, I believe a future version should respond to them.  At 
worst, it turns this, on balance, into a Bad Idea:

(1) Most of the functionality, for most cases, appears to be 
addressed by
   Reply-to: address1, address2, address3,...
Now, while that has rarely been well-implemented, it has been in 
the standard since 822 or earlier.  It is hard for me to see an 
argument that a newer set of headers would work significantly 
better.

(2) Your legacy discussion fails to discuss the interactions 
among a Reply-to field that is present and these new fields if 
they are also present.  It seems to me that could get 
complicated.

(3) To the extent to which this and Reply-to overlap, it gives 
us two different ways to do essentially the same thing.  That is 
rarely, if ever, a good idea.

(4) If these are sent out from a system that has no idea whether 
the recipient will honor or ignore them, it seems to me that we 
introduce the potential for huge levels of confusion and unmet 
expectations.   That isn't a showstopper, but argues against 
this, IMO, unless it is really clear that there would be 
significant benefits.

    john

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Date: Thu, 26 Aug 2004 10:47:36 +0200
Lines: 26
Cancel-Lock: sha1:u+rngUQwF32LoM2YDaJ+3fxwap8=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> I'm curious to know what people think of this idea.

I like the way it behave on legacy clients.  In comparison,
Mail-Followup-To and Mail-Reply-To doesn't work if a client isn't
aware of those headers.

I dislike that it require modification to the "submission" phase.  I
think one could argue that it is _more_ difficult to change submission
agents ("sendmail -t") AND all mail clients that do SMTP internally,
than it is to change only mail clients to support the MFT/MRT
approach.

Further, either the proposal or I am missing something: How can the
sender of a messages indicate that she _herself_ doesn't want to
receive any further replies?

I believe avoiding private e-mail copies to yourself is probably the
primary reason people use Mail-Followup-To and Mail-Reply-To.  If
NoReply do not support this mode, people will continue to use MFT/MRT.
In that case, it might be better to standardize MFT/MRT instead, but
perhaps the "Not Invented Here" syndrome prevent that.

Thanks,
Simon

Date: Wed, 25 Aug 2004 21:35:07 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Subject: [Fwd: I-D ACTION:draft-moore-mail-nr-fields-00.txt]
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------090207040906050605000906
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

here's something new to argue about :)

I'm curious to know what people think of this idea.

Keith

--------------090207040906050605000906
Content-Type: message/rfc822;
 name="I-D ACTION:draft-moore-mail-nr-fields-00.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="I-D ACTION:draft-moore-mail-nr-fields-00.txt"

To: [email-masked]
From: [email-masked]
Date: Wed, 25 Aug 2004 16:15:28 -0400
Subject: I-D ACTION:draft-moore-mail-nr-fields-00.txt
Reply-To: [email-masked]
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: NoReply Header Fields for Internet Mail
	Author(s)	: K. Moore
	Filename	: draft-moore-mail-nr-fields-00.txt
	Pages		: 13
	Date		: 2004-8-25
	
This memo introduces new header fields for use in Internet email, to
allow the author of a message to specify primary (To) and secondary (Cc)
recipients who, in the author's opinion, should not be recipients of a
reply to the message.  These new fields are designed to be highly
compatible with pre-existing mail handling programs, and to facilitate a
graceful transition.

If this document is favorably received, it is the author's intent to
submit this document (as an individual submission) for consideration for
Proposed Standard status.  Unless IESG directs otherwise, public
discussion of this document should be on the [email-masked] mailing
list.  Comments may also be sent to the author.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-moore-mail-nr-fields-00.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of the message.  
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-moore-mail-nr-fields-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-moore-mail-nr-fields-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body; access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID: <[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-moore-mail-nr-fields-00.txt

--OtherAccess
Content-Type: Message/External-body; name="draft-moore-mail-nr-fields-00.txt";
	site="ftp.ietf.org"; access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID: <[email-masked]>

--OtherAccess--

--NextPart
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/i-d-announce

--NextPart--

--------------090207040906050605000906--

Date: Wed, 25 Aug 2004 12:27:27 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

  > What bothers me (and I accept this may be a purely theoretical
> objection) is the potential for effectively merging and mixing of
> disparate mbox files.

it's been going on for years, due to NFS-mounted mail spool directories 
being written to by /bin/mail (and other) delivery programs and POP 
servers and user agents on various platforms.  not only were there 
slight differences in mbox format, there were several different and 
incompatible styles of locking. and yes, it caused problems. defining a 
MIME label for mbox isn't going to change any of this.

now if I were to write a recommendation for how to write mbox files,
I'd say that all body parts MUST be encoded if necessary to prevent
the characters "From" from appearing at the beginning of a line.
I'd also say that Content-Length MUST NOT be written, that From_ dates 
have to be in exactly such-and-such a format, that Return-Path fields 
are REQUIRED when storing a message received by SMTP, even if they're 
redundant with From_ lines, etc.  but that's not the problem we're 
currently trying to solve.

To: [email-masked]
Xref: clerew local.mime:3365
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Wed, 25 Aug 2004 13:04:53 GMT
Lines: 40
Sender: [email-masked]

In <20040825030307.GA9134@ender> Laird Breyer <[email-masked]> writes:

>It is one thing to send an mbox file to somebody, who saves it to disk
>as a separate file, which can be opened/read/written, or
>converted/imported to another format etc. This sort of thing has been
>done in the past, and works quite well by using
>application/octet-stream.

>However, unlike MIME, mbox is not a recursive format. It is a linear
>list of "blobs'. Thus if you send an mbox attachment without encoding
>the delimiters within the attachment (ie the "From_" lines etc.), then
>the common practice of appending such a message to the mail spool file
>can automatically destroy the original structure.

Not necessarily. The system that received the mail (containing the
application/mbox embedded within it) will presumably '>'-stuff all the
"From " lines before delivering it to its own mbox (or alternatively it
will insert a Content-Length header). A system that does not do one of
these things when delivering messages to its mbox is already severely
broken.

However, when you come to read the message with a view to extracting the
individual messages within the embedded mbox, then you find it has been
corrupted by all the '>' stuffing :-( . So you have to try to reverse the
stuffing, which may give rise to other errors.

So maybe you are right, that these things SHOULD/MUST always be base64
encoded.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 25 Aug 2004 13:03:08 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 24 2004, Charles Lindsey wrote:

> >- recommend base64 encoding as a default for transport of mbox files 
> >over email, since the encoder typically has no idea about the message 
> >contents and whether they might contain binary data.
> 
> No. In the common case where the 'blobs' are in fact RFC 2822 + MIME
> messages, they will already contain their own CTEs. And if they
> occasionally contain raw binary, then the sender (who knows where he got
> them from) is in the best position to decide whether an overall base64 is
> necessary.

What bothers me (and I accept this may be a purely theoretical
objection) is the potential for effectively merging and mixing of
disparate mbox files.

It is one thing to send an mbox file to somebody, who saves it to disk
as a separate file, which can be opened/read/written, or
converted/imported to another format etc. This sort of thing has been
done in the past, and works quite well by using
application/octet-stream.

However, unlike MIME, mbox is not a recursive format. It is a linear
list of "blobs'. Thus if you send an mbox attachment without encoding
the delimiters within the attachment (ie the "From_" lines etc.), then
the common practice of appending such a message to the mail spool file
can automatically destroy the original structure.

Instead of seeing a single message which contains an mbox attachment,
the appended spool is now seen to contain several messages, first the
actual message received, followed by the messages belonging to the
mbox attachment, as if they had been received individually.

[message 1 | message 2] + (message3 + attachment[message 4 | message 5])

=

[message 1 | message 2 | message 3 | message 4 | message 5]

Coping with such a case means making mbox parsers much smarter than
they currently are, or else making mbox writers smarter (so that the spool
is unambiguously appended).

Either way, this is a new difficulty I think, specifically introduced
by unencoded mbox file attachments (because the "From_" delimiters of
the attachment bleed through to the outer mbox containing the received
message). It can be prevented simply by making sure that "From_" 
lines in the attachment are unrecognizable (encoded).

Another consequence of this difficulty is that you can have mbox files
whose blob delimiters suddenly change partway through, when the
conventions used by an unencoded, attached, mbox file take over
temporarily. This can cause more trouble for existing mbox parsers. 

-- 
Laird Breyer.

Date: Tue, 24 Aug 2004 18:00:38 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Bruce Lilly <[email-masked]>
Cc: Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> Keith Moore wrote:

> > I don't believe that the creation of a new content-type is likely to
> > result in a lot of new mbox readers or writers.

> There is however a new issue, viz. that readers and writers are likely
> to be on separate systems, with possibly different conventions.

Well, I can't speak for your experience, but mine has been that once you move
past the very common case of a single piece of software that contains both a
reader and a writer, the next most common case is the use of mbox as a means of
exchanging material between different systems. The reason for this is simple:
It is very common for software to provide the ability to read mbox files, so if
you want to export, say, a VMS MAIL mailbox, mbox is a pretty reasonable
exchange format to use.

It would have been nice if multipart/digest had assumed this role, but in
practice it has not.

I don't expect a media type label to change this reality much if at all.

> > And I do believe that
> > whatever new software is written to read or write mbox files will be
> > similar to existing software that reads and writes mbox files.  If most
> > existing software doesn't need knobs to deal with format variation, I
> > don't suppose that most new software will need it either.

> The UW imap formats document mentioned earlier in this discussion
> lists at least 5 distinct flat file formats, identified and parsed
> separately, with variations ("knobs") in the two areas already
> specifically suggested for parameters, namely separator line format
> and line ending conventions.

UW IMAP supports several formats that have nothing to do with any variant of
mbox format out of the box, and this doesn't even take the various drivers
that have been written to handle even more bizzarre stuff.

				Ned

Date: Tue, 24 Aug 2004 17:55:31 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> Keith Moore <[email-masked]> writes:

> >> But it is too pessimistic to predict that they will not be used/generated
> >> correctly. Why ever not?

> >because such distinctions are only rarely recognzied by existing
> >mbox-reading software - and are of even less consequence to humans.

> But we are not talking about existing mbox-reading software (which is
> either tailored to some particular mbox format, or else tries to cover the
> most general case it can imagine).

Sure we are. Having a label mostly makes things easier to get the content to
and from such software.

> We are talking about a new application type, which we presume some people
> will write new software to generate/accept (otherwise there is little
> point in having the application type).

On the contrary, the main reason for having it is to make existing readers and
writers work better. Although some software will undoubtedly get written in due
course, it is unlikely in the extreme that a media type label (or lack of one)
will encourage (or impede) such development.

This is just another way of letting the mostly unachievable best be the enemy
of the plenty good enough, something I observe happening over and over and over
again in this discussion.

				Ned

Date: Tue, 24 Aug 2004 17:53:06 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Keith Moore <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> > >- my preference is to not have parameters describing format variants at
> > >all, as they're highly unlikely to be used correctly and even less
> > >likely to be generated correctly.  if you must have parameters, make
> > >them optional.
> >
> > I agree that the parameters should be optional (the sender might not know,
> > or he might have constructed the mbox by concatenating a bunch of other
> > mboxen of varied and dubious provenance).
> >
> > But it is too pessimistic to predict that they will not be used/generated
> > correctly. Why ever not?

> because such distinctions are only rarely recognzied by existing
> mbox-reading software - and are of even less consequence to humans.

> having more knobs introduces the potential for those knobs to be set
> incorrectly.  this can cause more interop problems than not having the
> knobs in the first place.

Exactly my point. I've been guilty of designing systems with too many
parameters myself; it is a mistake that's easy to make but a mistake
nevertheless.

> > >- recommend base64 encoding as a default for transport of mbox files
> > >over email, since the encoder typically has no idea about the message
> > >contents and whether they might contain binary data.
> >
> > No. In the common case where the 'blobs' are in fact RFC 2822 + MIME
> > messages, they will already contain their own CTEs. And if they
> > occasionally contain raw binary, then the sender (who knows where he got
> > them from) is in the best position to decide whether an overall base64 is
> > necessary.

> I said "reccommend...as a default" not "require".  If the sending MUA wants
> to scan the entire message to determine whether q-p or 8bit or 7bit is more
> appropriate, it is free to do so.

I guess I have no problem with recommending encoding as long as it isn't
required.

			Ned

Date: Tue, 24 Aug 2004 17:49:04 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> Keith Moore <[email-masked]> writes:

> >- my preference is to not have parameters describing format variants at
> >all, as they're highly unlikely to be used correctly and even less
> >likely to be generated correctly.  if you must have parameters, make
> >them optional.

> I agree that the parameters should be optional (the sender might not know,
> or he might have constructed the mbox by concatenating a bunch of other
> mboxen of varied and dubious provenance).

> But it is too pessimistic to predict that they will not be used/generated
> correctly. Why ever not?

Because that's what past experience with similar schemes indicates will happen.

> And even if not, then you know exactly where to
> point your finger.

Which sometimes works and other times only serves to get your finger bitten
off. Being right isn't always sufficient. For this reason it is much, much
better to eliminate as many potential casues for interop failure as possible.

> But the common case will be where the sender is transmitting his own
> inbox, or some folder from his own machine, which will have been generated
> previously by his own MDA or MUA; so it will be in a consistent and known
> format. So why on earth not let him describe that format in a parameter if
> he knows it?

For all the reason I have previously given.

> >- recommend base64 encoding as a default for transport of mbox files
> >over email, since the encoder typically has no idea about the message
> >contents and whether they might contain binary data.

> No. In the common case where the 'blobs' are in fact RFC 2822 + MIME
> messages, they will already contain their own CTEs. And if they
> occasionally contain raw binary, then the sender (who knows where he got
> them from) is in the best position to decide whether an overall base64 is
> necessary.

This I agree with. Encoding isn't always necessary or even appropriate.

				Ned

Date: Tue, 24 Aug 2004 15:06:30 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

>>I don't believe that the creation of a new content-type is likely to
>>result in a lot of new mbox readers or writers.
> 
> 
> There is however a new issue, viz. that readers and writers are likely
> to be on separate systems, with possibly different conventions.

that's not really a new issue, as readers and writers have been on 
separate systems for years, for two reasons.  one is that mbox is
the closest thing that exists to a de facto standard for exchanging
mail folders between systems or MUAs.  the second is that it's been 
common for various kinds of systems with different mbox variants to 
share a common NFS-mounted mail spool directory.

>>And I do believe that
>>whatever new software is written to read or write mbox files will be
>>similar to existing software that reads and writes mbox files.  If most
>>existing software doesn't need knobs to deal with format variation, I
>>don't suppose that most new software will need it either.
> 
> 
> The UW imap formats document mentioned earlier in this discussion
> lists at least 5 distinct flat file formats, identified and parsed
> separately, with variations ("knobs") in the two areas already
> specifically suggested for parameters, namely separator line format
> and line ending conventions.

there's a difference between 5 flat file formats and 5 flat formats that
all happen to resemble mbox.   there are other flat mailbox file formats 
that are completely different, including MM, BABYL, MMDF, and a couple 
of others whose names I don't recall.

Date: Tue, 24 Aug 2004 14:35:32 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Keith Moore wrote:

> I don't believe that the creation of a new content-type is likely to
> result in a lot of new mbox readers or writers.

There is however a new issue, viz. that readers and writers are likely
to be on separate systems, with possibly different conventions.

> And I do believe that
> whatever new software is written to read or write mbox files will be
> similar to existing software that reads and writes mbox files.  If most
> existing software doesn't need knobs to deal with format variation, I
> don't suppose that most new software will need it either.

The UW imap formats document mentioned earlier in this discussion
lists at least 5 distinct flat file formats, identified and parsed
separately, with variations ("knobs") in the two areas already
specifically suggested for parameters, namely separator line format
and line ending conventions.

Date: Tue, 24 Aug 2004 12:50:09 -0400
From: Keith Moore <[email-masked]>
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

> But we are not talking about existing mbox-reading software (which is
> either tailored to some particular mbox format, or else tries to cover the
> most general case it can imagine).

No, that's not what "we" are talking about.  And I don't claim to know 
what _you_ are talking about.

I don't believe that the creation of a new content-type is likely to 
result in a lot of new mbox readers or writers.  And I do believe that 
whatever new software is written to read or write mbox files will be 
similar to existing software that reads and writes mbox files.  If most 
existing software doesn't need knobs to deal with format variation, I 
don't suppose that most new software will need it either.

To: [email-masked]
Xref: clerew local.mime:3356
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Tue, 24 Aug 2004 13:14:21 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> But it is too pessimistic to predict that they will not be used/generated
>> correctly. Why ever not? 

>because such distinctions are only rarely recognzied by existing 
>mbox-reading software - and are of even less consequence to humans.

But we are not talking about existing mbox-reading software (which is
either tailored to some particular mbox format, or else tries to cover the
most general case it can imagine).

We are talking about a new application type, which we presume some people
will write new software to generate/accept (otherwise there is little
point in having the application type).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 23 Aug 2004 11:45:11 -0400
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

> >- my preference is to not have parameters describing format variants at 
> >all, as they're highly unlikely to be used correctly and even less 
> >likely to be generated correctly.  if you must have parameters, make 
> >them optional.
> 
> I agree that the parameters should be optional (the sender might not know,
> or he might have constructed the mbox by concatenating a bunch of other
> mboxen of varied and dubious provenance).
> 
> But it is too pessimistic to predict that they will not be used/generated
> correctly. Why ever not? 

because such distinctions are only rarely recognzied by existing 
mbox-reading software - and are of even less consequence to humans.

having more knobs introduces the potential for those knobs to be set
incorrectly.  this can cause more interop problems than not having the
knobs in the first place.

> >- recommend base64 encoding as a default for transport of mbox files 
> >over email, since the encoder typically has no idea about the message 
> >contents and whether they might contain binary data.
> 
> No. In the common case where the 'blobs' are in fact RFC 2822 + MIME
> messages, they will already contain their own CTEs. And if they
> occasionally contain raw binary, then the sender (who knows where he got
> them from) is in the best position to decide whether an overall base64 is
> necessary.

I said "reccommend...as a default" not "require".  If the sending MUA wants 
to scan the entire message to determine whether q-p or 8bit or 7bit is more 
appropriate, it is free to do so.

Keith

To: [email-masked]
Xref: clerew local.mime:3354
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Mon, 23 Aug 2004 13:24:54 GMT
Lines: 42
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>- my preference is to not have parameters describing format variants at 
>all, as they're highly unlikely to be used correctly and even less 
>likely to be generated correctly.  if you must have parameters, make 
>them optional.

I agree that the parameters should be optional (the sender might not know,
or he might have constructed the mbox by concatenating a bunch of other
mboxen of varied and dubious provenance).

But it is too pessimistic to predict that they will not be used/generated
correctly. Why ever not? And even if not, then you know exactly where to
point your finger.

But the common case will be where the sender is transmitting his own
inbox, or some folder from his own machine, which will have been generated
previously by his own MDA or MUA; so it will be in a consistent and known
format. So why on earth not let him describe that format in a parameter if
he knows it?

>- recommend base64 encoding as a default for transport of mbox files 
>over email, since the encoder typically has no idea about the message 
>contents and whether they might contain binary data.

No. In the common case where the 'blobs' are in fact RFC 2822 + MIME
messages, they will already contain their own CTEs. And if they
occasionally contain raw binary, then the sender (who knows where he got
them from) is in the best position to decide whether an overall base64 is
necessary.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 22 Aug 2004 10:58:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
cc: [email-masked]
Subject: Re: Diabling Replies
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Dave Crocker)  wrote on 21.08.04 in <[email-masked]>:

TH>> My understanding is that these tertiary recipients would only see the
TH>> initial message of the conversation. It's a visible form of Bcc.

> To nit pick: Then it's not a bcc in any fashion. Current bcc's can reply
> (though they are not supposed to) and current bcc's are not seen by the
> to/cc recipients.
>
> Still, there is certainly nothing wrong with having a visible set of
> addressees who are to be copied but who also are being told to keep
> their digital mouths shut.

Really, I fail to understand what is so hard to understand here, but it  
seems you consistently refuse to get the point.

Let's try again.

The scenario is this:

From: A
Subject: Current state of project X
To: B, C
Bcc: Boss

with the (intended) result that replies-to-all from B and C will not reach  
Boss.

*Nobody* is "told to keep their digital mouths shut". B and C are told  
"don't bother Boss, let's keep any discussion to A, B, and C".

Boss, of course, is free to decide to start a discussion of his own; it  
just isn't expected that he'll want to - the expectation is that he'll  
wait for the next status report.

MfG Kai

Date: Sat, 21 Aug 2004 16:18:06 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Tony Hansen <[email-masked]>, Jacob Palme <[email-masked]>
CC: [email-masked]
Subject: Re: Diabling Replies
Sender: [email-masked]

Tony,

TH> My understanding is that these tertiary recipients would only see the
TH> initial message of the conversation. It's a visible form of Bcc.

To nit pick: Then it's not a bcc in any fashion. Current bcc's can reply
(though they are not supposed to) and current bcc's are not seen by the
to/cc recipients.

Still, there is certainly nothing wrong with having a visible set of
addressees who are to be copied but who also are being told to keep
their digital mouths shut.

That said, I think this wanders down the path of the jillions of
interesting ideas one might implement, all of which sound interesting
and only some of which are useful to the general Internet user
population.

What's the market pull for this one?

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Sat, 21 Aug 2004 16:58:33 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked], [email-masked]
Subject: Re: The job of an MSA
Sender: [email-masked]

[email-masked] wrote:

> First, RFC 2746 is quite clear: Port 25 MAY be used for SUBMIT if for whatever
> reason 587 cannot be used.

2476 -- otherwise OK.

> More generally, what port something is run on is largely irrelevant; the
> important issues are service and protocol semantics.

If the protocols are to be treated as fundamentally different -- and 2476
says that an MSA runs ESMTP with some restrictions (N.B. not extensions) --
then that raises the question of how a client which connects to a server
on port 25 is able to determine whether that server is an MSA or an MTA.

> Of course this doesn't mean someone won't attempt to use SUBMIT-specific
> extensions in regular SMTP.

At the moment, as I understand it, there are no such "extensions", there
are only "restrictions" [RFC 2476 section 3.1].

> But just because some clueless ISP engages in
> egregiously bad practice is no reason to break our protocol specifications. I
> am dubious about the value of such an extension in any case for all the reason
> I've previsouly given; I would vehemently oppose making it part of regular
> SMTP.

It may be the case that presence of one or more hypothetical submission-
specific extensions might serve to distinguish an MSA from an MTA. That
would mean that the presence of such an extension would indicate that the
server in question is an MSA (running ESMTP), not an MTA (running ESMTP).

Of course if a particular MSA doesn't support any of the hypothetical
MSA-specific extensions, there is still the issue of how a client is
supposed to be able to determine whether it has connected to an MTA or
an MSA.  What do you suggest?

Date: Sat, 21 Aug 2004 15:00:52 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: 822 <[email-masked]>, [email-masked]
Subject: [Fwd: MSAs (RFC 2476), RFC 2822, and the Date field]
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------050909080108070104090807
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Oops, forgot the list copies...

--------------050909080108070104090807
Content-Type: message/rfc822;
 name*0*=ISO-8859-1'en-us'MSAs%20%28RFC%202476%29%2C%20RFC%202822%2C%20and
 name*1*=%20the%20Date%20field
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename*0*=ISO-8859-1'en-us'MSAs%20%28RFC%202476%29%2C%20RFC%202822%2C%20
 filename*1*=and%20the%20Date%20field

Message-ID: <[email-masked]>
Date: Thu, 19 Aug 2004 10:21:17 -0400
From: Bruce Lilly <[email-masked]>
Reply-To:  [email-masked]
Organization: Bruce Lilly
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.2) Gecko/20040803
X-Accept-Language: en-us
MIME-Version: 1.0
To:  [email-masked]
CC: Jacob Palme <[email-masked]>
Subject: MSAs (RFC 2476), RFC 2822, and the Date field
References: <[email-masked]> <[email-masked]> <p0611040bbd4902ddb2d8@[192 .168.168.22]> <[email-masked]> <p0611040fbd492cde8b38@[192.168.168.22]> <[email-masked]>
In-Reply-To: <[email-masked]>
X-Enigmail-Version: 0.85.0.0
X-Enigmail-Supports: pgp-inline, pgp-mime
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

[email-masked] wrote:

[quoting Jacob Palme]
>>like for examples is already accepted that the initital MTA
>>may add a "Date" header for a message lacking such a
>>header.
> 
> 
> Actually, it isn't accepted for an MTA to do this. It can be done by an MSA,
> which is a different beast.

Note that RFC 2822 defines the semantics for the Date field as
representing the time of completion of composition of the message,
not the time of entry into the transport stream.  Of course RFC 2476
predates RFC 2822, and RFC 822 and earlier were not as precise
regarding the Date field semantics.  The next revision of RFC 2476
as it progresses along the Standards Track should probably address
that discrepancy.

--------------050909080108070104090807--

Date: Sat, 21 Aug 2004 07:45:36 -0700 (PDT)
From: [email-masked]
Subject: Re: The job of an MSA
To: Bruce Lilly <[email-masked]>
Cc: Tony Hansen <[email-masked]>, [email-masked]
Sender: [email-masked]

> > Given my druthers, I'd much rather see it be an extension only to the
> > MSA SUBMIT protocol and not one that is expected to be implemented in MTAs.

> For the issue of efficient transfer where there is a bottleneck
> (e.g. a dialup connection to an ISP), it might not be practical
> to restrict it to the SUBMIT protocol; not all ISPs operate
> servers on port 587.

First, RFC 2746 is quite clear: Port 25 MAY be used for SUBMIT if for whatever
reason 587 cannot be used.

More generally, what port something is run on is largely irrelevant; the
important issues are service and protocol semantics.

Of course this doesn't mean someone won't attempt to use SUBMIT-specific
extensions in regular SMTP. But just because some clueless ISP engages in
egregiously bad practice is no reason to break our protocol specifications. I
am dubious about the value of such an extension in any case for all the reason
I've previsouly given; I would vehemently oppose making it part of regular
SMTP.

				Ned

Date: Thu, 19 Aug 2004 00:43:24 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Keith Moore <[email-masked]>
Cc: Dave Crocker <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], Keith Moore <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

Keith, very nicely put. I am in complete agreement with everything you
say here.

				Ned

> On Aug 17, 2004, at 8:38 PM, Dave Crocker wrote:

> >  If someone can summarize where
> > this thread is going and/or should go, I'd appreciate it.

> my recommendation to the author -

> - do another edit on the draft, mostly for nits and clarity, and
> resubmit for Informational

> - don't try to specify the application/mbox format in this document at
> all.  instead, point to various published definitions and descriptions
> (man pages, web pages, whatever) such as exist.

> - for that matter, don't try to specify how best to generate mbox
> files.  that's out of scope.

> - explicitly point out that there are varieties in how the format has
> been implemented which can affect interoperability.  (this should not
> be a showstopper, as there are varieties in how almost any other
> MIME-labeled format is implemented which can affect interoperability.)

> - my preference is to not have parameters describing format variants at
> all, as they're highly unlikely to be used correctly and even less
> likely to be generated correctly.  if you must have parameters, make
> them optional.

> - you might want to point out that a wide variety of mbox-reading tools
> generally manage to make sense of mbox files despite the variations.
> and yes, they barf sometimes.  some implementations' heuristics are
> better than others.  again, this is also true of Word, PDF, JPEG,
> PostScript, or any other format you could care to look at.  it's useful
> to be able to distinguish mbox files from other kinds of files, even if
> saying something is an mbox file is not perfectly precise.

> - point out that the existence of an application/mbox label is not a
> recommendation that mbox files be used to transport groups of messages
> -  at least, not without first defining a new variant of mbox files
> that is transparent (no >From), reliable (no message boundary
> synchronization errors),  immune to variations in line terminator, and
> doesn't use character or line counts (no content-length).

> - recommend base64 encoding as a default for transport of mbox files
> over email, since the encoder typically has no idea about the message
> contents and whether they might contain binary data.

> my recommendation to everyone else -

> let's wait for the revision and look at it then.  meanwhile, let's
> realize

> - that having a way to label mbox fiiles is much more useful than not
> having one;

> - that perfectly precise labels for anything are almost nonexistent in
> practice;

> - that having an imprecise MIME label for mbox files probably won't
> degrade interop significantly over what it already is

> - that there are probably better uses of our time (individually or
> collectively) than to define the perfect label for mbox files.

> Keith

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ietf

Date: Wed, 18 Aug 2004 22:47:37 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Jacob Palme <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> At 16.32 +0100 04-08-18, Nick Ing-Simmons wrote:
> >The summary just posted explained that MTA has no bussiness
> >removing Bcc headers. It is a MUA function that happens, on
> >Unix, to be implemented by another instance of sendmail
> >binary invoked a particular way.

> This is obviously the consensus of the discussion.

> But it is not unreasonable that a consenting MUA and MTA
> may agree to move part of the burden of the MUA to the MTA,

It is not unreasonable for two entities to agree to such a thing. But these
entities are no longer what we call an MUA and MTA.

> like for examples is already accepted that the initital MTA
> may add a "Date" header for a message lacking such a
> header.

Actually, it isn't accepted for an MTA to do this. It can be done by an MSA,
which is a different beast.

> Since MTA meddling with Bcc is highly
> controversial, this should of course never be done unless
> there is explicit agreement between MUA and MTA to do it
> this way. "-t" might be a way of indicating such an
> agreement, but only works when submission is done by
> execution, which is probably not the most common method of
> MUA-MTA communication.

-t causes sendmail to in effect assume part of the role of an MUA. It most
moves things way past what an MTA can or should be doing.

> If there is to be a method for an MUA to ask an MTA to
> meddle with Bcc, then it is important for this
> communication between MUA and MTA is standardized, so that
> misunderstandings will not occur, since misunderstandings
> can result in embarassing situations.

It would be possible to define a means to do this as part of our MSA SUBMISSION
protocol, but as you say it would have to be negotiated and therefore would in
no way make the present behavior of agents that blithely assume the transport
infrastructure will take care of of bcc issues. These agents are simply broken
and need to be fixed.

And since such an extension would be just that: An extension, MUAs for
the forseeable future could not assume the extension's presence. This means
they'll have to have their own bcc code.

This in turn begs the question of whether or not there is any real advantage to
having such an extension. It turns out there is: An MUA that is either memory
or bandwidth limited could push the message splitting activities over to the
MSA server and avoid having to send the same message multiple times. The price
we pay for this is having code to handle bcc in two places - a considerable
increase in complexity.

> Do not interpret the above to mean that I feel such a
> standard to be very urgent or important, nor that I am
> willing to be editor of it.

I for one have a great deal of difficulty accessing the real need for various
extensions that attempt to cater to the needs of memory or bandwidth limited
clients. Sure, some of these are nobrainers: If you want a cell phone to be
able to forward a feature-length film you gonna need a forward-without-download
facility along the lines of what the LEMONADE WG is developing, and I don't see
that need changing in the forseeable future. But the need for many other
possible extensions, including this one, are not nearly as obvious. I'm just
not sure whre the sweet spot really is.

				Ned

Date: Wed, 18 Aug 2004 21:37:04 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Jacob Palme writes:
> I have not been very active in standards work recent years, but I 
> remember there has been discussion of writing a special standard on 
> what the first receiving MTA, when getting a message from an MUA, is 
> allowed to do with the message, such as adding "Date" and 
> "Message-ID". Was such a standard ever developed?

I guess 2476 section 8 may be what you remember.

Arnt

Date: Wed, 18 Aug 2004 18:08:52 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: The Bcc Issue
Sender: [email-masked]

At 16.32 +0100 04-08-18, Nick Ing-Simmons wrote:
>The summary just posted explained that MTA has no bussiness
>removing Bcc headers. It is a MUA function that happens, on
>Unix, to be implemented by another instance of sendmail
>binary invoked a particular way.

This is obviously the consensus of the discussion.

But it is not unreasonable that a consenting MUA and MTA
may agree to move part of the burden of the MUA to the MTA,
like for examples is already accepted that the initital MTA
may add a "Date" header for a message lacking such a
header. Since MTA meddling with Bcc is highly
controversial, this should of course never be done unless
there is explicit agreement between MUA and MTA to do it
this way. "-t" might be a way of indicating such an
agreement, but only works when submission is done by
execution, which is probably not the most common method of
MUA-MTA communication.

If there is to be a method for an MUA to ask an MTA to
meddle with Bcc, then it is important for this
communication between MUA and MTA is standardized, so that
misunderstandings will not occur, since misunderstandings
can result in embarassing situations.

Do not interpret the above to mean that I feel such a
standard to be very urgent or important, nor that I am
willing to be editor of it.

I have not been very active in standards work recent
years, but I remember there has been discussion of
writing a special standard on what the first receiving
MTA, when getting a message from an MUA, is allowed
to do with the message, such as adding "Date" and
"Message-ID". Was such a standard ever developed?
it should be the place for a discussion of a possible
"Bcc"-meddling ESMTP extension.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Wed, 18 Aug 2004 09:22:59 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Bill McQuillan <[email-masked]>
Cc: RFC822 <[email-masked]>
Sender: [email-masked]

> On Wed, 2004-08-18 at 03:36:24, Philip Hazel <[email-masked]> wrote:

> <snip>

> > With one possible exception, everybody's opinion was that an MTA has no
> > business messing with Bcc: header lines. Here is a quote from Tony
> > Hansen:

> >    "The MUA, however it is constituted, should handle all BCC processing,
> >    and the cloning of messages with and without the Bcc: header."

> <snip>

> I haven't seen it mentioned here yet, but shouldn't the rule be that the
> agent that extracts the contents of the Bcc: header field, along with To:
> and Cc:, in order to construct the 2821 RCPT TO: commands, is the agent that
> determines the contents and/or (lack of) presence of the Bcc: field in the
> message. Any agent that receives the recipient addresses via RCPT TO: can
> be assured that the decision has already been made.

The part here about receiving addresses via RCPT TO is certainly fine. I 
believe the part about agents which construct envelopes from headers is also OK
- it is certainly how I've implemented it. However, since in our model this
sort of processing happens within the MUA, it isn't clear to be we can, or for
that matter should, specify it.

				Ned

Date: Wed, 18 Aug 2004 12:02:22 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Excellent summary, Philip!

Just for completeness, here is an excerpt from the sendmail man page, 
describing the -t option.

      -t    Read message for recipients. To:, Cc:, and Bcc:  lines
            will  be  scanned for people to send to. The Bcc: line
            will be deleted before transmission. Any addresses  in
            the  argument list will be suppressed. The NoRecipien-
            tAction Processing Option can be used  to  change  the
            behaviour when no legal recipients are included in the
            message.

Note that this strictly affects how the sendmail program is used from 
the command line and reading from standard input, and does not affect 
how it works when reading input using the SMTP protocol. (There is NO 
sendmail option which tells it to look at To/Cc/Bcc when it is using the 
SMTP protocol.)

	Tony Hansen
	[email-masked]

Philip Hazel wrote:

> 
> Summary of the Bcc discussion: posted to exim-users, exim-dev, & ietf-822
> 
> There were a few responses on the exim-users mailing list. There was a
> longer discussion on the ietf-822 mailing list, to which a number of
> people who have worked in email protocol development for many years
> contributed. These included Keith Moore (author of several RFCs), Dave 
> Crocker (author of RFC 822), and Tony Hansen (another RFC author).
> 
> Some more data points were posted: Smail-3 behaves like Exim; Elm and 
> Mulberry behave like Pine (they do their own Bcc: processing).
> 
> 
> 1. The substantive point
> 
> With one possible exception, everybody's opinion was that an MTA has no
> business messing with Bcc: header lines. Here is a quote from Tony
> Hansen:
> 
>   "The MUA, however it is constituted, should handle all BCC processing,
>   and the cloning of messages with and without the Bcc: header."
> 
> This confirms my belief that Exim's default behaviour is the correct 
> one. I hope this particular issue is now laid to rest once and for all.
> I have saved all the responses just in case it arises again in the future.
> 
> The possible exception was a post containing this paragraph:
> 
>   The /usr/[s]bin/sendmail interface is a proprietary interface
>   introduced by Sendmail. So all other applications providing this
>   interface should be as close to Sendmail's behaviour as possible, i.e.
>   they should remove Bcc headers.
> 
> This is the "my country, right or wrong" argument. There was one
> response that added "... when -t is specified", and that is also my
> feeling. A call to /usr/sbin/sendmail without -t is not guaranteed to be
> a submission. Indeed, the post continued:
> 
>   Philosophically, the mailers are still doing the MUA's job by
>   sanitizing the message and (finally) handling it over to the message
>   router (MTA). It just happens that that part of Unix MUAs is usually
>   shipped along with the MTA.
> 
> and I would argue that without -t, you cannot, and indeed should not,
> assume that the sending entity is an MUA that is submitting a new 
> message (see further discussion below).
> 
> 
> 2. Other discussion
> 
> Some interesting points of clarification came up in the discussion. The 
> functions of mail handling are now seen as split into four distinct 
> parts, as follows:
> 
>   MUA -> MSA -> MTA -> MDA -> MUA
> 
> (user agent, submission agent, transfer agent, delivery agent). There is 
> nothing to say that each of these functional distinctions has to be 
> implemented by a different program. Thus, it is generally agreed that a
> program handling a local incoming message using the "-t" command line
> option is acting partly as an MUA (constructing the envelope) and partly
> as an MSA (accepting a new message submission). If it goes on to deliver 
> the message over SMTP it is performing an MTA function. If it delivers 
> into a local mailbox it is performing an MDA function.
> 
> There was some opinion expressed that an MSA might be somewhere that 
> Bcc: header lines could be processed. As this is not sanctioned by RFC 
> 2476 (for MUA->MSA delivery over a network), it would be necessary to
> define an extension to the SUBMIT protocol so that the MUA could request
> the MSA to do that processing. In the case of a submission via the
> command line, the -t option is already available for this purpose.
> 
> When -t is not used, a general-purpose MTA such as Exim cannot guess
> that it is being called as an MSA. It has to be configured as such, for
> example by testing conditions such as "submitted from a host on the
> local LAN" or "submitted via port 587". If an extension to SUBMIT were 
> defined, another test could be "requested by the SUBMIT protocol". (It
> is quite easy to implement such tests in Exim and (for example) remove
> Bcc: lines based on the result, but this has to be an explicit
> configuration setting; it cannot be defaulted.)
> 
> Note that "submitted via the command line" (in the absence of -t) is not
> good enough as a test for an MSA. Such messages may have come from
> another email regime (e.g. UUCP - some people do still use it) or from a
> virus scanner or a mailing list exploder.
> 
> 
> 3. Uses of Bcc
> 
> As part of the discussion, a list of uses of bcc was enumerated. There 
> are at least four major ones:
> 
> (a) As an instruction from the end user to the MUA to send a copy of the
>     message to one or more addresses, without telling the To: and Cc:
>     recipients.
> 
> (b) When present in a copies sent to bcc recipients, to inform them why
>     they received the message (and, if more than one bcc recipient is
>     listed, to inform them about additional bcc addressees). This should
>     alert bcc recipients to take care when replying.
> 
> (c) To prevent unwanted group replies where the sender wants to send a
>     copy of a message to someone (whether or not the other recipients
>     should know about this), but not subject that person to lots of
>     replies from the other recipients who blindly "reply-to-all".
> 
> (d) When saved in the sender's sent-mail folder, a Bcc: header acts as a
>     record of who the message was sent to.
> 
> Is that overloaded or is that overloaded? :-)
> 
> Regards,
> Philip
> 

Date: Wed, 18 Aug 2004 08:50:18 -0700
From: Bill McQuillan <[email-masked]>
To: RFC822 <[email-masked]>
Subject: Re: The Bcc Issue
Sender: [email-masked]

On Wed, 2004-08-18 at 03:36:24, Philip Hazel <[email-masked]> wrote:

<snip>

> With one possible exception, everybody's opinion was that an MTA has no
> business messing with Bcc: header lines. Here is a quote from Tony
> Hansen:

>    "The MUA, however it is constituted, should handle all BCC processing,
>    and the cloning of messages with and without the Bcc: header."

<snip>

I haven't seen it mentioned here yet, but shouldn't the rule be that the
agent that extracts the contents of the Bcc: header field, along with To:
and Cc:, in order to construct the 2821 RCPT TO: commands, is the agent that
determines the contents and/or (lack of) presence of the Bcc: field in the
message. Any agent that receives the recipient addresses via RCPT TO: can
be assured that the decision has already been made.

-- 
Bill McQuillan <[email-masked]>

Cc: [email-masked]
Subject: Re: The Bcc Issue
To: [email-masked]
Date: Wed, 18 Aug 2004 16:32:26 +0100
From: Nick Ing-Simmons <[email-masked]>
Reply-To: Nick Ing-Simmons <[email-masked]>
Sender: [email-masked]

Jacob Palme <[email-masked]> writes:
>At 20.56 +0200 04-08-17, Kai Henningsen wrote:
>>... when using the -t option.
>
>I do not understand this "-t" option. "-t" looks to me like
>a Unix shell command parameter. 

It is.

>I thought that MUAs
>communicated with MTAs for outgoing mail using the SMTP
>protocol or a variant of SMTP adapted for communication
>between an MUA or the first MTA, and not by using a
>unix shell interface.

They do.

But in Unix world MUA-Part1 can commumicate with MUA-Part2
via a pipe, like:

cat message | /usr/lib/sendmail -t

This is using cat/sendmail pair as a MUA.

sendmail instance could do MTA job as well, 
or could open socket to ANOTHER instance of sendmail as MTA
and talk SMTP over that socket.

>
>Is "-t" an ESTMP facility? 

No.

>How, otherwise, can the MUA tell
>the MTA that it wants the MTA to remove Bcc headers from
>non-Bcc recipients?

The summary just posted explained that MTA has no bussiness 
removing Bcc headers. It is a MUA function that happens, on
Unix, to be implemented by another instance of sendmail
binary invoked a particular way.

Date: Wed, 18 Aug 2004 08:10:33 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Jacob Palme <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> At 20.56 +0200 04-08-17, Kai Henningsen wrote:
> >... when using the -t option.

> I do not understand this "-t" option. "-t" looks to me like
> a Unix shell command parameter. I thought that MUAs
> communicated with MTAs for outgoing mail using the SMTP
> protocol or a variant of SMTP adapted for communication
> between an MUA or the first MTA, and not by using a
> unix shell interface.

Some do, many do not. The lack of a standarized submission mechanism for most
of the time RFC822/SMTP has existed has led to a large number of ad-hoc schemes
being developed. Some are true MSA mechanisms while others actually implement a
split MUA. (And some are configurable and can do both.)

Some schemes are proprietary and only operate between one specific pair of
entities, others are more open. Submission can be done over protocol (not
necessarily TCP - I've seen DECnet, X.25, and even X.29 used), it can be done
via a subroutine library, or it can be done through a pipe. I'm sure there are
many other variants as well, and I haven't even touched on non-RFC822-based
email here.

				Ned

Date: Tue, 17 Aug 2004 21:20:34 -0700
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: John C Klensin <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Reply-To: Dave Crocker <[email-masked]>
Sender: [email-masked]

John,

JCK> Does that agree with your analysis, at least to a first order
JCK> approximation?

  well (harumph!) i can't let the question stand without answering it...

  I trust the assessments you folks have been making about the document,
  and I have not seen counter-arguments that looked at all persuasive.

  I also have not read the spec.
  
d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

Date: Wed, 18 Aug 2004 19:54:09 +0530
From: Venkat Ramisetty <[email-masked]>
To: [email-masked]
Subject: how to handle microsoft OLE messages?
Sender: [email-masked]

Hi,

I would like to know whether we can map a message composed with OLE 
attachments to MIME complaint?

Regards,
Venkat

Date: Wed, 18 Aug 2004 14:40:22 +0100 (BST)
From: Philip Hazel <[email-masked]>
To: Jacob Palme <[email-masked]>
cc: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

On Wed, 18 Aug 2004, Jacob Palme wrote:

> I do not understand this "-t" option. "-t" looks to me like
> a Unix shell command parameter. I thought that MUAs
> communicated with MTAs for outgoing mail using the SMTP
> protocol or a variant of SMTP adapted for communication
> between an MUA or the first MTA, and not by using a
> unix shell interface.

Not true. Some use SMTP, others call the MTA directly by running the 
command /usr/[s]bin/sendmail in a shell interface. (That was, I believe, 
the original method, before we had MUAs that ran on PCs.)

> Is "-t" an ESTMP facility? How, otherwise, can the MUA tell
> the MTA that it wants the MTA to remove Bcc headers from
> non-Bcc recipients?

Using SMTP (or the SUBMIT variant), it can't. That's the whole point.

Regards,
Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Wed, 18 Aug 2004 09:47:45 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Jacob Palme <[email-masked]>
CC: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Jacob Palme wrote:
> At 20.56 +0200 04-08-17, Kai Henningsen wrote:
> 
>>... when using the -t option.
> 
> 
> I do not understand this "-t" option. "-t" looks to me like
> a Unix shell command parameter.

It is an argument supplied to the program when executed.

> I thought that MUAs
> communicated with MTAs for outgoing mail using the SMTP
> protocol or a variant of SMTP adapted for communication
> between an MUA or the first MTA, and not by using a
> unix shell interface.

That's one way. Running an external program (usually on the
same host) is another method sometimes used, particularly by
older MUAs.

> Is "-t" an ESTMP facility?

No.

> How, otherwise, can the MUA tell
> the MTA that it wants the MTA to remove Bcc headers from
> non-Bcc recipients?

By invoking the external program with the specified argument
and providing the message as input.

Of course, not everybody wants to run (much less administer
and maintain) sendmail on every host with an MUA.  Nor is it
practical when run on a different host via remote execution
(user names.ids might not match, there are security implications,
etc.).

Date: Wed, 18 Aug 2004 09:43:03 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: "Eric A. Hall" <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Eric A. Hall wrote:
> 
> On 8/17/2004 2:09 PM, John C Klensin wrote:
> 
> 
>>To be clear about this, I think there are three choices which we
>>might prefer in descending order:
>>
>>	(1) There is a single canonical "wire" format in which
>>	these things are transmitted.
> 
> 
> Such a specification would surely dictate "a series of message/rfc822
> objects". But if we were to require that end-points perform conversion
> into a neutral form, we might as well go the whole nickel and just say
> "use multipart/digest"

So far, so good.

> We'd still be in the same place we're at today, of course, because HTTP
> and filesystem transfers aren't going to do mbox->digest conversion any
> more than they are going to do EOL conversion or Content-Length calcs.
> We'd still have opaque messaging databases that people wanted to transfer,
> search, open and otherwise automate, but couldn't.

With the proposal as it stands, you'd also still have an opaque
blob, with the same problems.  But not with multipart/digest. And
why do you claim that there will not be any conversions; multipart/digest
is the lingua franca of message collections and it is unrealistic
to expect that some system is going to be able to reliably handle
an opaque blob constructed on a foreign system, in the face of
common transport modifications, and with no information regarding
the content of the blob other than an extremely vague label.

>>	(3) These might as well be sent with
>>	application/octet-stream, with optional file name, etc.,
>>	information.
> 
> 
> That's where we are now and we already now that's not working.

Since you have opposed parameters, what you are left with is
merely a subtype tag change from "octet-stream" to "mbox".  Now
could you please explain exactly what "'s not working" about
that, and precisely how *merely* changing the subtype tag to
"mbox", with no supplementary information is going to solve
whatever those problems are. In other words, what *exactly* is
the problem that you're trying to solve that cannot be addressed
by existing media types, and *precisely* how will the proposal
solve that problem?

Date: Wed, 18 Aug 2004 15:00:58 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: The Bcc Issue
Sender: [email-masked]

At 20.56 +0200 04-08-17, Kai Henningsen wrote:
>... when using the -t option.

I do not understand this "-t" option. "-t" looks to me like
a Unix shell command parameter. I thought that MUAs
communicated with MTAs for outgoing mail using the SMTP
protocol or a variant of SMTP adapted for communication
between an MUA or the first MTA, and not by using a
unix shell interface.

Is "-t" an ESTMP facility? How, otherwise, can the MUA tell
the MTA that it wants the MTA to remove Bcc headers from
non-Bcc recipients?
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Wed, 18 Aug 2004 11:36:24 +0100 (BST)
From: Philip Hazel <[email-masked]>
To: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Summary of the Bcc discussion: posted to exim-users, exim-dev, & ietf-822

There were a few responses on the exim-users mailing list. There was a
longer discussion on the ietf-822 mailing list, to which a number of
people who have worked in email protocol development for many years
contributed. These included Keith Moore (author of several RFCs), Dave 
Crocker (author of RFC 822), and Tony Hansen (another RFC author).

Some more data points were posted: Smail-3 behaves like Exim; Elm and 
Mulberry behave like Pine (they do their own Bcc: processing).

1. The substantive point

With one possible exception, everybody's opinion was that an MTA has no
business messing with Bcc: header lines. Here is a quote from Tony
Hansen:

   "The MUA, however it is constituted, should handle all BCC processing,
   and the cloning of messages with and without the Bcc: header."

This confirms my belief that Exim's default behaviour is the correct 
one. I hope this particular issue is now laid to rest once and for all.
I have saved all the responses just in case it arises again in the 
future.

The possible exception was a post containing this paragraph:

   The /usr/[s]bin/sendmail interface is a proprietary interface
   introduced by Sendmail. So all other applications providing this
   interface should be as close to Sendmail's behaviour as possible, i.e.
   they should remove Bcc headers.

This is the "my country, right or wrong" argument. There was one
response that added "... when -t is specified", and that is also my
feeling. A call to /usr/sbin/sendmail without -t is not guaranteed to be
a submission. Indeed, the post continued:

   Philosophically, the mailers are still doing the MUA's job by
   sanitizing the message and (finally) handling it over to the message
   router (MTA). It just happens that that part of Unix MUAs is usually
   shipped along with the MTA.

and I would argue that without -t, you cannot, and indeed should not,
assume that the sending entity is an MUA that is submitting a new 
message (see further discussion below).

2. Other discussion

Some interesting points of clarification came up in the discussion. The 
functions of mail handling are now seen as split into four distinct 
parts, as follows:

   MUA -> MSA -> MTA -> MDA -> MUA

(user agent, submission agent, transfer agent, delivery agent). There is 
nothing to say that each of these functional distinctions has to be 
implemented by a different program. Thus, it is generally agreed that a
program handling a local incoming message using the "-t" command line
option is acting partly as an MUA (constructing the envelope) and partly
as an MSA (accepting a new message submission). If it goes on to deliver 
the message over SMTP it is performing an MTA function. If it delivers 
into a local mailbox it is performing an MDA function.

There was some opinion expressed that an MSA might be somewhere that 
Bcc: header lines could be processed. As this is not sanctioned by RFC 
2476 (for MUA->MSA delivery over a network), it would be necessary to
define an extension to the SUBMIT protocol so that the MUA could request
the MSA to do that processing. In the case of a submission via the
command line, the -t option is already available for this purpose.

When -t is not used, a general-purpose MTA such as Exim cannot guess
that it is being called as an MSA. It has to be configured as such, for
example by testing conditions such as "submitted from a host on the
local LAN" or "submitted via port 587". If an extension to SUBMIT were 
defined, another test could be "requested by the SUBMIT protocol". (It
is quite easy to implement such tests in Exim and (for example) remove
Bcc: lines based on the result, but this has to be an explicit
configuration setting; it cannot be defaulted.)

Note that "submitted via the command line" (in the absence of -t) is not
good enough as a test for an MSA. Such messages may have come from
another email regime (e.g. UUCP - some people do still use it) or from a
virus scanner or a mailing list exploder.

3. Uses of Bcc

As part of the discussion, a list of uses of bcc was enumerated. There 
are at least four major ones:

(a) As an instruction from the end user to the MUA to send a copy of the
     message to one or more addresses, without telling the To: and Cc:
     recipients.

(b) When present in a copies sent to bcc recipients, to inform them why
     they received the message (and, if more than one bcc recipient is
     listed, to inform them about additional bcc addressees). This should
     alert bcc recipients to take care when replying.

(c) To prevent unwanted group replies where the sender wants to send a
     copy of a message to someone (whether or not the other recipients
     should know about this), but not subject that person to lots of
     replies from the other recipients who blindly "reply-to-all".

(d) When saved in the sender's sent-mail folder, a Bcc: header acts as a
     record of who the message was sent to.

Is that overloaded or is that overloaded? :-)

Regards,
Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Wed, 18 Aug 2004 10:11:02 +0100 (BST)
From: Philip Hazel <[email-masked]>
To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>
cc: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---559023410-758783491-1092820262=:10417
Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE

On Mon, 17 Aug 2004, Claus F=E4rber wrote:

> The /usr/[s]bin/sendmail interface is a proprietary interface introduced
> by Sendmail. So all other applications providing this interface should
> be as close to Sendmail's behaviour as possible, i.e. they should remove
> Bcc headers.

In the case of -t, yes. I disagree in other cases because in those=20
situations it is not known to be a submission. The message may have come=20
in via UUCP, or be coming back from a virus scanner or mailing list=20
exploder.

Said G.K. Chesterton in "The Defendant": "'My country, right or wrong,'=20
is a thing that no patriot would think of saying. It is like saying, 'My=20
mother, drunk or sober'".

I think the same sentiment applies to Sendmail as to mothers.

Besides, people tell me that different versions/configurations of=20
Sendmail behave differently, though I have no direct evidence to support=20
this, so it may or may not be true.

Regards,
Philip

--=20
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.
---559023410-758783491-1092820262=:10417--

Date: Tue, 17 Aug 2004 21:20:34 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: John C Klensin <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

John,

JCK> Does that agree with your analysis, at least to a first order
JCK> approximation?

  well (harumph!) i can't let the question stand without answering it...

  I trust the assessments you folks have been making about the document,
  and I have not seen counter-arguments that looked at all persuasive.

  I also have not read the spec.
  
d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Wed, 18 Aug 2004 00:13:40 -0400
From: John C Klensin <[email-masked]>
To: Dave Crocker <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Sender: [email-masked]

--On Tuesday, 17 August, 2004 17:38 -0700 Dave Crocker
<[email-masked]> wrote:

> John,
> 
>>> Global parameters are useless if the parser is intelligent
>>> enough to figure out the message structure independently.
>>> Given that such intelligence is a prerequisite to having a
>>> half-baked parser, the global parameters are always
>>> unnecessary.
> 
> JCK> This is a minor point compared to the one below, and
> probably JCK> not an issue here, but I can't let the above
> stand.
> 
> 
> Actually, I suspect you (and we) can and should.
> 
> A draft has been presented.  Numerous objections from
> experienced folks have been lodged.  Is there more productive
> discussion likely?
> 
> This thread has not looked all that productive from the start
> and I am hard-pressed to see how that is going to change.
> Certainly countering arguments about the benefits of standards
> for labeling, semantics, and the like is not likely to
> convince anyone.  If they don't see it, the only real question
> is what they are doing in this venue.
> 
> Now, the real reason for my posting:  If someone can summarize
> where this thread is going and/or should go, I'd appreciate it.

Well, I had planned to just stop after that note, on the theory
that the IESG should now have enough information to make
whatever decision they are inclined to make.   My personal
recommendation, I trust obviously, is that they should say "no"
to this particular draft as a proposed standard: for
standardization purposes, it just isn't specific enough about
what is being transmitted.

If I were giving advice to the author, it would pretty closely
follow what I understand Keith to be suggesting.   Withdraw this
document and the request that it be considered for PS.  Generate
a new document that is very specific about how much or how
little can be assumed when something arrives with this content
type, makes it explicit that the receiver had better be prepared
to handle any of the various mbox forms, and points to whatever
documentation or other materials that can be easily identified
and might help in constructing a receiver-processor.   I think
that document should be Informational, with a media type in some
non-IETF tree, but could personally live with a Proposed
Standard if the interoperability constraints, and any security
risks from guessing wrong about properties of the format were
extremely explicit.

Does that agree with your analysis, at least to a first order
approximation?

    john

Date: 17 Aug 2004 20:57:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: The Bcc Issue
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Nathaniel Borenstein)  wrote on 16.08.04 in <[email-masked]>:

> Everyone seems to have simple rules, and yet I will freely confess that
> I do not have any coherent model at this point for how bcc is supposed
> to work.  The more I hear the more confused I get.

On the other hand, I see an overwhelming majority arguing for the exact  
same model - occasionally offering that some particular extension might  
make sense, but not actually arguing it should be done.

> I think that someone who thinks they have a coherent mental model for
> all of this should try to write it down.  -- Nathaniel

It's *only* the MUA's job to make sure everyone gets the right Bcc; header  
or absence of same. One particular way to implement it is via
/usr/{sbin,lib}/sendmail -t, or any other kind of contract with a  
different piece of software that is then considered part of the MUA  
implementation. The safest way is not handing that responsibility off, but  
sending separate copies whenever differences in Bcc: are wanted.

Nobody else ought to touch it.

MfG Kai

Date: 17 Aug 2004 21:05:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: Diabling Replies
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Dave Crocker)  wrote on 16.08.04 in <[email-masked]>:

TH>> He's suggesting a form of To/Cc that other people cannot accidentally
TH>> reply to.

> That's what I meant to describe.  The headers sent by the early MH for a
> bcc were:
>
> Date: xxx
> [To]: yyy
> [cc]: zzz
> Subject: aaa
> From: bbb
>
> So a regular reply command would not see the primary or secondary
> address lists.

Ah, no. The suggestion was NOT for the Bcc: recipients to be unable to  
reply, it was for the *non*-Bcc: recipients being unable to reply *to* the  
Bcc: recipients (by simply not having those addresses in the headers they  
see).

You could call that a one-off or snapshot copy.

> I had missed that.  So he wants these "tertiary" recipients to be listed
> explicitly, but to get copies that cannot be automatically replied to.

No, no, no, no. See above. You still got it the wrong way around.

> In other words, they are outside the fishbowl of the conversation.  They
> get to listen but not participate.

The whole point is they do *not* get to listen.

MfG Kai

Date: 17 Aug 2004 20:56:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: The Bcc Issue
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Claus Färber)  wrote on 17.08.04 in <[email-masked]>:

> Philip Hazel <[email-masked]> schrieb/wrote:
> > 1. Exim (which I maintain) removes Bcc: lines if, and only if, it is
> >     called with the -t option. In that case, it is constructing an
> >     envelope from the header data, and IMHO in doing so it is fulfilling
> >     an MUA function. In all other cases it leaves Bcc: lines alone.
>
> The /usr/[s]bin/sendmail interface is a proprietary interface introduced
> by Sendmail. So all other applications providing this interface should
> be as close to Sendmail's behaviour as possible, i.e. they should remove
> Bcc headers.

... when using the -t option.

> Philosophically, the mailers are still doing the MUA's job by sanitizing
> the message and (finally) handling it over to the message router (MTA).
> It just happens that that part of Unix MUAs is usually shipped along
> with the MTA.

And in fact, it would be entirely possible to implement a
/usr/sbin/sendmail that implemented *only* that part of the functionality,  
handing messages off to a separate MTA via SMTP or SUBMISSION.

MfG Kai

Cc: Keith Moore <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Tue, 17 Aug 2004 22:57:46 -0400
To: Dave Crocker <[email-masked]>
Sender: [email-masked]

On Aug 17, 2004, at 8:38 PM, Dave Crocker wrote:

>  If someone can summarize where
> this thread is going and/or should go, I'd appreciate it.

my recommendation to the author -

- do another edit on the draft, mostly for nits and clarity, and 
resubmit for Informational

- don't try to specify the application/mbox format in this document at 
all.  instead, point to various published definitions and descriptions 
(man pages, web pages, whatever) such as exist.

- for that matter, don't try to specify how best to generate mbox 
files.  that's out of scope.

- explicitly point out that there are varieties in how the format has 
been implemented which can affect interoperability.  (this should not 
be a showstopper, as there are varieties in how almost any other 
MIME-labeled format is implemented which can affect interoperability.)

- my preference is to not have parameters describing format variants at 
all, as they're highly unlikely to be used correctly and even less 
likely to be generated correctly.  if you must have parameters, make 
them optional.

- you might want to point out that a wide variety of mbox-reading tools 
generally manage to make sense of mbox files despite the variations.  
and yes, they barf sometimes.  some implementations' heuristics are 
better than others.  again, this is also true of Word, PDF, JPEG, 
PostScript, or any other format you could care to look at.  it's useful 
to be able to distinguish mbox files from other kinds of files, even if 
saying something is an mbox file is not perfectly precise.

- point out that the existence of an application/mbox label is not a 
recommendation that mbox files be used to transport groups of messages 
-  at least, not without first defining a new variant of mbox files 
that is transparent (no >From), reliable (no message boundary 
synchronization errors),  immune to variations in line terminator, and 
doesn't use character or line counts (no content-length).

- recommend base64 encoding as a default for transport of mbox files 
over email, since the encoder typically has no idea about the message 
contents and whether they might contain binary data.

my recommendation to everyone else -

let's wait for the revision and look at it then.  meanwhile, let's 
realize

- that having a way to label mbox fiiles is much more useful than not 
having one;

- that perfectly precise labels for anything are almost nonexistent in 
practice;

- that having an imprecise MIME label for mbox files probably won't 
degrade interop significantly over what it already is

- that there are probably better uses of our time (individually or 
collectively) than to define the perfect label for mbox files.

Keith

Date: Tue, 17 Aug 2004 17:38:52 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: John C Klensin <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

John,

>> Global parameters are useless if the parser is intelligent
>> enough to figure out the message structure independently.
>> Given that such intelligence is a prerequisite to having a
>> half-baked parser, the global parameters are always
>> unnecessary.

JCK> This is a minor point compared to the one below, and probably
JCK> not an issue here, but I can't let the above stand.

Actually, I suspect you (and we) can and should.

A draft has been presented.  Numerous objections from experienced folks
have been lodged.  Is there more productive discussion likely?

This thread has not looked all that productive from the start and I am
hard-pressed to see how that is going to change.  Certainly countering
arguments about the benefits of standards for labeling, semantics, and
the like is not likely to convince anyone.  If they don't see it, the
only real question is what they are doing in this venue.

Now, the real reason for my posting:  If someone can summarize where
this thread is going and/or should go, I'd appreciate it.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Tue, 17 Aug 2004 16:53:22 -0400
From: "Eric A. Hall" <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: Tony Hansen <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Sender: [email-masked]

On 8/17/2004 4:06 PM, John C Klensin wrote:

> In that context, unless I completely misunderstand what is going
> on here, the "...prerequisite to having a half-baked parser..."
> assertion borders on the silly.   Take the example to which Tony
> has been pointing.  Apparently the Solaris version of an mbox
> format is well-documented and based on content length
> information rather than key strings.

... it is well-defined for writing, but AFAIK the number of reliable
clients that depend on the presence of such data are zero.

This is the point really: Given that the database structure really is
undefined, and that any number of clients may have accessed the database
via ~NFS or via different mailers on the same local client (Pine and KMail
and ...), a half-baked mailer that wants to reliably read mbox data of any
kind will know better than to rely on Content-Length in isolation, and
will have to validate assumptions and check the clues before doing
anything substantive. This includes checking for EOL syntaxes, but also
includes things like default charsets for untagged data, default domains,
and so forth.

> Given that model, the key to an mbox format isn't the content of
> the blobs, it is the system used to decompose an mbox into a
> blob collection.

That's a function of the content parser and whatever task it is trying to
perform. The purpose of importing an mbox database into an IMAP store (as
is common with things like downloadable archives) is going to have several
different considerations and requirements than simple searching and
printing (as with local archives generated by ~Mozilla locally), but both
of them can be satisfied through basic sniff tests and local defaults.

Most of the spec work is appropriate and useful and arguably necessary for
*writing* to mbox files, especially if you are appending an existing file,
but even then it takes few smarts to figure out that you should write to
the existing content instead of druidic rules. Having said that, I'd love
to see a canonical authoritative format which could be referenced for such
purposes (not for transfers, but for parsers to use when they go about
generating content).

> Instead, you are making a case that this registration should be
> a family of, e.g.,
>   application/mbox-solaris-v5-and-later
>   application/mbox-sendmail-v2-v4

I hope not; unrecognized types match to application/octet-stream so many
types that nobody implements won't get us very far either.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 17 Aug 2004 16:06:03 -0400
From: John C Klensin <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Tony Hansen <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Sender: [email-masked]

--On Tuesday, 17 August, 2004 15:09 -0400 "Eric A. Hall"
<[email-masked]> wrote:

>> To be clear about this, I think there are three choices which
>> we might prefer in descending order:
>> 
>> 	(1) There is a single canonical "wire" format in which
>> 	these things are transmitted.
> 
> Such a specification would surely dictate "a series of
> message/rfc822 objects". But if we were to require that
> end-points perform conversion into a neutral form, we might as
> well go the whole nickel and just say "use multipart/digest",
> because that's where we'd end up after monhts of beating on
> each other.
>...
>> 	(2) The content-type specifies a conceptual form
>> 	("application/mbox") but has _required_ parameters that
>> 	specify the specific form being transmitted.
> 
> Global parameters are useless if the parser is intelligent
> enough to figure out the message structure independently.
> Given that such intelligence is a prerequisite to having a
> half-baked parser, the global parameters are always
> unnecessary.

This is a minor point compared to the one below, and probably
not an issue here, but I can't let the above stand.  My
impression of the MIME design, from the beginning, was that I
should be able to inspect the content-type --including both the
primary type and any parameters-- before deciding whether to
retrieve and open the content of the body part.  Yes, we have
implementations and protocols that don't take advantage of that,
or that can take advantage of the content-type only and not the
parameters, but the MIME design, and hence the media type
design, is that I should be able to tell whether I can parse
(see below) the body part without opening it and applying
heuristics to it.

In addition, what you seem to mean by "intelligent enough to
figure out the message structure independently" is what I would
mean if I said "know most or all of the formats and apply
heuristics to figure out which one to apply".  We just don't do
that, at least knowingly and with standards-track media types.
You might rationally argue that this should be an exception if
you could demonstrate that a reasonable set of heuristics would
_always_ make the choice correctly, but that would require you
to document either the heuristics or all of the variations on
mbox and how to tell them apart (or both) -- something that
several people have claimed is basically impossible and that you
clearly haven't expressed a desire to do.

In that context, unless I completely misunderstand what is going
on here, the "...prerequisite to having a half-baked parser..."
assertion borders on the silly.   Take the example to which Tony
has been pointing.  Apparently the Solaris version of an mbox
format is well-documented and based on content length
information rather than key strings.   That implies that, if (i)
I know that what is coming is in that Solaris format and (ii) I
have a rather primitive parser that knows how to find and deal
with content lengths, then I can parse the format without any
ability to "figure out the structure" at all.   However, if I
attempt to apply that parser to something to uses key strings,
rather than lengths, or something that violates the assumptions
Solaris makes about what gets included in the length
computations, the parser is going to be bewildered at best and
yield silly results at worst -- and that is exactly what content
types and their parameters are intended to let receiving
applications guard against.

> Actually, global parameters are more than useless. What if we
> have a mixed mbox file, where some messages are untagged BIG5
> and others are untagged 8859-1, or we have some messages have
> VMS::Mail addresses and others have MS/Mail addresses, or so
> forth? The global nature of global parameters ignores the
> per-message reality of the mbox structure.
> 
> Global parameters can also be harmful if they conflict with
> reality.

That brings us to the main problem, or misunderstanding, or
strawman, depending on one's perspective.

I may be being excessively dense here, but, if I am, I seem to
have significant company.   I am not asking (or suggesting) that
you provide the information that would be required for
multipart/, e.g., the content-type and associated parameters
(such as charset, for your examples above) for each message
(much less, for multipart/digest, that you force everything into
the required subset of an RFC822 message body).  While I have
concerns that we didn't get multipart quite right and that it is
too late to fix it, those concerns don't interact with my
concern about application/mbox at all.

So let's move back a half-step.  To me, the essence of the mbox
format, conceptually, is that it consists of a sequence of blobs
that are normally interpreted as messages in some format or
other. You've made several convincing arguments that we should
see them as blobs, not as messages, and I (and I think others)
accepted them long ago.   I think that such a blob collection is
a reasonable thing to want to mail or otherwise use as a media
type.  And, again, I accept your argument that the blobs may not
be valid 822/2822 messages or encapsulated 2821 messages, and,
indeed, that there may be considerable format and content
heterogeneity from one blob to the next. 

Given that model, the key to an mbox format isn't the content of
the blobs, it is the system used to decompose an mbox into a
blob collection.  If this were a multipart structure, the
corresponding issue of interest would be whether the Boundary
parameter provided enough information to separate the parts.  It
would not be what was in each of the parts or even what _their_
content types were.  It would simply be the ability to separate
the aggregate mbox into separate blobs.

If I know that blobs are length-delimited according to some
specific set of rules, I have that information and can build a
trivial parser (and understand what it can't handle).   If I
know they are separated by indicator strings that obey some
specific set of rules, then I can build a trivial parser (and
understand what it can't handle).  And so on for different sets
of rules.   But, as far as I can tell, you don't want to give us
that information.  Instead, you want us to accept "well, it is
some sort of mbox format, and you either need to guess at how to
separate the blobs by examining the content or you need to get
that information out of band".   If that is what you intend this
specification to imply, IMnvHO, unacceptable for a
standards-track media type.

Finally, if you argue that global parameters are unacceptable as
a means of making the type of format/ de-blob-ing distinctions
outlined below, I suggest that you are not making a good case
for leaving the parameters (and associated information) out.
Instead, you are making a case that this registration should be
a family of, e.g.,
  application/mbox-solaris-v5-and-later
  application/mbox-sendmail-v2-v4
and similar things (examples made up).

       john 

Date: Tue, 17 Aug 2004 15:09:07 -0400
From: "Eric A. Hall" <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: [email-masked], Tony Hansen <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

On 8/17/2004 2:09 PM, John C Klensin wrote:

> To be clear about this, I think there are three choices which we
> might prefer in descending order:
> 
> 	(1) There is a single canonical "wire" format in which
> 	these things are transmitted.

Such a specification would surely dictate "a series of message/rfc822
objects". But if we were to require that end-points perform conversion
into a neutral form, we might as well go the whole nickel and just say
"use multipart/digest", because that's where we'd end up after monhts of
beating on each other.

We'd still be in the same place we're at today, of course, because HTTP
and filesystem transfers aren't going to do mbox->digest conversion any
more than they are going to do EOL conversion or Content-Length calcs.
We'd still have opaque messaging databases that people wanted to transfer,
search, open and otherwise automate, but couldn't.

> 	(2) The content-type specifies a conceptual form
> 	("application/mbox") but has _required_ parameters that
> 	specify the specific form being transmitted.

Global parameters are useless if the parser is intelligent enough to
figure out the message structure independently. Given that such
intelligence is a prerequisite to having a half-baked parser, the global
parameters are always unnecessary.

Actually, global parameters are more than useless. What if we have a mixed
mbox file, where some messages are untagged BIG5 and others are untagged
8859-1, or we have some messages have VMS::Mail addresses and others have
MS/Mail addresses, or so forth? The global nature of global parameters
ignores the per-message reality of the mbox structure.

Global parameters can also be harmful if they conflict with reality.

> 	(3) These might as well be sent with
> 	application/octet-stream, with optional file name, etc.,
> 	information.

That's where we are now and we already now that's not working.

Date: Tue, 17 Aug 2004 14:09:34 -0400
From: John C Klensin <[email-masked]>
To: [email-masked], Tony Hansen <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Sender: [email-masked]

--On Tuesday, 17 August, 2004 13:05 -0400
[email-masked] wrote:

>...
> So - where is the *one true canonical* definition of an mbox
> that actually answers all these basic questions that an
> implementer *needs* to know the answer to?

Or, as an alternative, where is the set of required parameters
and values that would give the recipient at least a strong clue
as to which of many definitions and variations is in use?

To be clear about this, I think there are three choices which we
might prefer in descending order:

	(1) There is a single canonical "wire" format in which
	these things are transmitted.   It is the responsibility
	of the sender to get from whatever is used locally into
	that form and the responsibility of the recipient to get
	from that form into whatever is needed locally.  And the
	wire format is precisely defined, without handwaving or
	system-specific variations.
	
	(2) The content-type specifies a conceptual form
	("application/mbox") but has _required_ parameters that
	specify the specific form being transmitted.  The
	recipient does not require either out of band
	information or heuristics on the body part content
	itself to figure out what was received and whether local
	facilities exist to decode it.
	
	(3) These might as well be sent with
	application/octet-stream, with optional file name, etc.,
	information.

I just don't see any other options that are consistent with the
media type model.

     john

To: Tony Hansen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard 
From: [email-masked]
Date: Tue, 17 Aug 2004 13:05:27 -0400
Sender: [email-masked]

--==_Exmh_-1395191580P
Content-Type: text/plain; charset=us-ascii

On Mon, 16 Aug 2004 22:47:52 EDT, Tony Hansen said:

> The claim in Appendix A is that there were no authoritative sources of 
> documentation for the mbox formats and otherwise it's "only documented 
> in anecdotal form". I'm sorry, but the the definitions ARE there, and 
> ARE almost always authoritative for those systems.

Somehow, I can't get thrilled by the concept of saying a format is documented
because we have (for example) 3 systems, and each has an authoritative
definition of the version it uses, and the definitions are incompatible (and
yes, the Solaris 'content-length:' scheme and '>from ' escaping are basically
incompatible - there exist messages that can't be converted from one to the
other without information loss).

> Because Solaris 8 is System Vr4-derived, you should look at 'man mail' 
> for the definitive definition. You'll find Content-Length: documented there.

It says:

     A letter is composed of some  header  lines  followed  by  a
     blank line followed by the message content. The header lines
     section of the letter consists of one  or  more  UNIX  post-
     marks:

           From     sender     date_and_time     [remote     from
          remote_system_name]

     followed by one or more standardized message header lines of
     the form:

           keyword-name: [printable text]

     where keyword-name  is  comprised  of  any  printable,  non-
     whitespace  characters  other  than  colon (`:'). A Content-
     Length: header line, indicating the number of bytes  in  the
     message  content  will  always  be present unless the letter
     consists of only header lines with  no  message  content.

For bonus points - is the 'crlf-crlf' between the header and the body included
in the Content-Length:?  There's other issues as well - what if the
Content-Length: is computed across a non-canonified message - how do
you send it across the wire?

'man mail' doesn't mention escaping a 'From ' inside a message,
except for this:

     The default mode for printing messages is  to  display  only
     those header lines of immediate interest. These include, but
     are not limited to,  the  UNIX  From  and  >From  postmarks,
     From:,  Date:,  Subject:,  and Content-Length: header lines,
     and any recipient header lines such as To:, Cc:,  Bcc:,  and
     so  forth.  After the header lines have been displayed, mail

Of course, that's because Solaris doesn't use '>From ' escaping
because it has Content-Length instead.

Should other systems trust the value of a Content-Length:?

Should other systems be required to include a Content-Length?

Should other systems escape a 'From ' iff there's no Content-Length?

What if an mbox file has a Content-Length on some items but not others?

How do you recover from a corrupted Content-Length?

So - where is the *one true canonical* definition of an mbox that actually
answers all these basic questions that an implementer *needs* to know the
answer to?

--==_Exmh_-1395191580P
Content-Type: application/pgp-signature

--==_Exmh_-1395191580P--

To: [email-masked]
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
Subject: Re: The Bcc Issue
Date: 17 Aug 2004 00:48:00 +0200
Lines: 20
Sender: [email-masked]

Philip Hazel <[email-masked]> schrieb/wrote:
> 1. Exim (which I maintain) removes Bcc: lines if, and only if, it is
>     called with the -t option. In that case, it is constructing an
>     envelope from the header data, and IMHO in doing so it is fulfilling
>     an MUA function. In all other cases it leaves Bcc: lines alone.

The /usr/[s]bin/sendmail interface is a proprietary interface introduced
by Sendmail. So all other applications providing this interface should
be as close to Sendmail's behaviour as possible, i.e. they should remove
Bcc headers.

Philosophically, the mailers are still doing the MUA's job by sanitizing
the message and (finally) handling it over to the message router (MTA).
It just happens that that part of Unix MUAs is usually shipped along
with the MTA.

Claus
-- 
http://www.faerber.muc.de

Date: Mon, 16 Aug 2004 22:47:52 -0400
From: Tony Hansen <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

[email-masked] wrote:

> Umm.. Tony?  I hate to say it, but if there have been several variants used in
> the wild, and the man pages for said variants document different formats,
> that's awfully close to "anecdotally defined" when you're doing a standard.

The claim in Appendix A is that there were no authoritative sources of 
documentation for the mbox formats and otherwise it's "only documented 
in anecdotal form". I'm sorry, but the the definitions ARE there, and 
ARE almost always authoritative for those systems.

This doesn't preclude there being other formats in the wild, nor does it 
preclude there being other tools on those same machines that conflict.

> For example, a Solaris 8 box across the hall says in 'man mail.local':
> ...
> 
> Any doubts that this whole mess is at best anecdotally defined can be dispelled by
> mentioning "Content-Length:" (interestingly enough, not even mentioned in the
> Solaris or Sendmail man pages, although the Sendmail source tree does mention
> that building on Solaris 2.3 or later will turn it on.  Of interest mostly because
> the Content-Length: is so easily broken by later >-stuffing/unstuffing or other
> similar conversion...

Because Solaris 8 is System Vr4-derived, you should look at 'man mail' 
for the definitive definition. You'll find Content-Length: documented there.

	Tony Hansen
	[email-masked]

Date: Tue, 17 Aug 2004 10:53:56 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 16 2004, Bruce Lilly wrote:

> Specifically, From stuffing and any special treatment of control
> characters can cause problems with delimiters.  In the specific
> case of a Content-Length delimiter, *any* change to the content
> which changes its length (e.g. adding or removing trailing
> whitespace on lines) will cause problems.

Agreed.

> Note that "From" is valid base64 output and that transport can
> result in addition of trailing whitespace.

Ah, you're right! A single 3 byte attachment could fake a minimal
From_ delimiter (to get an encoding of the form empty line + "From" +
space padding + empty line). Any bigger attachment wouldn't, if the
Base64 is properly formatted. So any nonempty (ie containing 
at least one message header) mbox attachment would still be safe. 

> > This could make proper parsing of, e.g. a QP encoding of
> > an mbox file attached to a message saved in an mbox archive,
> > needlessly difficult for common tools such as e.g. formail, especially
> > if the tool happens to be not MIME aware.
> 
> I'm not sure I understand your point; processing of an attached
> media type has several prerequisites:
> 1. a user indicates that processing may proceed (N.B. no
>    automatic execution!)
> 2. a MIME-aware application decodes any transfer encoding &
>    extracts the attachment
> 3. some application capable of processing the content -- it
>    need not be MIME-aware
> The MIME-aware application (typically a MUA) may pass
> additional information (from parameters) to the media-
> specific application
> 

I'm thinking of the case of an application which iterates over an mbox
file extracting messages for processing (perhaps sending to a MIME
aware utility for display or otherwise).

If the extractor itself isn't MIME aware, all spurious From_ lines can
cause difficulty (e.g. do you also check if the following lines are
standard headers before accepting this as a message boundary?).  This
difficulty happens before any MIME-aware tools are handed whatever
fragment is deemed to be a full message.

Of course, it can be argued that either the spurious From_ lines would
always be escaped by whoever created the mbox file, or that MIME is
sufficiently old that any decent extracting tool should be aware of
attachment formats and boundary lines.

Perhaps formail is a bad example, but from the man page on my system,
it states:

     The regular expression that is used to find `real' postmarks is:
              "\n\nFrom [\t ]*[^\t\n ]+[\t ]+[^\n\t ]"

Also, it's worth pointing out that a QP encoding of "From " can be
either "From " (ie identical output) or "From=20", where the latter
would be much preferable, but the choice is up to the encoding
software, which probably doesn't know it's encoding a mail message.

-- 
Laird Breyer.

Date: Mon, 16 Aug 2004 18:57:13 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: The job of an MSA
Sender: [email-masked]

> > Given my druthers, I'd much rather see it be an extension only to the
> > MSA SUBMIT protocol and not one that is expected to be implemented in MTAs.
> 
> For the issue of efficient transfer where there is a bottleneck
> (e.g. a dialup connection to an ISP), it might not be practical
> to restrict it to the SUBMIT protocol; not all ISPs operate
> servers on port 587.

if they're going to upgrade their servers to support the extension,
they may as well upgrade them to listen on port 587 at the same time.

this really needs to be an extension for SUBMIT only

Keith

Date: Mon, 16 Aug 2004 14:50:11 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Bruce Lilly <[email-masked]>
CC: [email-masked]
Subject: Re: Disabling Replies
Sender: [email-masked]

Bruce,

BL> Dave Crocker wrote:

>> That's what I meant to describe.  The headers sent by the early MH for a
>> bcc were:
>> 
>> Date: xxx
>> [To]: yyy
>> [cc]: zzz
>> Subject: aaa
>> From: bbb

BL> Gack! I'm assuming "early" means before RFC 2822.  Was it before RFC 822?

Yes. It was right around the time of RFC733, plus or minus a few months.

BL> RFC 822 required at least one destination address header field (RFC 2822
BL> has relaxed that, and it was not required in RFCs 733 and 724), and there
BL> are none in that example!

yeah. it probably violated that (later?) requirement, but it would be
easy to hack in a pro-forma To.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Mon, 16 Aug 2004 17:41:35 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Dave Crocker <[email-masked]>
CC: [email-masked]
Subject: Disabling Replies
Sender: [email-masked]

Dave Crocker wrote:

> That's what I meant to describe.  The headers sent by the early MH for a
> bcc were:
> 
> Date: xxx
> [To]: yyy
> [cc]: zzz
> Subject: aaa
> From: bbb

Gack! I'm assuming "early" means before RFC 2822.  Was it before RFC 822?
RFC 822 required at least one destination address header field (RFC 2822
has relaxed that, and it was not required in RFCs 733 and 724), and there
are none in that example!

Date: Mon, 16 Aug 2004 17:33:16 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Tony Hansen <[email-masked]>
CC: [email-masked]
Subject: Re: The job of an MSA
Sender: [email-masked]

Tony Hansen wrote:

> Given my druthers, I'd much rather see it be an extension only to the
> MSA SUBMIT protocol and not one that is expected to be implemented in MTAs.

For the issue of efficient transfer where there is a bottleneck
(e.g. a dialup connection to an ISP), it might not be practical
to restrict it to the SUBMIT protocol; not all ISPs operate
servers on port 587.

Date: Mon, 16 Aug 2004 17:30:04 -0400
From: Tony Hansen <[email-masked]>
CC: [email-masked]
Subject: Re: Diabling Replies
Sender: [email-masked]

Dave Crocker wrote:

> TH> These people will receive a copy of your message, but will not 
> TH> see replies. He doesn't mind that the To/Cc people know that these other 
> TH> people have received a copy, so Bcc isn't really quite what he wants.
> 
> I had missed that.  So he wants these "tertiary" recipients to be listed
> explicitly, but to get copies that cannot be automatically replied to.
> In other words, they are outside the fishbowl of the conversation.  They
> get to listen but not participate.

My understanding is that these tertiary recipients would only see the 
initial message of the conversation. It's a visible form of Bcc.

	Tony Hansen
	[email-masked]

Date: Mon, 16 Aug 2004 14:25:47 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Tony Hansen <[email-masked]>
CC: [email-masked]
Subject: Re: Diabling Replies
Sender: [email-masked]

Tony,

TH> He's suggesting a form of To/Cc that other people cannot accidentally
TH> reply to.

That's what I meant to describe.  The headers sent by the early MH for a
bcc were:

Date: xxx
[To]: yyy
[cc]: zzz
Subject: aaa
From: bbb

So a regular reply command would not see the primary or secondary
address lists.

TH> These people will receive a copy of your message, but will not 
TH> see replies. He doesn't mind that the To/Cc people know that these other 
TH> people have received a copy, so Bcc isn't really quite what he wants.

I had missed that.  So he wants these "tertiary" recipients to be listed
explicitly, but to get copies that cannot be automatically replied to.
In other words, they are outside the fishbowl of the conversation.  They
get to listen but not participate.

Certainly that is a reasonable mode of 'participation'.  At the least, it would take
a new address field for the special class of recipients.

We probably don't need to disable the to/cc fields, for them, because
the semantics of the new, third field would be "do not allow a Reply
command to to/cc".  Of course, any MUA could choose to violate that
standard...

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Mon, 16 Aug 2004 14:25:10 -0700
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Tony Hansen <[email-masked]>
CC: [email-masked]
Subject: Re: Diabling Replies
Sender: [email-masked]

Tony,

TH> He's suggesting a form of To/Cc that other people cannot accidentally
TH> reply to.

That's what I meant to describe.  The headers sent by the early MH for a
bcc were:

Date: xxx
[To]: yyy
[cc]: zzz
Subject: aaa
From: bbb

So a regular reply command would not see the primary or secondary
address lists.

TH> These people will receive a copy of your message, but will not 
TH> see replies. He doesn't mind that the To/Cc people know that these other 
TH> people have received a copy, so Bcc isn't really quite what he wants.

I had missed that.  So he wants these "tertiary" recipients to be listed
explicitly, but to get copies that cannot be automatically replied to.
In other words, they are outside the fishbowl of the conversation.  They
get to listen but not participate.

Certainly that is a reasonable mode of 'participation'.  At the least, it would take
a new address field for the special class of recipients.

We probably don't need to disable the to/cc fields, for them, because
the semantics of the new, third field would be "do not allow a Reply
command to to/cc".  Of course, any MUA could choose to violate that
standard...

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Mon, 16 Aug 2004 17:05:25 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: The job of an MSA
Sender: [email-masked]

Bruce Lilly wrote:

> Sending separate Bcc and non-bcc copies would be preferable
> functionally, but of course requires additional bandwidth. The
> bandwidth issue could theoretically be handled by offloading the
> "splitting" functionality on an MTA closer to the Internet backbone,
> e.g. using an ESMTP extension as Jacob has suggested.

Given my druthers, I'd much rather see it be an extension only to the 
MSA SUBMIT protocol and not one that is expected to be implemented in MTAs.

	Tony Hansen
	[email-masked]

Date: Mon, 16 Aug 2004 16:57:23 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: Dave Crocker <[email-masked]>
Subject: Re: The job of an MSA
Sender: [email-masked]

Bruce Lilly wrote:

> This message serves a couple of purposes. When composed (within Mozilla
> 1.7), it included a Bcc field. Mozilla is configured to send to sendmail
> listening on the submission port [the sendmail process happens to be
> running on the same host as Mozilla in this instance]. Mozilla is also
> configured to save a copy of sent messages in Mozilla format in a local
> "folder".  The Bcc copy should be sent by sendmail to a Cyrus IMAP
> server (also running on the same host, as it happens) via LMTP. The
> list should receive a copy w/o the Bcc field.  Mozilla has little in
> the way of configuration options for sending -- I'm not sure if it
> checks Bccs and sends multiple messages via port 587, or if it relies
> on the submission server to do that processing; but I intend to run
> a traffic sniffer to find out, and I'll report anything significant
> in a subsequent message.

Mozilla sent the message (once, with no Bcc header field) to the
MSA, with envelope recipients (RCPT TO) set to all of the recipients
specified, including bcc.  The local Mozilla "Sent" "folder" copy
includes the Bcc field. The copy sent to the bcc address (and to the
list -- it was sent only once to the MSA -- does not include any Bcc
header field).

Regarding functionality discussed:
1. I-sent-a-bcc-to only appears in the local Mozilla "folder". It
   therefore cannot be accessed (w/o some contortions and hoop-jumping)
   anywhere else, or by any other MUA.
2. Bcc-should-be-sent-to is apparently how Mozilla uses the field
3. Why-you-got-this doesn't apply to the bcc recipient -- he has no
   indication that the message was sent as a bcc.
4. Non-reply-to seems to be another Mozilla trait; if I go to the
   local copy and try "Reply All", there's no copy sent to the bcc
   address.

#1 and #3 are apparently disadvantages of the way this particular
combination of software operates.  Sending separate Bcc and non-bcc
copies would be preferable functionally, but of course requires
additional bandwidth.  The bandwidth issue could theoretically be
handled by offloading the "splitting" functionality on an MTA closer
to the Internet backbone, e.g. using an ESMTP extension as Jacob has
suggested.

Date: Mon, 16 Aug 2004 13:55:25 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Tony Hansen <[email-masked]>
CC: [email-masked]
Subject: Re: The job of an MSA
Sender: [email-masked]

Tony,

TH> The current SUBMIT protocol clearly identifies which headers it is 
TH> allowed to mess around with. The To/Cc/Bcc headers are not in that list.

TH> But, I do agree with your sentiment -- SUBMIT was clearly intended to 
TH> and designed to off-load the MUA from certain tasks. So I could see 

 The point raised by Ned(?) -- that an MUA function that is off-loaded to
 a server does not make it an MSA function -- is well-taken.

 I'm trying to assess people's views on the PRECISE dividing line
 between MUA and MSA, rather than trying to espouse a particular view.

 I'm fine with the idea of a "split" MUA (and that's a term from the
 mid-70's), though it does beg the question of the protocol used between
 the local and remote MUA portions.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Mon, 16 Aug 2004 16:54:51 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: Diabling Replies
Sender: [email-masked]

Dave Crocker wrote:

> JJP> Bcc actually has two functions:
> ...
> JP> (2) To stop replies, i.e. working as a kind of "Non-Reply-To".
> 
> While I think it would be entirely reasonable for an rMUA to take the
> presence of BCC as meaning that the rMUA should not allow a reply
> command to automatically include TO or CC, I do not believe that it a
> common rMUA behavior.  In fact, I've never heard of it.
> 
> In the earliest version of MH (pre-Rose), the posting command sent the
> BCC copy separately from the To/CC copy.  One of the things is did was
> to make To and CC be [To] and [CC].  So they looked like normall to and
> cc fields, but the reply commands did not see them present.
> 
> This got removed from later versions of MH and I haven't see anything
> similar since then.
> 
> Accidentally copying to the to and cc folks is certainly an important
> issue.

Dave, I think the behaviour in early-MH that you describe is different 
from Jacob's suggested use of Bcc.

He's suggesting a form of To/Cc that other people cannot accidentally 
reply to. These people will receive a copy of your message, but will not 
see replies. He doesn't mind that the To/Cc people know that these other 
people have received a copy, so Bcc isn't really quite what he wants. 
Right now, the only way to achieve this type of behaviour is to use Bcc 
and then put some text in the message saying "I've also bcc'd this 
message to x, y and z".

If I understand correctly, your description of early-MH's behaviour is 
that the copy of the message that the BCC recipients receive would have 
their To/Cc headers zapped such that the BCC recipient will see who got 
the message, but cannot do a reply.

	Tony Hansen
	[email-masked]

Date: Mon, 16 Aug 2004 16:47:04 -0400
From: Tony Hansen <[email-masked]>
To: Dave Crocker <[email-masked]>
CC: [email-masked]
Subject: Re: The job of an MSA
Sender: [email-masked]

Dave Crocker wrote:

> From the mail-arch writing effort, I am not so clear about the boundary
> between MUA and MSA in the community's mind. I'd like to change that.
> In general, my sense is that the MSA is entitled to perform quite a bit
> of MUA funtionality, in order to off-load the MUA, both computationally
> and in terms of network traffic.
> If this is a correct assessment, then messing with headers is
> acceptable.

There are headers and then there are headers. :-)

The current SUBMIT protocol clearly identifies which headers it is 
allowed to mess around with. The To/Cc/Bcc headers are not in that list.

But, I do agree with your sentiment -- SUBMIT was clearly intended to 
and designed to off-load the MUA from certain tasks. So I could see 
there being an extension to the SUBMIT protocol that allowed it to look 
at the To/Cc/Bcc headers and acted accordingly. But it would have to be 
an extension.

	Tony Hansen
	[email-masked]

Date: Mon, 16 Aug 2004 16:14:50 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Jacob Palme <[email-masked]>
CC: [email-masked]
Subject: Re: Message-ID and Bcc Was: The Bcc Issue
Sender: [email-masked]

Jacob Palme wrote:

> Arguments agains the same Message-ID: Many mailers suppress 
> multiple copies of the same message with the same 
> Message-ID.

That's not necessarily always an argument against keeping the
same msg-id -- some recipients *like* having effective
duplicate suppression.

Date: Mon, 16 Aug 2004 15:04:34 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Dave Crocker <[email-masked]>
CC: [email-masked]
Subject: Re: The job of an MSA
Sender: [email-masked]

Dave Crocker wrote:

> In general, my sense is that the MSA is entitled to perform quite a bit
> of MUA funtionality, in order to off-load the MUA, both computationally
> and in terms of network traffic.
> 
> If this is a correct assessment, then messing with headers is
> acceptable.
> 
> I would very much appreciate getting some dialogue about this on the
> list.

This message serves a couple of purposes. When composed (within Mozilla
1.7), it included a Bcc field. Mozilla is configured to send to sendmail
listening on the submission port [the sendmail process happens to be
running on the same host as Mozilla in this instance]. Mozilla is also
configured to save a copy of sent messages in Mozilla format in a local
"folder".  The Bcc copy should be sent by sendmail to a Cyrus IMAP
server (also running on the same host, as it happens) via LMTP. The
list should receive a copy w/o the Bcc field.  Mozilla has little in
the way of configuration options for sending -- I'm not sure if it
checks Bccs and sends multiple messages via port 587, or if it relies
on the submission server to do that processing; but I intend to run
a traffic sniffer to find out, and I'll report anything significant
in a subsequent message.

Date: Mon, 16 Aug 2004 11:40:46 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Jacob Palme <[email-masked]>
CC: [email-masked]
Subject: Diabling Replies
Sender: [email-masked]

JJP> Bcc actually has two functions:
...
JP> (2) To stop replies, i.e. working as a kind of "Non-Reply-To".

While I think it would be entirely reasonable for an rMUA to take the
presence of BCC as meaning that the rMUA should not allow a reply
command to automatically include TO or CC, I do not believe that it a
common rMUA behavior.  In fact, I've never heard of it.

In the earliest version of MH (pre-Rose), the posting command sent the
BCC copy separately from the To/CC copy.  One of the things is did was
to make To and CC be [To] and [CC].  So they looked like normall to and
cc fields, but the reply commands did not see them present.

This got removed from later versions of MH and I haven't see anything
similar since then.

Accidentally copying to the to and cc folks is certainly an important
issue.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Mon, 16 Aug 2004 11:35:38 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Tony Hansen <[email-masked]>
CC: [email-masked]
Subject: The job of an MSA
Sender: [email-masked]

Tony,

TH> An MSA listening on either port 25 or 587 should NOT be looking at 
TH> To/Cc/Bcc headers. They should only be looking at RCPT TO. See RFC 2476.

>From the mail-arch writing effort, I am not so clear about the boundary
between MUA and MSA in the community's mind.

I'd like to change that.

In general, my sense is that the MSA is entitled to perform quite a bit
of MUA funtionality, in order to off-load the MUA, both computationally
and in terms of network traffic.

If this is a correct assessment, then messing with headers is
acceptable.

I would very much appreciate getting some dialogue about this on the
list.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Mon, 16 Aug 2004 19:06:36 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: The Bcc Issue
Sender: [email-masked]

At 10.15 -0400 04-08-16, Tony Hansen wrote:
>It's a simple rule: the MTA and MSA should only touch the headers their respective protocols say they should be adding or touching. And that list is very small.

If someone wants the MTA to remove Bcc headers when
forwarding a message, and to split a message depending on
whether the recipients are in BCC or not, they should get
IETF to standardize a new ESMTP extension to control this
feature from the MUA in communication with the MTA.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Mon, 16 Aug 2004 12:55:04 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Sender: [email-masked]

Laird Breyer wrote:

> Because the mbox file format is not well defined, any incidental
> reformatting of an attached mbox file during mail transport would be
> harmful.

Specifically, From stuffing and any special treatment of control
characters can cause problems with delimiters.  In the specific
case of a Content-Length delimiter, *any* change to the content
which changes its length (e.g. adding or removing trailing
whitespace on lines) will cause problems.

> So it seems to me that an mbox media type should always require a
> protective content encoding, either QP or Base64.

Perhaps.

> Of those two encodings, Base64 seems much preferable, because the QP
> encoding can allow the mbox format delimiters (From_) to stay
> unencoded.

Note that "From" is valid base64 output and that transport can
result in addition of trailing whitespace.

> This could make proper parsing of, e.g. a QP encoding of
> an mbox file attached to a message saved in an mbox archive,
> needlessly difficult for common tools such as e.g. formail, especially
> if the tool happens to be not MIME aware.

I'm not sure I understand your point; processing of an attached
media type has several prerequisites:
1. a user indicates that processing may proceed (N.B. no
   automatic execution!)
2. a MIME-aware application decodes any transfer encoding &
   extracts the attachment
3. some application capable of processing the content -- it
   need not be MIME-aware
The MIME-aware application (typically a MUA) may pass
additional information (from parameters) to the media-
specific application

Date: Mon, 16 Aug 2004 17:51:08 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
Subject: Re: The Bcc Issue
Cc: Tony Hansen <[email-masked]>, [email-masked]
Sender: [email-masked]

Nathaniel Borenstein writes:
> I think that someone who thinks they have a coherent mental model for 
> all of this should try to write it down.

Rule no. 1: If you don't want a given recipient to see a particular 
header field, don't send it in the first place.

an arrogant
Arnt

Date: Mon, 16 Aug 2004 08:26:31 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Tony Hansen <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> > In that case, it would be reasonable for an MTA to remove a Bcc header
> > that did not agree with the envelope address. Whether that would be
> > expressed as a MAY, SHOULD or MUST is another matter of course :-( .

> No, this is not reasonable at all. Consider the case of a bcc recipient
> whose email is forwarded. The envelope address won't agree with the bcc
> header, but you still don't want the bcc header touched.

In the case of an MTA this is quite correct - forwarding makes it completely
impractical. X.400 made the mistake of thinking envelope-header correlation of
this sort was feasble as part of its MDN design. The result was that MDNs were
a crapshoot in X.400 - sometimes they worked, other times they didn't, and
often as not even exports had difficulty figuring out why. We really don't want
to go here.

Another issue with this is that it creates another condition under which the
MTA would have to perform splitting. (The reasons it does so should be
obvious.) Depending on overall MTA architecture additional splitting cases can
add real complexity, which is bad.

A more interesting case is that of bcc removal by a MSA. My guess is that the
number of cases where an MUA applies different rewrites to header and envelope
so they don't match is fairly small. So having an MSA perform bcc removal is a
considerably safer proposition.

However, this ignores another use-case: That of a file-copy listing all bcc's
that were sent. And the splitting issue remains. There's also the very real
possibility  of creep out of the SUBMISSION space.

It's a much closer call, but I don't think this is worth it even if it were
restricted to SUBMISSION and MSAs.

> It's a simple rule: the MTA and MSA should only touch the headers their
> respective protocols say they should be adding or touching. And that
> list is very small.

Yep. Simple and powerful and perilous to mess with.

				Ned

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: The Bcc Issue
Date: Mon, 16 Aug 2004 11:29:52 -0400
To: Tony Hansen <[email-masked]>
Sender: [email-masked]

Everyone seems to have simple rules, and yet I will freely confess that 
I do not have any coherent model at this point for how bcc is supposed 
to work.  The more I hear the more confused I get.

I think that someone who thinks they have a coherent mental model for 
all of this should try to write it down.  -- Nathaniel

On Aug 16, 2004, at 10:15 AM, Tony Hansen wrote:

>
> Charles Lindsey wrote:
>
> > Dave Crocker <[email-masked]> writes:
>>> In fact it is not unreasonable to have each bcc recipient get a copy 
>>> of
>>> the message with a bcc header that contains (only) their name in it.
>> In that case, it would be reasonable for an MTA to remove a Bcc header
>> that did not agree with the envelope address. Whether that would be
>> expressed as a MAY, SHOULD or MUST is another matter of course :-( .
>
> No, this is not reasonable at all. Consider the case of a bcc 
> recipient whose email is forwarded. The envelope address won't agree 
> with the bcc header, but you still don't want the bcc header touched.
>
> It's a simple rule: the MTA and MSA should only touch the headers 
> their respective protocols say they should be adding or touching. And 
> that list is very small.
>
> 	Tony Hansen
> 	[email-masked]
>
>
>

Date: Mon, 16 Aug 2004 10:15:21 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Charles Lindsey wrote:

 > Dave Crocker <[email-masked]> writes:
>>In fact it is not unreasonable to have each bcc recipient get a copy of
>>the message with a bcc header that contains (only) their name in it.
> 
> In that case, it would be reasonable for an MTA to remove a Bcc header
> that did not agree with the envelope address. Whether that would be
> expressed as a MAY, SHOULD or MUST is another matter of course :-( .

No, this is not reasonable at all. Consider the case of a bcc recipient 
whose email is forwarded. The envelope address won't agree with the bcc 
header, but you still don't want the bcc header touched.

It's a simple rule: the MTA and MSA should only touch the headers their 
respective protocols say they should be adding or touching. And that 
list is very small.

	Tony Hansen
	[email-masked]

Cc: Keith Moore <[email-masked]>, [email-masked], Tony Hansen <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: The Bcc Issue
Date: Mon, 16 Aug 2004 09:31:13 -0400
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

>> we can do a bit better than that - we can also state that we design 
>> and test things to interoperate when they comply with the standards, 
>> and not (in general) under other conditions.  if someone writes or 
>> configures a protocol engine to deviate from the standards, he does 
>> so at the risk of degrading interoperability, and impairing the 
>> network for his own users and possibly for others.  very few users 
>> have the background and analysis skills to understand the 
>> implications of deviating from the standards.
>
> And when the standards are as clear as muck, don't we have an 
> obligation then?
>
> I can't count the number of communications snafus I've seen that I 
> would attribute to the bcc feature.  At the very least, we should 
> provide a clear model for implementors to follow and convey to users.  
> We certainly haven't done that yet, and I'm not sure how we can at 
> this point.

I do think we have an obligation to make specifications clear, and I do 
think it's possible to improve over what we currently have.

Keith

Cc: [email-masked], Tony Hansen <[email-masked]>, [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: The Bcc Issue
Date: Mon, 16 Aug 2004 09:29:13 -0400
To: Keith Moore <[email-masked]>
Sender: [email-masked]

On Aug 16, 2004, at 7:55 AM, Keith Moore wrote:
>
> we can do a bit better than that - we can also state that we design 
> and test things to interoperate when they comply with the standards, 
> and not (in general) under other conditions.  if someone writes or 
> configures a protocol engine to deviate from the standards, he does so 
> at the risk of degrading interoperability, and impairing the network 
> for his own users and possibly for others.  very few users have the 
> background and analysis skills to understand the implications of 
> deviating from the standards.

And when the standards are as clear as muck, don't we have an 
obligation then?

I can't count the number of communications snafus I've seen that I 
would attribute to the bcc feature.  At the very least, we should 
provide a clear model for implementors to follow and convey to users.  
We certainly haven't done that yet, and I'm not sure how we can at this 
point.    -- Nathaniel

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: The Bcc Issue
Date: Mon, 16 Aug 2004 08:04:42 -0400
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

>> In fact it is not unreasonable to have each bcc recipient get a copy 
>> of
>> the message with a bcc header that contains (only) their name in it.
>
> In that case, it would be reasonable for an MTA to remove a Bcc header
> that did not agree with the envelope address. Whether that would be
> expressed as a MAY, SHOULD or MUST is another matter of course :-( .

Actually it's occasionally quite useful to be able to let two Bcc: 
recipients know that they've both been sent the email.  It's just that 
the typical MUA is quite inflexible in letting the user determine 
exactly which Bcc: recipients get to see which Bcc addresses.

Keith

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: To and CC without reply (was: the Bcc issue)
Date: Mon, 16 Aug 2004 08:01:45 -0400
To: Jacob Palme <[email-masked]>
Sender: [email-masked]

> Bcc actually has two functions:
>
> (1) To hide recipients.
>
> (2) To stop replies, i.e. working as a kind of "Non-Reply-To".

good point.  I've certainly used Bcc: in that way.  of course, it has 
the unintended side-effect of hiding those recipients from view by 
other recipients.

> If IETF had standardized a new header "Group-Reply-To"
> (similar to "Follow-Up-To" in Usenet News), and if all
> mailers would by default use these names when people used
> the "reply-to-all" command, this would have been better.

IMHO, To-noreply: and CC-noreply: fields would be far better.  sending 
MUAs would use them to generate envelope recipients just as if they 
were To and Cc.  receiving MUAs would display them, but not send 
replies to them.

Keith

Cc: Keith Moore <[email-masked]>, Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: The Bcc Issue
Date: Mon, 16 Aug 2004 07:55:37 -0400
To: [email-masked]
Sender: [email-masked]

>> perhaps, but an agent should not be performing that part of the MUA's
>> function for messages accepted via the SUBMISSION protocol.
>
> I never said it was or should do so. But what people actually run on 
> various
> ports is, in the final analysis, their own affair. We can only state 
> whether or
> not something is standards compliant.

we can do a bit better than that - we can also state that we design and 
test things to interoperate when they comply with the standards, and 
not (in general) under other conditions.  if someone writes or 
configures a protocol engine to deviate from the standards, he does so 
at the risk of degrading interoperability, and impairing the network 
for his own users and possibly for others.  very few users have the 
background and analysis skills to understand the implications of 
deviating from the standards.

To: [email-masked]
Xref: clerew local.mime:3286
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: The Bcc Issue
Date: Mon, 16 Aug 2004 10:35:03 GMT
Lines: 74
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>Folks,

>I think Nathaniel's note does an excellent job of representing a popular
>line of thinking about these issues. It's critical of that line of
>thinking, rather than of Nathaniel...

>We need to make sure that our assertions about human communications
>issues have some relationship to the normal, non-computer range of human
>behaviors. Unfortunately in the world of email standards,tend to be both
>arbitrary and restrictive in those assertions.

>NB> The essence of a "bcc" is "I want a copy to go to this person, but I
>NB> don't want the 'real' recipient to know it."  Given that

>Nit pick:  The usual term is 'primary', rather than 'real'.

>NB> motivation/intent, writing it down in a header field is a somewhat  
>NB> dubious concept to begin with.

>The concept of making such a note for the author's agent, such as a
>secretary, is not dubious at all.  It is the only way the author can
>tell the agent how to handle the message.

>NB> Anyone who puts in a "bcc" field  
>NB> pretty obviously intends for it to be taken out before it reaches the  
>NB> ultimate recipient.

>In fact that did not used to be standard practise.

>The benefit of having a BCC field in a message is that the recipient
>does not need to guess why they got the message. (Lest anyone says 'if i
>got the message but am not listed in it then i am obviously a bcc
>recipient' we need to observe both the baroqueness (baroquity?) of such
>a (non-)labeling technique and its potential to misbehave.)

>In fact it is not unreasonable to have each bcc recipient get a copy of
>the message with a bcc header that contains (only) their name in it.

In that case, it would be reasonable for an MTA to remove a Bcc header
that did not agree with the envelope address. Whether that would be
expressed as a MAY, SHOULD or MUST is another matter of course :-( .

>In some scenarios, it would even be reasonable to have all the bcc
>recipients get the same copy and have it list all of them.  This is a
>way to keep a special (closed) group informed about some public
>activity, and permit them to continue private discussions about it.

Nice, but not so easy to implement.

>NB>   This suggests, however, an obvious way to handle  
>NB> such headers:  Take them out of the headers and (probably) inject them  
>NB> into the SMTP recipient stream as soon as you possibly can!

Indeed so. With the understanding that any MUA which uses the '-t' option
of its MTA is simply subcontracting part of its MUA activity, and it had
better be aware of exactly what the MTA has been subcontracted to do
(actually, use of the '-t' option of an MTA is more likely to occur in
special-purpose mailing scripts than in regular general-purpose MUAs).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3285
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Mon, 16 Aug 2004 10:26:52 GMT
Lines: 54
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

q>FWIW, I do not regard schemes using Content-length: (there are several
>incompatible variants) in lieu of colonless from lines as being legitimate mbox
>files.

Why not? They exist (they even work in the contexts they were intended
for, which does not include the Wire). You can't bury your head in the
sand and assume otherwise.

Clearly, it needs to be written down somewhere that the Content-Length
MUST NOT occur within individual RFC 2822 messages (and must be
ignored/removed if it does). It is only meaningful where it was created by
the person/entity that created the mbox, and only for as long as the
message remained in that mbox. (A bit like the Xref header in Netnews,
which in only meaningful on the server for which it was written.)

>(1) An optional parameter will likely be neither generated or read. I certainly
>    wouldn't bother with it in the vaarious tools I've developed that deal
>    with mbox files.

>(2) A mandatory parameter stands a good chance of scaring people away from
>    ever using the media type. Alternately, they'll use the media type
>    but ignore the parameter requirement.

Yes, it certainly must not be mandatory.

>(3) To the extent that people would generate such a parameter, I view the
>    chances that it would be done correctly as fairly low.

>(4) There are performance and security issues associated with the use of
>    regular expressions. Writing a regexp that consumes vast amounts of
>    CPU isn't hard, which means anyone developing, say, an automatic
>    tool that accepts and converts labelled mbox files from random sources
>    would be vulnerable. Very large mbox files are pretty common as well -
>    I've seen a bunch weighing in at in excess of 2GB, so the performance
>    impact of regexp scanning also cannot be ignored.

This is one of the things to be looked into when you come to design that
parameter. Yes it is possible to write obfuscated regular expressions, but
that does not mean that people will regularly do it. And remember that it
is only those lines beginning with "^From " that will ever consume any CPU
load.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3284
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Mon, 16 Aug 2004 10:11:43 GMT
Lines: 33
Sender: [email-masked]

In <[email-masked]> "Eric A. Hall" <[email-masked]> writes:

>A definitive authoritative specification for all variations of the mbox
>database format is explicitly not the objective, for several reasons. For
>one thing, such a definition is outside the IETF's purview, the same as a
>definition for Outlook or Eudora or other vendor/platform-centric database
>formats would be.

For standards-track, yes. But if this document (or some other) were to
describe the mbox format in full, with variations listed, then it would be
Informational. That is still not a reason not to include the format and
the tag in the same document. And if the tag is to have some optional
parameters to assist the recipient, then the format and the parameters
need to be considered at the same time when designing the document(s).

Note that the purposes of the parameters is primarily
   1. To ensure that the mbox is split into individual messages correctly
   2. To minimize the need for ">From " stuffing which is an irreversible
      kludge.
The parameters are, or course, optional, but if they are omitted the
recipient has to resort to guesswork (i.e. clues derived from examination
of the acutal content).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3287
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: The Bcc Issue
Date: Mon, 16 Aug 2004 10:37:01 GMT
Lines: 18
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Not just the (Bcc recipient) user; also to remind the author (via his
>file copy) to whom he sent bccs (perhaps some time after the fact).

For those MUAs which maintain a 'sent' folder (not all do), that would be
a sensible place to keep the full Bcc set.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 16 Aug 2004 09:56:39 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Message-ID and Bcc Was: The Bcc Issue
Sender: [email-masked]

Another related issue is whether "Message-ID" should be the 
same or different, when a message is sent to some 
recipients with "Bcc" headers and to other recipients 
without.

Arguments for the same Message-ID: This means that threads 
will be intact for all recipients whose mailers use the 
Message-ID to keep track of threads.

Arguments agains the same Message-ID: Many mailers suppress 
multiple copies of the same message with the same 
Message-ID. Thus, if a person receives two copies of a 
message, one with and one without Bcc, then depending on 
the order of reciept, only one of them will be retained.

Of course, the ideal mailer, which suppresses multiple 
copies of a message, should merge the headers of the 
multiple copies, but this is not very commonly done.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Mon, 16 Aug 2004 09:52:25 +0200
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: The Bcc Issue
Sender: [email-masked]

At 11.44 -0400 04-08-13, Nathaniel Borenstein wrote:
>As one who has always despised the whole concept of bcc's, everything I say should be taken with a large grain of salt.  However...

Bcc actually has two functions:

(1) To hide recipients.

(2) To stop replies, i.e. working as a kind of "Non-Reply-To".

Most MUAs interpret "Reply-To" to mean "Send replies
intended for the originator of this message here", and thus
the common command "Reply-to-all" usually sends a reply to
all addresses in "To", "Cc" and "Bcc". But since non-Bcc
recipients will not see any "Bcc", their
reply-to-all-replies are not sent to "Bcc" recipients.

For example, I sometimes send a copy of a message I have
sent to my boss, but want to protect my boss from seeing
all future messages in a possibly long thread of future
discussion on an issue. To achieve this, I use "Bcc".

I thus do not actually want to hide the fact that my boss
is getting a copy, but I have to use "Bcc" to protect my
boss from getting a long thread of messages.

If IETF had standardized a new header "Group-Reply-To"
(similar to "Follow-Up-To" in Usenet News), and if all
mailers would by default use these names when people used
the "reply-to-all" command, this would have been better.
But, as many of us remember, IETF never could reach
consensus on "Group-Reply-To", and thus "Bcc" is needed to
achieve the effect described above.

Note: Some of you may say that it is the task for the
recipients, not for me, to decide whether my boss should
get a copy of their replies. However,most mail users just
use the "reply-to-all" command in their mailers, not
thinking of who the individual recipients of their reply
will be.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Sun, 15 Aug 2004 20:07:49 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Keith Moore <[email-masked]>
Cc: [email-masked], Keith Moore <[email-masked]>, Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, [email-masked]
Sender: [email-masked]

> > > An MSA listening on either port 25 or 587 should NOT be looking at
> > > To/Cc/Bcc headers. They should only be looking at RCPT TO. See RFC
> > > 2476.

> > Or flipping it around, an agent that examines To/CC/Bcc headers around
> > the time
> > of message submission is considered to be part of the MUA.

> perhaps, but an agent should not be performing that part of the MUA's
> function for messages accepted via the SUBMISSION protocol.

I never said it was or should do so. But what people actually run on various
ports is, in the final analysis, their own affair. We can only state whether or
not something is standards compliant. Something that performs MUA functions on
server side clearly is not compliant with either SMTP or SUBMISSION.

				Ned

Date: Mon, 16 Aug 2004 12:37:05 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 13 2004, John C Klensin wrote:
> To repeat Bruce's point from a
> different perspective, if all that is intended is "here are some
> nice bits, which someone has decided to call "mbox" without any
> real semantics, and any information you are going to get about
> how to process them is coming out of band and by agreement among
> the communicating parties", then application/octet-stream is at
> least nearly adequate.

I agree with this point, and would like to expand it slightly further.
Because the mbox file format is not well defined, any incidental
reformatting of an attached mbox file during mail transport would be
harmful. So it seems to me that an mbox media type should always require a
protective content encoding, either QP or Base64.

Of those two encodings, Base64 seems much preferable, because the QP
encoding can allow the mbox format delimiters (From_) to stay
unencoded.  This could make proper parsing of, e.g. a QP encoding of
an mbox file attached to a message saved in an mbox archive,
needlessly difficult for common tools such as e.g. formail, especially
if the tool happens to be not MIME aware.

-- 
Laird Breyer.

Cc: Keith Moore <[email-masked]>, Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: The Bcc Issue
Date: Sun, 15 Aug 2004 22:27:29 -0400
To: [email-masked]
Sender: [email-masked]

>> An MSA listening on either port 25 or 587 should NOT be looking at
>> To/Cc/Bcc headers. They should only be looking at RCPT TO. See RFC 
>> 2476.
>
> Or flipping it around, an agent that examines To/CC/Bcc headers around 
> the time
> of message submission is considered to be part of the MUA.

perhaps, but an agent should not be performing that part of the MUA's 
function for messages accepted via the SUBMISSION protocol.

Keith

Date: Sun, 15 Aug 2004 17:01:40 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Tony Hansen <[email-masked]>
Cc: Nathaniel Borenstein <[email-masked]>, [email-masked]
Sender: [email-masked]

> An MSA listening on either port 25 or 587 should NOT be looking at
> To/Cc/Bcc headers. They should only be looking at RCPT TO. See RFC 2476.

Or flipping it around, an agent that examines To/CC/Bcc headers around the time
of message submission is considered to be part of the MUA. The fact that
this can happen in a different process or even on a different machine
doesn't make it an MSA.

				Ned

Date: Sun, 15 Aug 2004 19:23:37 -0400
From: Tony Hansen <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
CC: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

An MSA listening on either port 25 or 587 should NOT be looking at 
To/Cc/Bcc headers. They should only be looking at RCPT TO. See RFC 2476.

	Tony Hansen
	[email-masked]

Nathaniel Borenstein wrote:

> OK, fine, got me again, I meant an MSA -- which is generally, in 
> practice, the same code as an MTA, but functioning as an MSA (e.g. 
> sendmail listening on port 25 or 587 for message submission).
> 
> As to why this is worse than the similar situation with To/CC/etc 
> (Keith's question) -- nobody ever *intends* for those field values to be 
> hidden from the recipient, as they do with bcc.  -- Nathaniel

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: The Bcc Issue
Date: Sun, 15 Aug 2004 11:44:47 -0400
To: Tony Hansen <[email-masked]>
Sender: [email-masked]

OK, fine, got me again, I meant an MSA -- which is generally, in 
practice, the same code as an MTA, but functioning as an MSA (e.g. 
sendmail listening on port 25 or 587 for message submission).

As to why this is worse than the similar situation with To/CC/etc 
(Keith's question) -- nobody ever *intends* for those field values to 
be hidden from the recipient, as they do with bcc.  -- Nathaniel

On Aug 13, 2004, at 10:03 PM, Tony Hansen wrote:

> Nathaniel Borenstein wrote:
>
>> I stand corrected:  Three different functions.  The line
>>     BCC: [email-masked]
>> might better be represented as
>>     I-Sent-a-BCC-To: [email-masked] (for my files)
>>     BCC-Should-Be-Sent-To:  [email-masked]
>>         (written by a UA as a request to an MTA)
>>     Why-You-Got-This:  You were BCC'ed as [email-masked]         
>> (for viewing by the bcc recipient)
>
> Nit here on your second item: An MTA never looks at the header of a 
> message! The MTA is expected to look at an envelope, not the header. 
> If you're passing a message to something that IS expected to look at 
> the To/Cc/Bcc headers and act on them, that thing is NOT an MTA.
>
> Even MSA's are not expected to look at the To/Cc/Bcc headers.
>
> Now, if you want to argue that some software DOES look at those 
> headers to determine the addresses to send a message to, that's fine. 
> Those software packages are working as part of the MUA system, and 
> it's a private contract between those software packages to do so.
>
> But an MTA should NOT be looking at the To/Cc/Bcc headers.
>
> 	Tony Hansen
> 	[email-masked]
>
>
>
>

From: Robert Elz <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard 
Date: Sat, 14 Aug 2004 21:04:59 +0700
Sender: [email-masked]

    Date:        Fri, 13 Aug 2004 11:33:18 -0400
    From:        "Eric A. Hall" <[email-masked]>
    Message-ID:  <[email-masked]>

  | Others should note that RFC2048 is designed to facilitate registrations --
  | more definitions for common data-types are widely preferred over a
  | proliferation of "x-foo" media-types that result from high registration
  | barriers (read the intro to 2048 if you don't believe me) -- and that's
  | the limited objective of this proposal.

All that is fine, and not an issue with anyone.   The problem is that for
the tag to be meaningful, there must be at the very least a common
understanding of what it means and how it can be processed.   Better of
course is an actual published specification of the content format (somewhere).

Without that all you have is a label for a meaningless bucket of bits.

When I send a message how do I know whether it is appropriate to use the
new tag or not?   What content qualifies, and what does not?   Is it
sufficient if I'm sending a file whose name is "mbox"?   (Or which
has an extension of ".mbox" or similar).   Or is it necessary that the
content contain mail in some form?   If so, is it enough if I have a jpeg
file showing a mail box containing letters?

It is 100% useless to define a label without that referring to some
well understood content.

kre

Date: Fri, 13 Aug 2004 23:03:44 -0400
From: John C Klensin <[email-masked]>
To: "Eric A. Hall" <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed	Standard
Sender: [email-masked]

--On Friday, 13 August, 2004 11:33 -0400 "Eric A. Hall"
<[email-masked]> wrote:

>...
> Suggestions about parameters has come up before (John Klensin
> suggested it to me a couple of months ago). Unfortunately,
> these kind of helper tags attempt to define content rules
> rather than transfer rules, and therefore represent a
> non-trivial layer violation.

Whoops!   Whatever the other issues here,  content-type does
define content rules and not transfer rules (transfer rules are
captured in C-T-E), and parameters/hints on content-type are
just more content rules.   So I don't have any idea what you are
talking about when you refer to a layer violation.

To restate that position more generally, the purpose of having
content types is to permit the recipient  to figure out how to
parse/process the content.  To repeat Bruce's point from a
different perspective, if all that is intended is "here are some
nice bits, which someone has decided to call "mbox" without any
real semantics, and any information you are going to get about
how to process them is coming out of band and by agreement among
the communicating parties", then application/octet-stream is at
least nearly adequate.  From that point of view, if you want an
IETF type, you should either have to claim that all (or nearly
all) of the applications that process MBOX format critters can
process all variations on the format, or there really need to be
clues in either the type or the parameters.  Otherwise, you are
just creating a "gotcha" situation, and that is not what content
types are supposed to be able.

> They are analogous to using a
> "version=" tag for app/postscript and relying on that
> meta-information instead of embedded clue data. Obviously,
> content agents should be aware of the content formatting
> rules. [what happens when the helper meta-tags are lost?
> should the content agent NOT look at the content to make a
> determination? that's where the logic belongs in the first
> place, so putting it into the transfer layer is not only
> irrelevant it is possibly harmful.] This data should not be
> overflowed to the transfer agent except where it affects the
> transfer process proper, which is not the case with any of the
> suggested tags.

Again, in the context of MIME media type definitions, I have no
idea what you are talking about.

     john

Date: Fri, 13 Aug 2004 22:03:53 -0400
From: Tony Hansen <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
CC: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Nathaniel Borenstein wrote:

> 
> I stand corrected:  Three different functions.  The line
> 
>     BCC: [email-masked]
> 
> might better be represented as
> 
>     I-Sent-a-BCC-To: [email-masked] (for my files)
>     BCC-Should-Be-Sent-To:  [email-masked]
>         (written by a UA as a request to an MTA)
>     Why-You-Got-This:  You were BCC'ed as [email-masked]         (for 
> viewing by the bcc recipient)

Nit here on your second item: An MTA never looks at the header of a 
message! The MTA is expected to look at an envelope, not the header. If 
you're passing a message to something that IS expected to look at the 
To/Cc/Bcc headers and act on them, that thing is NOT an MTA.

Even MSA's are not expected to look at the To/Cc/Bcc headers.

Now, if you want to argue that some software DOES look at those headers 
to determine the addresses to send a message to, that's fine. Those 
software packages are working as part of the MUA system, and it's a 
private contract between those software packages to do so.

But an MTA should NOT be looking at the To/Cc/Bcc headers.

	Tony Hansen
	[email-masked]

Cc: Keith Moore <[email-masked]>, Dave Crocker <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: The Bcc Issue
Date: Fri, 13 Aug 2004 21:04:13 -0400
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

> But the bottom line is that software uses bcc for two very different 
> functions:  to tell another email transport-related entity to send a 
> bcc, or to tell the user that he has received a bcc.

While I agree, I don't see how the dual use of Bcc is worse than the 
dual uses of the To and Cc fields.

To: Tony Hansen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard 
From: [email-masked]
Date: Fri, 13 Aug 2004 20:40:05 -0400
Sender: [email-masked]

--==_Exmh_-96652284P
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, 12 Aug 2004 17:18:19 EDT, Tony Hansen said:
> The information about the mbox format being anecdotally defined is=20
> incorrect. The mbox format has traditionally been documented in the=20
> binmail(1) or mail.local(8) man pages (BSD UNIX derivatives) or mail(1)=
=20
> man page (UNIX System 3/5/III/V derivatives). There have been several=20
> variants of the mbox format in use by those different systems. The most=
=20
> complete description of an mbox format can be seen in the man page from=
=20
> any UNIX System Vr4 derived system, such as Solaris.

Umm.. Tony?  I hate to say it, but if there have been several variants us=
ed in
the wild, and the man pages for said variants document different formats,=

that's awfully close to =22anecdotally defined=22 when you're doing a sta=
ndard.

For example, a Solaris 8 box across the hall says in 'man mail.local':

     Each delivered mail message in the mailbox is preceded by  a
     =22Unix From line=22 with the following format:

          From sender_address time_stamp

     The sender_address  is  extracted  from  the  SMTP  envelope
     address  (the  envelope  address  is  specified  with the -f
     option).

     A trailing blank line is also added to the end of each  mes-
     sage.

Hmm. Nothing about whether the sender_address is, or should be, <brackete=
d>.
Nothing about the format of the time_stamp. Nothing about '>From ' stuffi=
ng
(and yes, I've seen systems that don't do it at all, and systems that onl=
y
>-stuff if the From line matched a regexp for what *they* think the entir=
e 'From '
line looks like(*)). The Sendmail 8.13.1 mail.local does say >-stuffing
happens for lines that =22which could be mistaken for a =60=60From '' del=
imiter
line=22, and the code actually checks for exactly 5 chars...

Any doubts that this whole mess is at best anecdotally defined can be dis=
pelled by
mentioning =22Content-Length:=22 (interestingly enough, not even mentione=
d in the
Solaris or Sendmail man pages, although the Sendmail source tree does men=
tion
that building on Solaris 2.3 or later will turn it on.  Of interest mostl=
y because
the Content-Length: is so easily broken by later >-stuffing/unstuffing or=
 other
similar conversion...

(*) time_stamp. Argh.  Fought with this during a data/machine migration.
Write code that will accept a 26 byte ctime format: 'Fri Sep 13 00:00:00 =
1986=5Cn=5C0'.
Works fine once you realize that some systems just used 'From envelop_add=
ress'
without a timestamp.

Then I get handed this: 'Fri Aug 13 20:21:32 EDT 2004'.  Fix that, and fi=
nd some
joker running in a French locale: 'vendredi, 13 ao=FBt 2004, 20:22:01 EDT=
'.
And yes, his b0rked software only >-stuffed 'From ' lines that regexp-mat=
ched
the *French* variant. Took me *quite* some time to twig into THAT one...

--==_Exmh_-96652284P
Content-Type: application/pgp-signature

--==_Exmh_-96652284P--

From: Nathaniel Borenstein <[email-masked]>
Subject: Re: The Bcc Issue
Date: Fri, 13 Aug 2004 19:44:53 -0400
To: [email-masked]
Sender: [email-masked]

I stand corrected:  Three different functions.  The line

	BCC: [email-masked]

might better be represented as

	I-Sent-a-BCC-To: [email-masked] (for my files)
	BCC-Should-Be-Sent-To:  [email-masked]
		(written by a UA as a request to an MTA)
	Why-You-Got-This:  You were BCC'ed as [email-masked] 		(for viewing 
by the bcc recipient)

None of this makes me any more fond of bcc's.  -- N

On Aug 13, 2004, at 7:15 PM, Bruce Lilly wrote:

>
> Nathaniel Borenstein wrote:
>
>> But the bottom line is that software uses bcc
>> for two very different functions:  to tell another email
>> transport-related entity to send a bcc, or to tell the user that he 
>> has
>> received a bcc.
>
> Not just the (Bcc recipient) user; also to remind the author (via his
> file copy) to whom he sent bccs (perhaps some time after the fact).
>
>
>

Date: Fri, 13 Aug 2004 19:15:30 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Nathaniel Borenstein <[email-masked]>
CC: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Nathaniel Borenstein wrote:

> But the bottom line is that software uses bcc
> for two very different functions:  to tell another email
> transport-related entity to send a bcc, or to tell the user that he has
> received a bcc.

Not just the (Bcc recipient) user; also to remind the author (via his
file copy) to whom he sent bccs (perhaps some time after the fact).

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: The Bcc Issue
Date: Fri, 13 Aug 2004 18:27:56 -0400
To: Dave Crocker <[email-masked]>
Sender: [email-masked]

It sure would have been good if we had defined this about 20 years ago.

It was not my intention to imply that you shouldn't leave the bcc 
header in for the viewing of the person named in it, merely that it be 
removed from the view of anyone *else* as early in the process as 
possible.  I don't much mind leaving in the bcc that explains why you 
received it, although I tend to prefer "Apparently-to" for that purpose 
(yes, I know, totally non standard).  But the bottom line is that 
software uses bcc for two very different functions:  to tell another 
email transport-related entity to send a bcc, or to tell the user that 
he has received a bcc.  I don't see how this dual use can possibly be a 
good state of affairs.

For me, though, the most important issue is not violating user-level 
expectations, particularly when they are privacy-sensitive.  
Maintaining the alleged purity of the email architecture is not a 
winning argument by comparison.  -- Nathaniel

On Aug 13, 2004, at 5:52 PM, Dave Crocker wrote:

>
> Folks,
>
> I think Nathaniel's note does an excellent job of representing a 
> popular
> line of thinking about these issues. It's critical of that line of
> thinking, rather than of Nathaniel...
>
> We need to make sure that our assertions about human communications
> issues have some relationship to the normal, non-computer range of 
> human
> behaviors. Unfortunately in the world of email standards,tend to be 
> both
> arbitrary and restrictive in those assertions.
>
>
> NB> The essence of a "bcc" is "I want a copy to go to this person, but 
> I
> NB> don't want the 'real' recipient to know it."  Given that
>
> Nit pick:  The usual term is 'primary', rather than 'real'.
>
>
> NB> motivation/intent, writing it down in a header field is a somewhat
> NB> dubious concept to begin with.
>
> The concept of making such a note for the author's agent, such as a
> secretary, is not dubious at all.  It is the only way the author can
> tell the agent how to handle the message.
>
>
> NB> Anyone who puts in a "bcc" field
> NB> pretty obviously intends for it to be taken out before it reaches 
> the
> NB> ultimate recipient.
>
> In fact that did not used to be standard practise.
>
> The benefit of having a BCC field in a message is that the recipient
> does not need to guess why they got the message. (Lest anyone says 'if 
> i
> got the message but am not listed in it then i am obviously a bcc
> recipient' we need to observe both the baroqueness (baroquity?) of such
> a (non-)labeling technique and its potential to misbehave.)
>
> In fact it is not unreasonable to have each bcc recipient get a copy of
> the message with a bcc header that contains (only) their name in it.
>
> In some scenarios, it would even be reasonable to have all the bcc
> recipients get the same copy and have it list all of them.  This is a
> way to keep a special (closed) group informed about some public
> activity, and permit them to continue private discussions about it.
>
>
> NB>   This suggests, however, an obvious way to handle
> NB> such headers:  Take them out of the headers and (probably) inject 
> them
> NB> into the SMTP recipient stream as soon as you possibly can!
>
> NB> In other words, a conservative default would be that *any* software
> NB> that received a message with such a field should feel very free to 
> take
> NB> out the field, and should probably send the bcc as well.
>
> I am trying to imagine the meta-policy that is behind this being
> generally acceptable: any software that receives a message with some
> objectionable headers should feel free to take out the field?
>
>
> NB>  The nastiest
> NB> failure case is when the message is delivered with the bcc field
> NB> intact, so it seems to me we should try to err in the other 
> direction
> NB> and pre-empt a faux pas if we can.
>
> First of all, we are talking about the MSA or MUA.  Not 'any software'.
>
> Second of all, Internet standards like this would do better to have no
> 'default' for such situations. Defaults invite problems with the
> expectation differing between the creator and the consumer of the
> defaulted material.
>
>
> NB> The Puritan in me thinks you should have to type "~b" at an 
> absolute
> NB> minimum in order to send a bcc, and probably something even more
> NB> obscure.  :-)  But it's still the software's job to try to 
> minimize the
> NB> harm that  happens to you once you use the feature, which is why I
> NB> argue for implementing this at more or less *every* protocol hop.  
> --
>
> but, then, we don't do user interface standardization here.
>
> d/
> --
>  Dave Crocker <dcrocker-at-brandenburg-dot-com>
>  Brandenburg InternetWorking <www.brandenburg.com>
>  Sunnyvale, CA  USA <tel:+1.408.246.8253>
>
>
>

Date: Fri, 13 Aug 2004 14:52:45 -0700
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Folks,

I think Nathaniel's note does an excellent job of representing a popular
line of thinking about these issues. It's critical of that line of
thinking, rather than of Nathaniel...

We need to make sure that our assertions about human communications
issues have some relationship to the normal, non-computer range of human
behaviors. Unfortunately in the world of email standards,tend to be both
arbitrary and restrictive in those assertions.

NB> The essence of a "bcc" is "I want a copy to go to this person, but I
NB> don't want the 'real' recipient to know it."  Given that

Nit pick:  The usual term is 'primary', rather than 'real'.

NB> motivation/intent, writing it down in a header field is a somewhat  
NB> dubious concept to begin with.

The concept of making such a note for the author's agent, such as a
secretary, is not dubious at all.  It is the only way the author can
tell the agent how to handle the message.

NB> Anyone who puts in a "bcc" field  
NB> pretty obviously intends for it to be taken out before it reaches the  
NB> ultimate recipient.

In fact that did not used to be standard practise.

The benefit of having a BCC field in a message is that the recipient
does not need to guess why they got the message. (Lest anyone says 'if i
got the message but am not listed in it then i am obviously a bcc
recipient' we need to observe both the baroqueness (baroquity?) of such
a (non-)labeling technique and its potential to misbehave.)

In fact it is not unreasonable to have each bcc recipient get a copy of
the message with a bcc header that contains (only) their name in it.

In some scenarios, it would even be reasonable to have all the bcc
recipients get the same copy and have it list all of them.  This is a
way to keep a special (closed) group informed about some public
activity, and permit them to continue private discussions about it.

NB>   This suggests, however, an obvious way to handle  
NB> such headers:  Take them out of the headers and (probably) inject them  
NB> into the SMTP recipient stream as soon as you possibly can!

NB> In other words, a conservative default would be that *any* software  
NB> that received a message with such a field should feel very free to take  
NB> out the field, and should probably send the bcc as well.

I am trying to imagine the meta-policy that is behind this being
generally acceptable: any software that receives a message with some
objectionable headers should feel free to take out the field?

NB>  The nastiest  
NB> failure case is when the message is delivered with the bcc field  
NB> intact, so it seems to me we should try to err in the other direction  
NB> and pre-empt a faux pas if we can.

First of all, we are talking about the MSA or MUA.  Not 'any software'.

Second of all, Internet standards like this would do better to have no
'default' for such situations. Defaults invite problems with the
expectation differing between the creator and the consumer of the
defaulted material.

NB> The Puritan in me thinks you should have to type "~b" at an absolute
NB> minimum in order to send a bcc, and probably something even more  
NB> obscure.  :-)  But it's still the software's job to try to minimize the  
NB> harm that  happens to you once you use the feature, which is why I  
NB> argue for implementing this at more or less *every* protocol hop.  --  

but, then, we don't do user interface standardization here.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Fri, 13 Aug 2004 11:06:33 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Arnt Gulbrandsen <[email-masked]>
Cc: Philip Hazel <[email-masked]>, [email-masked]
Sender: [email-masked]

> there's an RFC covering smarthosts now, 2476. It hasn't been very widely
> deployed, but IMO no good MUA can be blatantly incompatible with it.

> RFC 2476 doesn't suggest that the message submission servers hould remove bcc.

> For this reason, I think that a MUA which wants bcc removed cannot trust
> the smarthost to do so. It must do the job itself.

I agree with this assessment. An MUA that cares about bcc handling has no
choice but to do it itself, if for no other reason than it cannot count on it
being done elsewhere. I happen to think the transport infrastructure shouldn't
muck with bcc, but whether it is correct or not really doesn't change the 
reality for an MUA.

> Philip Hazel writes:
> > 4. Among the MUAs, I think that Pine never sends Bcc: lines, but Mutt by
> >    default does, assuming that the MTA will deal with them (Mutt does
> >    not use the -t option).

> IMNSHO, mutt is in the wrong.

I agree.

Although I recognize that customers can and will demand that MSAs/MTAs/MTAs
be able to remove bcc and have provided the means to do it in my own
code, I regard it as a misconfiguration to actually set things up this way.

				Ned

Date: Fri, 13 Aug 2004 14:34:24 -0700 (PDT)
From: [email-masked]
Subject: Re: The Bcc Issue
To: Nathaniel Borenstein <[email-masked]>
Cc: Philip Hazel <[email-masked]>, [email-masked]
Sender: [email-masked]

> As one who has always despised the whole concept of bcc's, everything I
> say should be taken with a large grain of salt.  However...

> The essence of a "bcc" is "I want a copy to go to this person, but I
> don't want the 'real' recipient to know it."  Given that
> motivation/intent, writing it down in a header field is a somewhat
> dubious concept to begin with.   Anyone who puts in a "bcc" field
> pretty obviously intends for it to be taken out before it reaches the
> ultimate recipient.  This suggests, however, an obvious way to handle
> such headers:  Take them out of the headers and (probably) inject them
> into the SMTP recipient stream as soon as you possibly can!

This IMO completely misses the point of the bcc field. The reason the
field is there is not to notify all the recipients that a blind copy
was made. That would be silly. Rather, the point is to notify the
bcc recipient that his or her copy is a bcc and should be handled
accordingly.

> In other words, a conservative default would be that *any* software
> that received a message with such a field should feel very free to take
> out the field, and should probably send the bcc as well.

This approach actually presents something of a security exposure, as it results
in someone getting a copy of message without any indication it was done as a
bcc. This is the reason that some UAs have started stuffing "This is a bcc"
text into the message body (often without  regard to things like MIME,
unfortunately).

> The nastiest
> failure case is when the message is delivered with the bcc field
> intact, so it seems to me we should try to err in the other direction
> and pre-empt a faux pas if we can.

The nastiest case is when a MUA incorrectly assumes that bcc will
be deleted somewhere down the line and it isn't. But the case where
the MUA assumes that bcc will be carried through and it isn't also
can lead to exposures, albeit less directly.

				Ned

Date: Fri, 13 Aug 2004 15:18:28 -0400
From: Tony Hansen <[email-masked]>
To: Philip Hazel <[email-masked]>
CC: [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

Philip Hazel wrote:

> The issue of who handles Bcc: header lines has again been raised, and I
> am seeking opinions as widely as I can. The requirement is
> straightforward: non-Bcc recipients of a message should not see the
> addresses of any Bcc recipients, that is, their copies of the message
> should not contain Bcc: header lines.

The MUA, however it is constituted, should handle all BCC processing, 
and the cloning of messages with and without the Bcc: header.

IMO, each Bcc: recipient should receive a copy with the Bcc: header for 
themself and only themself. No other copies should mention that Bcc: 
recipient.

	Tony Hansen
	[email-masked]

Date: Fri, 13 Aug 2004 10:26:50 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Keith Moore <[email-masked]>
Cc: Eric A Hall <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> > A definitive authoritative specification for all variations of the
> > mbox database format is explicitly not the objective, for several
> > reasons.

> that's fine.  I fully support registering application/mbox as a media
> type.

> > For
> > one thing, such a definition is outside the IETF's purview, the same
> > as a definition for Outlook or Eudora or other vendor/platform-centric
> > database formats would be.

> I have to disagree here.  Perhaps standardizing mbox would be outside
> of IETF's purview, but I think it would be valuable for IETF to publish
> a peer-reviewed Informational description of (1) existing practice for
> mbox files and (2) recommended practice for reading and writing mbox
> files.  I just don't think that this should be critical path for getting
> application/mbox registered.

I agree. Such a specification shouldn't be a requirement for this registration,
but an informational document would be a nice thing to have. Whether or not
sufficient energy exists to get it done is another matter - a similar
informational document on the many variations of uuencode has been on my to-do
for well over a decade now, and I don't see myself getting to it any time
soon...

				Ned

Date: Fri, 13 Aug 2004 10:02:34 -0700 (PDT)
From: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
To: Keith Moore <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> > > > This proposal needs to be sent back for further consideration, either
> > > > by a
> > > > WG, or at least by some mailing list of those with knowledge of the
> > > > format.

> > > > It has just missed too many opportunities, including the opportunity to
> > > > document the mbox format once and for all (and warts and all).

> > > uh, no.  this is an application for a media-type.  it's not trying to
> > > define the mbox format, it's just trying to define a label for the mbox
> > > format that can be used in contexts where MIME media-type labels are
> > > required.  the intent is clearly that this label should be usable with
> > > the variety of mbox files that exist, not with one specific variant of
> > > mbox file.

> > But to do that, it needs to specify which variant of mbox the data is, and
> > to do that, there needs to be at least a description of the various
> > variants.

> there are lots of media types for which there isn't a description of all of
> the variants available.  we find it useful to label things even without
> those descriptions.

Exactly right. A label can be of value even if the thing it labels is  a bit
fuzzy.

Eric Hall already made this point, but it bears repeating: Having names names
for commonly used formats is a lot more important than insisting on precise
definitiosn of those formats. MIME tried it the other way at first, and we
still haven't fully recovered from the mess it led to operationally.

> > From looking at the formail man page (just something quickly found on my
> > machine), I see mention of a variant using Content-Length:; a "traditional
> > Berkeley mbox format"; and a "BABYL rmail format" (*is* that mbox?).

> no, BABYL format is not the same as mbox format.

FWIW, I do not regard schemes using Content-length: (there are several
incompatible variants) in lieu of colonless from lines as being legitimate mbox
files.

> > > trying to write a definitive specification for the mbox format is not
> > > the same thing as trying to define a label for that format.  nor is it
> > > the same thing as trying to document existing practice.  many authors
> > > and groups get these confused, with frequently unfortunate results.

> > However (again), it *does* need to provide a way to identify the variants
> > in actual use. If it doesn't tell me which variant of mbox it is, it might
> > as well just say "application/octet-stream".

> lots of tools seem to be able to make use of mbox files without knowing
> precisely which variant they're reading.

Bingo.

I will also add that the addition of parameters specifying regular expressions
will almost certainly harm interoperability more than it helps it. There
are many reasons for this:

(1) An optional parameter will likely be neither generated or read. I certainly
    wouldn't bother with it in the vaarious tools I've developed that deal
    with mbox files.

(2) A mandatory parameter stands a good chance of scaring people away from
    ever using the media type. Alternately, they'll use the media type
    but ignore the parameter requirement.

(3) To the extent that people would generate such a parameter, I view the
    chances that it would be done correctly as fairly low.

(4) There are performance and security issues associated with the use of
    regular expressions. Writing a regexp that consumes vast amounts of
    CPU isn't hard, which means anyone developing, say, an automatic
    tool that accepts and converts labelled mbox files from random sources
    would be vulnerable. Very large mbox files are pretty common as well -
    I've seen a bunch weighing in at in excess of 2GB, so the performance
    impact of regexp scanning also cannot be ignored.

(5) Most mbox delimiter lines involve one or more spaces. Spaces in
    MIME parameter values are sometimes problematic due to header
    folder, and while parameter encoding exists as a solution to this
    problem the number of agents that don't support it is pretty large.

I should also note that specifying a boundary and line terminator in a
parameter might not even be sufficient; there's also from stuffing to consider.

Now, some may be inclined to compare this sort of parameter with the boundary
parameter on multipart. However, the two cases are actually fairly different.
For one thing, the first line of an mbox provides you with a sample delimiter;
there's no need to scan for it and no chance of it getting confused with other
stuff in the preamble.  Additionally, mbox stores a sequential list of message
wherease MIME multiparts can be nested; this leads to interesting cases where
it can be useful to have a definitive indictor of the boundary. (The obvious
example is a maformed multipart with no legal boundaries stored inside a
legitimate multipart.) And finally, multipart boundaries are fixed strings;
there are no regpexps in sight.

				Ned

Date: Fri, 13 Aug 2004 18:55:41 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Philip Hazel <[email-masked]>
Subject: Re: The Bcc Issue
Cc: [email-masked]
Sender: [email-masked]

Hi,

there's an RFC covering smarthosts now, 2476. It hasn't been very widely 
deployed, but IMO no good MUA can be blatantly incompatible with it.

RFC 2476 doesn't suggest that the message submission servers hould remove bcc.

For this reason, I think that a MUA which wants bcc removed cannot trust 
the smarthost to do so. It must do the job itself.

Philip Hazel writes:
> 4. Among the MUAs, I think that Pine never sends Bcc: lines, but Mutt by
>    default does, assuming that the MTA will deal with them (Mutt does
>    not use the -t option).

IMNSHO, mutt is in the wrong.

Whether exim is right is a different question. I guess that boils down 
to whether you prefer to Do Things Right or add workarounds for 
brokenware.

Arnt

Date: Fri, 13 Aug 2004 11:52:30 -0400
From: Keith Moore <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

> 
> On Aug 13, 2004, at 11:21 AM, Keith Moore wrote:
> 
> > sometimes I think we need a separate label for documents -
> > "least harmful current practice".  but we don't have one yet.
> 
> We have the label, and it's "best current practice" -- you just have
> to resign yourself to the fact that sometimes the best you can do is
> to minimize the harm people do. 

the problem with BCP is that it's a recommendation.  essentially BCP is
a standard for something for which interoperability tests aren't
suitable.  

there's a useful distinction between "here's what we recommend and
believe will work well" and "we don't recommend this, but if you're
going to do this, please do it this way".  (or maybe even "we don't
recommend this, but if you're going to do this, at least don't do it any
of these ways...")

Date: Fri, 13 Aug 2004 11:47:34 -0400
From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

> A definitive authoritative specification for all variations of the
> mbox database format is explicitly not the objective, for several
> reasons. 

that's fine.  I fully support registering application/mbox as a media
type.

> For
> one thing, such a definition is outside the IETF's purview, the same
> as a definition for Outlook or Eudora or other vendor/platform-centric
> database formats would be. 

I have to disagree here.  Perhaps standardizing mbox would be outside
of IETF's purview, but I think it would be valuable for IETF to publish
a peer-reviewed Informational description of (1) existing practice for 
mbox files and (2) recommended practice for reading and writing mbox
files.  I just don't think that this should be critical path for getting
application/mbox registered.

Cc: [email-masked], "Charles Lindsey" <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Fri, 13 Aug 2004 11:44:50 -0400
To: Keith Moore <[email-masked]>
Sender: [email-masked]

On Aug 13, 2004, at 11:21 AM, Keith Moore wrote:

> sometimes I think we need a separate label for documents -
> "least harmful current practice".  but we don't have one yet.

We have the label, and it's "best current practice" -- you just have to 
resign yourself to the fact that sometimes the best you can do is to 
minimize the harm people do.  Sort of like best current environmental 
practices for oil companies.  From a perspective of spam control, for 
example, the truly *best* current practice would be to stop using email 
altogether.  -- Nathaniel

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: The Bcc Issue
Date: Fri, 13 Aug 2004 11:44:45 -0400
To: Philip Hazel <[email-masked]>
Sender: [email-masked]

As one who has always despised the whole concept of bcc's, everything I  
say should be taken with a large grain of salt.  However...

The essence of a "bcc" is "I want a copy to go to this person, but I  
don't want the 'real' recipient to know it."  Given that  
motivation/intent, writing it down in a header field is a somewhat  
dubious concept to begin with.   Anyone who puts in a "bcc" field  
pretty obviously intends for it to be taken out before it reaches the  
ultimate recipient.  This suggests, however, an obvious way to handle  
such headers:  Take them out of the headers and (probably) inject them  
into the SMTP recipient stream as soon as you possibly can!

In other words, a conservative default would be that *any* software  
that received a message with such a field should feel very free to take  
out the field, and should probably send the bcc as well.  The nastiest  
failure case is when the message is delivered with the bcc field  
intact, so it seems to me we should try to err in the other direction  
and pre-empt a faux pas if we can.

Thus, I would argue ever-so-impartially that mutt and exim are both  
broken and should both be fixed.  Exim might be easier to fix, since  
it's just a matter of changing the default setting.  Mutt's objections  
might be more philosophical (see my PS) but physicians need to treat  
patients who haven't followed their advice, and Mutt should do its best  
to protect its users if they send bcc's. -- Nathaniel

PS -- Aside from the ambiguities in their  
interpretation/implementation, I despise bcc's as Temptations to Sin.   
While a few situations (e.g. fyis to celebrities) clearly justify  
bcc's, more often bcc's promote poor communication and bad behavior.   
The Puritan in me thinks you should have to type "~b" at an absolute  
minimum in order to send a bcc, and probably something even more  
obscure.  :-)  But it's still the software's job to try to minimize the  
harm that  happens to you once you use the feature, which is why I  
argue for implementing this at more or less *every* protocol hop.  --  
NB

On Aug 13, 2004, at 6:24 AM, Philip Hazel wrote:

>
> The Bcc Issue: posted to the exim-users, exim-dev, and ietf-822  
> mailing lists
> ----------------------------------------------------------------------- 
> ------
>
> The issue of who handles Bcc: header lines has again been raised, and I
> am seeking opinions as widely as I can. The requirement is
> straightforward: non-Bcc recipients of a message should not see the
> addresses of any Bcc recipients, that is, their copies of the message
> should not contain Bcc: header lines.
>
> The dispute is over who achieves this end by removing Bcc: lines when
> they should not be present. Is it the MUA or the MTA? This is what I
> know about current behaviour:
>
> 1. Exim (which I maintain) removes Bcc: lines if, and only if, it is
>    called with the -t option. In that case, it is constructing an
>    envelope from the header data, and IMHO in doing so it is fulfilling
>    an MUA function. In all other cases it leaves Bcc: lines alone.
>
> 2. It has been reported to me that OpenWave InterMail, MS SMTP, MS
>    Exchange, and qmail behave in the same way as Exim for incoming SMTP
>    messages (i.e. they leave Bcc: lines alone).
>
> 3. I have also been told that Sendmail, at least in some  
> configurations,
>    always removes any Bcc: lines from any message that it handles. The
>    same is apparently true of Postfix and Lotus Notes 5.
>
> 4. Among the MUAs, I think that Pine never sends Bcc: lines, but Mutt  
> by
>    default does, assuming that the MTA will deal with them (Mutt does
>    not use the -t option).
>
> The (re-)discussion of this issue arose because the combination of
> (default configured) Mutt and Exim does not behave the way people
> expect. Mutt sends Bcc: lines, and Exim lets them through to all
> recipients.
>
> The RFC permits several different behaviours in connection with Bcc
> recipients. For copies of the message that are sent to Bcc recipients,
> the Bcc: line may be absent, or contain just the address of the single
> recipient, or contain the addresses of all the Bcc recipients. For
> copies sent to non-Bcc recipients, there should be no Bcc: header line.
>
> If the MUA handles this, it is possible for it to allow any of the
> alternative behaviours, as the sender of the message wishes. An MTA has
> no means of distinguishing what the sender wanted. If it is to handle
> the Bcc: header at all, the only choice is always to remove it. Thus,  
> an
> MTA that always removes Bcc: header lines will frustrate the wishes of
> someone who wants them included in copies to Bcc recipients. (I found
> one posting on the www that complained of exactly this problem.)
>
> Of course, it is possible for MTAs to have configuration options to
> change their behaviour. (It is easy to configure Exim always to remove
> Bcc: lines, for instance.) However, only clueful sysadmins who are  
> aware
> of the issue will investigate such options. Thus, it is important for
> the default behaviour to be "correct" in the sense that it is what the
> RFCs and the community experts generally agree on.
>
> So .... please vent your opinions and I will try to pull together a
> summary in due course.
>
> I am posting this separately to the three list mentioned above because
> the Exim lists are subscriber-posting only, and a cross-post might  
> cause
> problems with replies.
>
> Philip
>
> -- 
> Philip Hazel            University of Cambridge Computing Service,
> [email-masked]      Cambridge, England. Phone: +44 1223 334714.
>
>
>

Date: Fri, 13 Aug 2004 11:33:18 -0400
From: "Eric A. Hall" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Keith Moore <[email-masked]> wrote on Thu, 12 Aug 2004 08:45:24 -0400:

> this is an application for a media-type. it's not trying to define the
> mbox format, it's just trying to define a label for the mbox format
> that can be used in contexts where MIME media-type labels are required.

That's correct; this is a tag definition, not a format specification.

Others should note that RFC2048 is designed to facilitate registrations --
more definitions for common data-types are widely preferred over a
proliferation of "x-foo" media-types that result from high registration
barriers (read the intro to 2048 if you don't believe me) -- and that's
the limited objective of this proposal.

There are a few places where this tag is necessary or useful. Online
downloadable archives of mailing lists would be easier to import, search
and otherwise manipulate if a media-type were defined and which allowed a
transfer agent to bridge the data to a local content agent (most of the
IETF archives are in mbox format, and I'm aware that certain cellular
email systems allow for downloadable SMS/email logs via mbox, not to
mention the proliferation of web/local mail systems that use mbox files
which are often transferred back and forth). Local actions such as
importing or opening a local mbox file would also be easier to perform if
there was a media-type that the OS could use (I have many mbox files of
project-specific archives and would appreciate better OS/messaging
integration with these files [such as opening and searching folders the
same as I can with message/rfc822 file objects], but need an OS-level
media-type to get there). And so forth.

A definitive authoritative specification for all variations of the mbox
database format is explicitly not the objective, for several reasons. For
one thing, such a definition is outside the IETF's purview, the same as a
definition for Outlook or Eudora or other vendor/platform-centric database
formats would be. Second, if the IETF *did* try to write such a
definition, it would quickly settle on the fact that multipart/digest
already exists for such purposes, and that defining an mbox format would
be redundant. If development proceeded beyond that (unlikely at best),
then the spec would necessarily conform with other messaging definitions,
and would end up declaring that mbox files SHOULD/MUST be compliant with
RFC2822 messages at a minimum, which ignores the reality of eight-bit
content, long lines, untagged charsets, EOL variations, etc. Other folks
can start a bof/wg and go through this process if they want to, but
frankly I'm happy just registering the media-type (the purpose of the
process, and the objective of this I-D).

On top of all that, mbox parsers need to be aware of the limitations and
variations on the content, and this cannot be adequately handled with a
media-type definition.

Suggestions about parameters has come up before (John Klensin suggested it
to me a couple of months ago). Unfortunately, these kind of helper tags
attempt to define content rules rather than transfer rules, and therefore
represent a non-trivial layer violation. They are analogous to using a
"version=" tag for app/postscript and relying on that meta-information
instead of embedded clue data. Obviously, content agents should be aware
of the content formatting rules. [what happens when the helper meta-tags
are lost? should the content agent NOT look at the content to make a
determination? that's where the logic belongs in the first place, so
putting it into the transfer layer is not only irrelevant it is possibly
harmful.] This data should not be overflowed to the transfer agent except
where it affects the transfer process proper, which is not the case with
any of the suggested tags.

> however I don't think this should be up for PS, as the standards track
> is intended for protocols that can be subject to interoperability tests,
> and this is just a label. Informational seems more appropriate to me.

I'll go along with that. 2048 only requires IESG approval, but does not
require standards-track, and dropping to informational gets us there.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Fri, 13 Aug 2004 11:21:01 -0400
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

> 
> In <[email-masked]> Keith Moore <[email-masked]> writes:
> 
> >> Both these practices are already widespread, and they are not going to 
> >> stop just because we say so.
> 
> >that's not a good reason for standardizing them.  and they don't meet 
> >the criteria in 2026.
> 
> But I don't want to standardize them. All I want to say is that "if" you
> want to use a bogus address "for whatever reason", then please at least do
> it in such and such a way.
> 
> You can still say that, even if you deprecate the use of bogus addresses
> at the same time. Because users are going to do it whether you deprecate
> it or not (there is no way to stop them). King Canute has already been
> mentioned in this thread.

sometimes I think we need a separate label for documents -
"least harmful current practice".  but we don't have one yet.

Date: Fri, 13 Aug 2004 11:19:05 -0400
From: Keith Moore <[email-masked]>
To: Philip Hazel <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: The Bcc Issue
Sender: [email-masked]

> The issue of who handles Bcc: header lines has again been raised, and
> I am seeking opinions as widely as I can. The requirement is
> straightforward: non-Bcc recipients of a message should not see the
> addresses of any Bcc recipients, that is, their copies of the message
> should not contain Bcc: header lines.
> 
> The dispute is over who achieves this end by removing Bcc: lines when
> they should not be present. Is it the MUA or the MTA? 

It's the MUA.  MTAs should not be altering message headers other than
to add Received fields and a Return-Path field on delivery.  

I believe the confusion results from the common practice of asking 
"sendmail -t" (be it the original sendmail program or some clone) to 
perform some of the MUA's functions - in particular, generating 
envelope recipient lists from the to/cc/bcc message header fields.
In this case "sendmail" is still performing an MUA function, even
though we think of it as an MTA.  (it might be the case that sendmail 
is older than the MHS model and the formal MTA/MUA split)

So when an MUA asks a "sendmail -t" interface to parse header fields
generate a recipient list, and submit the message, that sendmail-like 
program should be removing Bcc fields from the copy of a message sent to
any recipient that isn't in the Bcc list.  It MAY remove the Bcc field
from all copies and still comply with the specifications, but this is
Bad.  

OTOH, when an MUA uses SMTP (or the SUBMISSION protocol) to submit a
message for delivery, it's the MUA's responsibility to generate multiple
envelopes for any message containing Bcc's - one envelope for the Bcc
recipients (which may include the Bcc header field) and one envelope
for the non-Bcc'ed recipients (which MUST NOT include the Bcc field).
The SMTP or SUBMISSION server should not be removing or interpreting
the Bcc fields in this case.

The MUA is never required to include a Bcc field in the messages sent
to Bcc recipients - but it's the Right Thing to do.

-Keith

Date: Fri, 13 Aug 2004 11:04:01 -0400
From: Keith Moore <[email-masked]>
To: [email-masked] (Kai Henningsen)
Cc: [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

> 
> [email-masked] (Keith Moore)  wrote on 12.08.04 in <[email-masked]>:
> 
> > > This proposal needs to be sent back for further consideration, either
> > > by a
> > > WG, or at least by some mailing list of those with knowledge of the
> > > format.
> > >
> > > It has just missed too many opportunities, including the opportunity to
> > > document the mbox format once and for all (and warts and all).
> >
> > uh, no.  this is an application for a media-type.  it's not trying to
> > define the mbox format, it's just trying to define a label for the mbox
> > format that can be used in contexts where MIME media-type labels are
> > required.  the intent is clearly that this label should be usable with
> > the variety of mbox files that exist, not with one specific variant of
> > mbox file.
> 
> But to do that, it needs to specify which variant of mbox the data is, and  
> to do that, there needs to be at least a description of the various  
> variants.

there are lots of media types for which there isn't a description of all of
the variants available.  we find it useful to label things even without 
those descriptions.

> From looking at the formail man page (just something quickly found on my  
> machine), I see mention of a variant using Content-Length:; a "traditional  
> Berkeley mbox format"; and a "BABYL rmail format" (*is* that mbox?).

no, BABYL format is not the same as mbox format.

> > trying to write a definitive specification for the mbox format is not
> > the same thing as trying to define a label for that format.  nor is it
> > the same thing as trying to document existing practice.  many authors
> > and groups get these confused, with frequently unfortunate results.
> 
> However (again), it *does* need to provide a way to identify the variants  
> in actual use. If it doesn't tell me which variant of mbox it is, it might  
> as well just say "application/octet-stream".

lots of tools seem to be able to make use of mbox files without knowing
precisely which variant they're reading.

Keith

To: [email-masked]
Xref: clerew local.mime:3254
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Fri, 13 Aug 2004 09:45:04 GMT
Lines: 32
Sender: [email-masked]

In <[email-masked]> "Charles Lindsey" <[email-masked]> writes:

>But none of these is of any use to the man who wants to include a mailbox
>of some sort (to distinguish him from all the other Joe Does), or who even
>want to hide his real mail address inside the bogus one in a form that a
>human can easily disentangle.

>Both these practices are already widespread, and they are not going to
>stop just because we say so.

And to see why they are so widespread, look at the following FAQ which is
published regularly on news.admin.net-abuse.email and other places.

Archive-name: net-abuse-faq/munging-address
Posting-frequency: weekly
Last-modified: August 8, 1999
Version: 2.01
URL: http://members.aol.com/emailfaq/mungfaq.html
FTP: ftp://members.aol.com/emailfaq/mungfaq.txt
Copyright: (c) 1996-1999 WD Baseley
Maintainer: [email-masked] (WD Baseley)

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3253
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Fri, 13 Aug 2004 08:48:31 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> Both these practices are already widespread, and they are not going to 
>> stop just because we say so.

>that's not a good reason for standardizing them.  and they don't meet 
>the criteria in 2026.

But I don't want to standardize them. All I want to say is that "if" you
want to use a bogus address "for whatever reason", then please at least do
it in such and such a way.

You can still say that, even if you deprecate the use of bogus addresses
at the same time. Because users are going to do it whether you deprecate
it or not (there is no way to stop them). King Canute has already been
mentioned in this thread.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3252
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Fri, 13 Aug 2004 08:39:40 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> [email-masked] (Kai Henningsen) writes:

>But to do that, it needs to specify which variant of mbox the data is, and  
>to do that, there needs to be at least a description of the various  
>variants.

Exactly. There is currently no one place where all the variants are
documented. The URL provided in the draft may be the best available, but
URLs are ephemeral and should not be relied on for normative purposes.
There is nothing inherently wrong with defining the full format of a new
media type in the document that introduces that type (though it is not
necessary if some document elsewhere can be referenced).

So here we have an opportunity to document it, which would have the added
advantage of preventing yet more variations in the future.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 13 Aug 2004 11:24:09 +0100 (BST)
From: Philip Hazel <[email-masked]>
To: [email-masked]
Subject: The Bcc Issue
Sender: [email-masked]

The Bcc Issue: posted to the exim-users, exim-dev, and ietf-822 mailing lists
-----------------------------------------------------------------------------

The issue of who handles Bcc: header lines has again been raised, and I
am seeking opinions as widely as I can. The requirement is
straightforward: non-Bcc recipients of a message should not see the
addresses of any Bcc recipients, that is, their copies of the message
should not contain Bcc: header lines.

The dispute is over who achieves this end by removing Bcc: lines when
they should not be present. Is it the MUA or the MTA? This is what I
know about current behaviour:

1. Exim (which I maintain) removes Bcc: lines if, and only if, it is
    called with the -t option. In that case, it is constructing an
    envelope from the header data, and IMHO in doing so it is fulfilling
    an MUA function. In all other cases it leaves Bcc: lines alone.

2. It has been reported to me that OpenWave InterMail, MS SMTP, MS
    Exchange, and qmail behave in the same way as Exim for incoming SMTP
    messages (i.e. they leave Bcc: lines alone).

3. I have also been told that Sendmail, at least in some configurations,
    always removes any Bcc: lines from any message that it handles. The
    same is apparently true of Postfix and Lotus Notes 5.

4. Among the MUAs, I think that Pine never sends Bcc: lines, but Mutt by
    default does, assuming that the MTA will deal with them (Mutt does
    not use the -t option).

The (re-)discussion of this issue arose because the combination of
(default configured) Mutt and Exim does not behave the way people
expect. Mutt sends Bcc: lines, and Exim lets them through to all
recipients.

The RFC permits several different behaviours in connection with Bcc
recipients. For copies of the message that are sent to Bcc recipients,
the Bcc: line may be absent, or contain just the address of the single
recipient, or contain the addresses of all the Bcc recipients. For
copies sent to non-Bcc recipients, there should be no Bcc: header line.

If the MUA handles this, it is possible for it to allow any of the
alternative behaviours, as the sender of the message wishes. An MTA has
no means of distinguishing what the sender wanted. If it is to handle
the Bcc: header at all, the only choice is always to remove it. Thus, an
MTA that always removes Bcc: header lines will frustrate the wishes of
someone who wants them included in copies to Bcc recipients. (I found
one posting on the www that complained of exactly this problem.)

Of course, it is possible for MTAs to have configuration options to
change their behaviour. (It is easy to configure Exim always to remove
Bcc: lines, for instance.) However, only clueful sysadmins who are aware
of the issue will investigate such options. Thus, it is important for
the default behaviour to be "correct" in the sense that it is what the
RFCs and the community experts generally agree on.

So .... please vent your opinions and I will try to pull together a
summary in due course.

I am posting this separately to the three list mentioned above because
the Exim lists are subscriber-posting only, and a cross-post might cause
problems with replies.

Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Thu, 12 Aug 2004 20:10:13 -0400
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: "Eric A. Hall" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Reply-To: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

Eric A. Hall wrote:

> Do you have a specific URL to a specific man page that you think would be
> appropriate and authoritative?
> 
> I spent a while looking (see also http://makeashorterlink.com/?X61D16909
> for example), and couldn't find anything that seemed to be authoritative
> enough to reference.

The docs/formats.txt file in the UW IMAP distribution
ftp://ftp.cac.washington.edu/mail/imap.tar.Z
describes several variants, some of which are not addressed
by the Bernstein document.  It also discusses a number of
interoperability issues not covered in the captioned draft.

As noted by Mark Crispin in the messages at the URL which you
have provided, different software can have somewhat different
interpretations of separator lines.  The only hope of having
any semblance of interoperability in exchanging such mbox
files between systems is to include an adequate specification
of the *specific* separator format used in the *particular*
file being transported (and I have suggested a regular
expression as a possible means of communicating that specification
via a Content-Type field parameter).

And I agree with Kai Henningsen's statement about use of
application/octet-stream; if there is not provision for
sufficient information (e.g. via parameters) to account for
the interoperability issues, then there's no point in having
a separate application subtype for it -- just use octet-stream
and let the communicating parties thrash out the details
out-of-band.  Or better yet (as briefly noted in the draft)
just use multipart/digest with local-system translations at
each end.

There are some additional issues that should be considered
(and either resolved or documented as known technical
omissions):
1. It is possible that processing by some transport (especially
   gateway) software may alter content which is not protected
   by transport encoding (and quoted-printable encoding might
   be insufficient).  That includes adding or removing trailing
   whitespace, modification of lines beginning with "From ", etc.
2. It is at best unclear how various software which uses one of
   the mbox format variants handles some 8bit and binary content,
   and indeed even 7bit content in MIME messages.  In particular,
   any attempt to convert line endings to/from the on-the-wire
   RFC [2]822 CRLF ending is likely to result in problems. For
   an example of binary content, see
   http://users.erols.com/blilly/mparse/tm35
   Note that the issue is not limited to binary content; "line
   ending" only has meaning in text media (not application, audio,
   image, video, model) and attempts at transforming non-text
   content (not merely text line endings, but e.g. content that
   happens to have <CR><LF>From<SP> in application, audio, etc.
   media) may irreversibly corrupt the content.

My opinion is that none of the mbox variants can handle the above
issues adequately.  Multipart/digest avoids the message separator
problem by using separators which are required not to be present
within the body of the message text in question.  Rigid formats
such as used by mbox variants are guaranteed to conflict with
some content at some point.  Some of the mbox variants appear
to have resulted from attempts to deal with various aspects of
the above and similar issues; the local storage methods that
appear to have the best chance of avoiding all of the problems
are those that use message-per-file storage in RFC [2]822
format (i.e. no attempt to convert line endings or otherwise
fold, spindle or mutilate message content) -- of course those
aren't "mbox" formats.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

Date: Thu, 12 Aug 2004 20:10:13 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: "Eric A. Hall" <[email-masked]>
CC: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

Eric A. Hall wrote:

> Do you have a specific URL to a specific man page that you think would be
> appropriate and authoritative?
> 
> I spent a while looking (see also http://makeashorterlink.com/?X61D16909
> for example), and couldn't find anything that seemed to be authoritative
> enough to reference.

The docs/formats.txt file in the UW IMAP distribution
ftp://ftp.cac.washington.edu/mail/imap.tar.Z
describes several variants, some of which are not addressed
by the Bernstein document.  It also discusses a number of
interoperability issues not covered in the captioned draft.

As noted by Mark Crispin in the messages at the URL which you
have provided, different software can have somewhat different
interpretations of separator lines.  The only hope of having
any semblance of interoperability in exchanging such mbox
files between systems is to include an adequate specification
of the *specific* separator format used in the *particular*
file being transported (and I have suggested a regular
expression as a possible means of communicating that specification
via a Content-Type field parameter).

And I agree with Kai Henningsen's statement about use of
application/octet-stream; if there is not provision for
sufficient information (e.g. via parameters) to account for
the interoperability issues, then there's no point in having
a separate application subtype for it -- just use octet-stream
and let the communicating parties thrash out the details
out-of-band.  Or better yet (as briefly noted in the draft)
just use multipart/digest with local-system translations at
each end.

There are some additional issues that should be considered
(and either resolved or documented as known technical
omissions):
1. It is possible that processing by some transport (especially
   gateway) software may alter content which is not protected
   by transport encoding (and quoted-printable encoding might
   be insufficient).  That includes adding or removing trailing
   whitespace, modification of lines beginning with "From ", etc.
2. It is at best unclear how various software which uses one of
   the mbox format variants handles some 8bit and binary content,
   and indeed even 7bit content in MIME messages.  In particular,
   any attempt to convert line endings to/from the on-the-wire
   RFC [2]822 CRLF ending is likely to result in problems. For
   an example of binary content, see
   http://users.erols.com/blilly/mparse/tm35
   Note that the issue is not limited to binary content; "line
   ending" only has meaning in text media (not application, audio,
   image, video, model) and attempts at transforming non-text
   content (not merely text line endings, but e.g. content that
   happens to have <CR><LF>From<SP> in application, audio, etc.
   media) may irreversibly corrupt the content.

My opinion is that none of the mbox variants can handle the above
issues adequately.  Multipart/digest avoids the message separator
problem by using separators which are required not to be present
within the body of the message text in question.  Rigid formats
such as used by mbox variants are guaranteed to conflict with
some content at some point.  Some of the mbox variants appear
to have resulted from attempts to deal with various aspects of
the above and similar issues; the local storage methods that
appear to have the best chance of avoiding all of the problems
are those that use message-per-file storage in RFC [2]822
format (i.e. no attempt to convert line endings or otherwise
fold, spindle or mutilate message content) -- of course those
aren't "mbox" formats.

Date: Thu, 12 Aug 2004 18:31:13 -0400
From: "Eric A. Hall" <[email-masked]>
To: Tony Hansen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

On 8/12/2004 5:18 PM, Tony Hansen wrote:

> The information about the mbox format being anecdotally defined is 
> incorrect. The mbox format has traditionally been documented in the 
> binmail(1) or mail.local(8) man pages (BSD UNIX derivatives) or mail(1)
>  man page (UNIX System 3/5/III/V derivatives).

I checked each of those and none of them seem to adequately describe the
message or database format.

> The most complete description of an mbox format can be seen in the man
> page from any UNIX System Vr4 derived system, such as Solaris.
>
> The description found in http://qmail.org./man/man5/mbox.html, referred
> to at the end of Appendix A, does cover these variations fairly 
> succinctly. However Appendix A doesn't cover the variations properly.

Do you have a specific URL to a specific man page that you think would be
appropriate and authoritative?

I spent a while looking (see also http://makeashorterlink.com/?X61D16909
for example), and couldn't find anything that seemed to be authoritative
enough to reference.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 12 Aug 2004 17:18:19 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

The information about the mbox format being anecdotally defined is 
incorrect. The mbox format has traditionally been documented in the 
binmail(1) or mail.local(8) man pages (BSD UNIX derivatives) or mail(1) 
man page (UNIX System 3/5/III/V derivatives). There have been several 
variants of the mbox format in use by those different systems. The most 
complete description of an mbox format can be seen in the man page from 
any UNIX System Vr4 derived system, such as Solaris.

The description found in http://qmail.org./man/man5/mbox.html, referred 
to at the end of Appendix A, does cover these variations fairly 
succinctly. However Appendix A doesn't cover the variations properly.

	Tony Hansen
	[email-masked]

The IESG wrote:

> The IESG has received a request from an individual submitter to
> consider the following document:
>
>- 'The APPLICATION/MBOX Media-Type '
>   <draft-hall-mime-app-mbox-02.txt> as a Proposed Standard
>
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action. Please send any comments to the 
> [email-masked] or [email-masked] mailing lists by 2004-09-06.
>
> The file can be obtained via
> http://www.ietf.org/internet-drafts/draft-hall-mime-app-mbox-02.txt

Date: 12 Aug 2004 19:31:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 12.08.04 in <[email-masked]>:

> > This proposal needs to be sent back for further consideration, either
> > by a
> > WG, or at least by some mailing list of those with knowledge of the
> > format.
> >
> > It has just missed too many opportunities, including the opportunity to
> > document the mbox format once and for all (and warts and all).
>
> uh, no.  this is an application for a media-type.  it's not trying to
> define the mbox format, it's just trying to define a label for the mbox
> format that can be used in contexts where MIME media-type labels are
> required.  the intent is clearly that this label should be usable with
> the variety of mbox files that exist, not with one specific variant of
> mbox file.

But to do that, it needs to specify which variant of mbox the data is, and  
to do that, there needs to be at least a description of the various  
variants.

>From looking at the formail man page (just something quickly found on my  
machine), I see mention of a variant using Content-Length:; a "traditional  
Berkeley mbox format"; and a "BABYL rmail format" (*is* that mbox?).

> trying to write a definitive specification for the mbox format is not
> the same thing as trying to define a label for that format.  nor is it
> the same thing as trying to document existing practice.  many authors
> and groups get these confused, with frequently unfortunate results.

However (again), it *does* need to provide a way to identify the variants  
in actual use. If it doesn't tell me which variant of mbox it is, it might  
as well just say "application/octet-stream".

MfG Kai

Date: Thu, 12 Aug 2004 16:45:54 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Keith Moore wrote:

> I think you're missing my point.  There have been several informal
> proposals mentioned in the past few days.  I don't know why yours is "on
> the table" and the others are not.

In my original message on 15 Jul 2004 I described the issue, the
background, similar changes made in the past, analyzed the impact
of making the From field optional on normal replies as well as
MDNs and DSNs. I mentioned the related Resent-From field and
brought up possible considerations w.r.t. the Sender field. I
pointed out that some wording would need to be changed. In
subsequent discussions, issues related to core protocols (e.g.
DNS) have been discussed.

If there have been other proposals with a similar degree of
analysis and presentation (e.g. analysis of Resent-From, MDNs,
and DSNs, which I haven't seen mentioned since my 15 Jul message),
with specific implications for RFC 2822's successor, then I haven't
seen them (perhaps you are referring to "half-baked idea[s]",
which fall short of a specific proposal considering the relevant
implications).

Date: Thu, 12 Aug 2004 14:30:15 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

> Keith Moore wrote:
> > 
> > well, this not being a group with a charter or a chair, I'm not sure
> > who determines what is a proposal (vs. a half-baked idea), and what
> > is or is not on the table.  
> 
> OK, it's not a *formal* proposal in the context of a WG, but it is a
> proposal nevertheless (and if you think it's a half-baked idea, you're
> welcome to present that opinion with justification). 

I think you're missing my point.  There have been several informal
proposals mentioned in the past few days.  I don't know why yours is "on
the table" and the others are not.

Keith

Date: Thu, 12 Aug 2004 14:05:38 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Keith Moore wrote:
> 
> well, this not being a group with a charter or a chair, I'm not sure who
> determines what is a proposal (vs. a half-baked idea), and what is or is
> not on the table.  

OK, it's not a *formal* proposal in the context of a WG, but it is a
proposal nevertheless (and if you think it's a half-baked idea, you're
welcome to present that opinion with justification).  A successor to
RFC 2822 is expected and in progress, and this seems to be the
appropriate forum to publicly discuss issues that might be worth
considering for that successor.

By the way, another point in favor of making From optional is that
it provides message originators a way to avoid being bombarded by
poorly designed software which is inappropriately configured by
ill-informed users, e.g.:

--------------------
[...]
From: [email-masked]
To: [email-masked]
Subject: =?ISO-8859-1?Q?Fr=E5nvaro=2C_autosvar=3ALast_Call=3A_=27The_AP?=
	=?ISO-8859-1?Q?PLICATION/MBOX_Media-Type=27_to_Proposed_Standard?=
Date: Thu, 12 Aug 2004 14:37:59 +0200

=C4r p=E5 semster till 30/7 - d=E4refter p=E5 IETF-m=F6te till 6/8 och =
sedan semseter
igen till 20/8.

=C5ter/will be back 23 of August.

Joakim Str=E5lmark
---------------------

in response to a message where none of sender envelope, Sender, or
Reply-To pointed to the address to which the above "vacation"-like
message (in response to a mailing list message!) was sent.  The
message header From field should *never* be used for such messages
(moreover, such messages should *never* be sent in response to
mailing list messages received by a subscribed list recipient),
and omitting a From field permits message authors to avoid being
pestered by such nonsense.

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Thu, 12 Aug 2004 08:49:16 -0400
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

>> my gut sense is that omitting From: will cause fewer problems than 
>> either From: <> or From: <""@[]>

> But none of these is of any use to the man who wants to include a 
> mailbox
> of some sort (to distinguish him from all the other Joe Does),

there's no good reason to include a bogus mailbox.  if the sender wants 
to be distinguished from other John Does, define a different way to do 
that.

> or who even want to hide his real mail address inside the bogus one in 
> a form that a human can easily disentangle.

again, if the human recipient has to "manually" disentangle the 
address, there's no reason it has to be in the From field.

> Both these practices are already widespread, and they are not going to 
> stop just because we say so.

that's not a good reason for standardizing them.  and they don't meet 
the criteria in 2026.

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Thu, 12 Aug 2004 08:45:24 -0400
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

>
> This proposal needs to be sent back for further consideration, either 
> by a
> WG, or at least by some mailing list of those with knowledge of the
> format.
>
> It has just missed too many opportunities, including the opportunity to
> document the mbox format once and for all (and warts and all).

uh, no.  this is an application for a media-type.  it's not trying to 
define the mbox format, it's just trying to define a label for the mbox 
format that can be used in contexts where MIME media-type labels are 
required.  the intent is clearly that this label should be usable with 
the variety of mbox files that exist, not with one specific variant of 
mbox file.

trying to write a definitive specification for the mbox format is not 
the same thing as trying to define a label for that format.  nor is it 
the same thing as trying to document existing practice.  many authors 
and groups get these confused, with frequently unfortunate results.

however I don't think this should be up for PS, as the standards track 
is intended for protocols that can be subject to interoperability 
tests, and this is just a label.  Informational seems more appropriate 
to me.

Cc: Keith Moore <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Thu, 12 Aug 2004 08:40:34 -0400
To: [email-masked]
Sender: [email-masked]

>
> Yes, that's why the first is the proposal on the table.
>

well, this not being a group with a charter or a chair, I'm not sure 
who determines what is a proposal (vs. a half-baked idea), and what is 
or is not on the table. 

Date: Wed, 11 Aug 2004 20:06:46 -0400
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: [email-masked], [email-masked]
Cc: 822 <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Reply-To: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

The IESG wrote:
> 
> The IESG has received a request from an individual submitter to consider
> the
> following document:
> 
> - 'The APPLICATION/MBOX Media-Type '
>    <draft-hall-mime-app-mbox-02.txt> as a Proposed Standard
> 
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action.  Please send any comments to the
> [email-masked] or [email-masked] mailing lists by 2004-09-06.
> 
> The file can be obtained via
> http://www.ietf.org/internet-drafts/draft-hall-mime-app-mbox-02.txt
> 
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ietf-announce

The draft mentions that "some form of external
     negotiation or prior agreement is often necessary in order to
     ensure that the contents of the database are accurately read by
     messaging systems on other hosts".

MIME provides for such mechanisms via the Content-Type field
parameters.  It would appear to be desirable to define at least
two parameter attributes for use with this media type, viz. one
defining the separator between messages (perhaps as a regular
expression), and another defining the line-ending sequence in use
(defaulting to CRLF).  Note that RFC 2231 provides mechanisms for
quoting and encoding parameter content, so that for example a lone
LF line-ending sequence could be encoded in the parameter value as
"%0A".

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

To: [email-masked]
Xref: clerew local.mime:3240
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Date: Thu, 12 Aug 2004 09:30:54 GMT
Lines: 39
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>The IESG wrote:
>> 
>> - 'The APPLICATION/MBOX Media-Type '
>>    <draft-hall-mime-app-mbox-02.txt> as a Proposed Standard

>The draft mentions that "some form of external
>     negotiation or prior agreement is often necessary in order to
>     ensure that the contents of the database are accurately read by
>     messaging systems on other hosts".

>MIME provides for such mechanisms via the Content-Type field
>parameters.  It would appear to be desirable to define at least
>two parameter attributes for use with this media type, viz. one
>defining the separator between messages (perhaps as a regular
>expression), and another defining the line-ending sequence in use
>(defaulting to CRLF).

Yes, there are many ways that such parameters could be used to indicate
the conventions used in the construction of that particular mbox file.

This proposal needs to be sent back for further consideration, either by a
WG, or at least by some mailing list of those with knowledge of the
format.

It has just missed too many opportunities, including the opportunity to
document the mbox format once and for all (and warts and all).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3241
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Thu, 12 Aug 2004 09:34:10 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>>    From: <""@[]>
>> is syntactically valid, though.

>yeech.

Agreed.

>my gut sense is that omitting From: will cause fewer problems than 
>either From: <> or From: <""@[]>

But none of these is of any use to the man who wants to include a mailbox
of some sort (to distinguish him from all the other Joe Does), or who even
want to hide his real mail address inside the bogus one in a form that a
human can easily disentangle.

Both these practices are already widespread, and they are not going to
stop just because we say so.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 11 Aug 2004 20:06:46 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: [email-masked], [email-masked]
CC: 822 <[email-masked]>
Subject: Re: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

The IESG wrote:
> 
> The IESG has received a request from an individual submitter to consider
> the
> following document:
> 
> - 'The APPLICATION/MBOX Media-Type '
>    <draft-hall-mime-app-mbox-02.txt> as a Proposed Standard
> 
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action.  Please send any comments to the
> [email-masked] or [email-masked] mailing lists by 2004-09-06.
> 
> The file can be obtained via
> http://www.ietf.org/internet-drafts/draft-hall-mime-app-mbox-02.txt
> 
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ietf-announce

The draft mentions that "some form of external
     negotiation or prior agreement is often necessary in order to
     ensure that the contents of the database are accurately read by
     messaging systems on other hosts".

MIME provides for such mechanisms via the Content-Type field
parameters.  It would appear to be desirable to define at least
two parameter attributes for use with this media type, viz. one
defining the separator between messages (perhaps as a regular
expression), and another defining the line-ending sequence in use
(defaulting to CRLF).  Note that RFC 2231 provides mechanisms for
quoting and encoding parameter content, so that for example a lone
LF line-ending sequence could be encoded in the parameter value as
"%0A".

Date: Wed, 11 Aug 2004 19:42:24 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Keith Moore wrote:

>>    From: <""@[]>
>> is syntactically valid, though.
> 
> 
> yeech.

Agreed.

> my gut sense is that omitting From: will cause fewer problems than
> either From: <> or From: <""@[]>

Yes, that's why the first is the proposal on the table.

Cc: Keith Moore <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Wed, 11 Aug 2004 08:33:30 -0400
To: [email-masked]
Sender: [email-masked]

> Keith Moore wrote:
>
>> Users who don't want to include a valid From address should just use 
>> <>.
>
> That would be a syntax error.

true.

>    From: <""@[]>
> is syntactically valid, though.

yeech.

my gut sense is that omitting From: will cause fewer problems than 
either From: <> or From: <""@[]>

Date: Tue, 10 Aug 2004 19:29:11 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
>>
>>>Well I have provided strong and substantive objections for a start.
> 
> 
>>Exactly what objection to making From optional do you think you have
>>provided?
> 
> 
> If you cannot be bothered to read the discussion to find the objections
> made to your proposal, then I have not the time to do your reading for
> you.

Charles, I have read your remarks; there is a great deal of blathering
about ".invalid", which is not the proposal at hand.  I have seen
no substantive objections to relaxing the requirement for a From
header field.  If you cannot provide a succinct summary of any such
objections when politely requested to do so, it can only be assumed
that you have none.

Date: Tue, 10 Aug 2004 19:24:41 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Keith Moore wrote:

> Users who don't want to include a valid From address should just use <>.

That would be a syntax error.
   From: <""@[]>
is syntactically valid, though.

Date: Tue, 10 Aug 2004 16:57:34 -0400
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

> But finally, he makes the very sensible point that what user agents really
> need to do is to bring up an instant Warning to the user "you appear to be
> about to send email to a non-existent TLD; do you want to continue?" If
> the user persists in spite of such a warning, then on his head be it.

I get the strong impression that you're trying to solve the wrong problem.
Users who don't want to include a valid From address should just use <>.
It doesn't need a new TLD, it doesn't impact the root servers, etc.

To: [email-masked]
Xref: clerew local.mime:3232
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Tue, 10 Aug 2004 11:43:28 GMT
Lines: 47
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> No, that is exactly what RFC 3696 does _not_ say. You have already 
>> presumed to understand what the authors of RFC 2606 intended to say,
>> and  been proved wrong. Do you want me to write to John Klensin and ask
>> him  what he really meant in RFC 3696?

>Knock yourself out. Go ahead and ask him if he thinks hard-coding
>recognition of ".invalid" is a good idea or not.

I have written to John Klensin, and he makes several interesting comments.

Firstly, he points out that IANA has less authority to reserve TLDs than
it had when RFC 2606 appeared, though he agrees that it is hardly
conceivable that ICANN would ever allow it to be taken (note that RFC 2606
has BCP status, which implies an IESG approval).

However, he also points out that use of .invalid should NOT cause any
serious problem for the root servers, since apparently the standards
require negative caching to be properly implemented.

As regards RFC 3696, it certainly does not forbid local checking, but
rather places an onus on those doing the checking to be careful that they
are acting upon up-to-date information; moreover affirmative exclusions
(such as provided by RFC 2606) provide much better evidence than mere
observation of what currently does and does not work.

But finally, he makes the very sensible point that what user agents really
need to do is to bring up an instant Warning to the user "you appear to be
about to send email to a non-existent TLD; do you want to continue?" If
the user persists in spite of such a warning, then on his head be it.

I would have no problem in modifying the USEFOR draft to take a line like
that.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:3233
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Tue, 10 Aug 2004 11:45:11 GMT
Lines: 23
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:
>>>
>> Well I have provided strong and substantive objections for a start.

>Exactly what objection to making From optional do you think you have
>provided?

If you cannot be bothered to read the discussion to find the objections
made to your proposal, then I have not the time to do your reading for
you.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 9 Aug 2004 09:57:00 -0700
To: [email-masked]
From: The IESG <[email-masked]>
Subject: Last Call: 'The APPLICATION/MBOX Media-Type' to Proposed Standard
Sender: [email-masked]

The IESG has received a request from an individual submitter to consider the
following document:

- 'The APPLICATION/MBOX Media-Type '
    <draft-hall-mime-app-mbox-02.txt> as a Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action.  Please send any comments to the
[email-masked] or [email-masked] mailing lists by 2004-09-06.

The file can be obtained via
http://www.ietf.org/internet-drafts/draft-hall-mime-app-mbox-02.txt

_______________________________________________
IETF-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf-announce

Date: Sat, 7 Aug 2004 11:10:56 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: RFC validation samples?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 06 2004, Keith Moore wrote:
> > There's a funny connection between these types of protocol elements,
> > spam and statistical learning. 
> 
> only in the short term.  eventually spammers learn to make their message
> headers look "just like" legitimate messages.
> 
> these days, spammers seem to learn more quickly, and upgrade their 
> software more often, than MUA and MTA writers.

This isn't necessarily a disadvantage. If a spammer message header
evolves to the correct format, statistical filtering decisions will
naturally gravitate towards the differences in the mail transport
path, I expect. 

Moreover, we shouldn't forget why spammer software sometimes use a
token such as "jAn" instead of the common form "Jan". To the transport
mechanism (ie according to the RFCs), both "Jan" and "jAn" are
identical, so there is no harm in using one or the other. But widely
deployed filters use a hash of the message header, or parts of the
message body, etc.  When hashing a header containing "Jan", you get a
different answer than if the header contains "jAn", allowing the spam
to slip by. Now obviously in this example it would be easy to
normalize occurrences of "Jan" before hashing, but this explains why
spammer software likes to add nonstandard elements in mail messages. 

If spammer messages look more "legitimate", they become vulnerable to 
more defenses. And of course, for a statistical filter which updates/learns
continuously from spam examples, whether messages start or stop using tricks
such as "jAn" is just another fact for adjusting the weights, there's no need
to change any algorithms at all.

-- 
Laird Breyer.

Date: Sat, 7 Aug 2004 10:40:04 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: RFC validation samples?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 06 2004, Bruce Lilly wrote:
> 
> "from", "by", "with", "via", "id", and "for" are protocol elements

<snip>

That's more information than I was expecting, but you've nicely
cleared up areas of discrepancy between RFCs, thanks.

My question was intended to address parsing the Microsoft non-RFC
format specifically, since it's likely to be needed for data mining
some Received: lines. The example you gave strongly suggests to me
whoever programmed that line used code resembling

fprintf(out, "Received: from %s by %s with %s; %s\r\n", ...);

> Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
>          Mon, 14 Jun 2004 16:13:25 -0700

Which is why I asked if Microsoft's format seems to consistently use "from"
"by" "with" as delimiters (because we can't rely on white space obviously).
I also wouldn't be surprised if the Microsoft date/time stamp is localized,
and shows up e.g. in German for servers installed in German locales (?).

> The illegal syntax is generated by Microsoft Windows 2000 (and
> probably its successors) and was reported to Microsoft years ago
> (shortly after Windows 2000 was introduced).  As of Windows 2000
> SP3 it was not fixed, so I suppose one could say that the bug
> responsible for the broken format is stable...

I'm not surprised. Now, I wouldn't expect anyone on this list to have a
Windows 2000 SMTP server on their premises ;-) but hypothetically if
someone had access to the binary or dll, it may be possible to run strings(1)
on it and look for the actual format line used, assuming that line is
really generated in such a simple way.

-- 
Laird Breyer.

Date: Fri, 06 Aug 2004 18:36:51 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822mailing list <[email-masked]>
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: ietf-822mailing list <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Charles Lindsey wrote:
> 
> On Thu, 05 Aug 2004 21:39:21 -0400, Bruce Lilly <[email-masked]>  wrote:
> 
>>
>> More than two weeks have passed with no substantive objection that I  can
>> see to the principle of making the From field optional in the 
>> successor to
>> RFC 2822.  If there are any such objections, please speak up soon.
>>
> Well I have provided strong and substantive objections for a start.

Exactly what objection to making From optional do you think you have
provided?

Date: Fri, 06 Aug 2004 23:23:07 +0100
From: "Charles Lindsey" <[email-masked]>
To: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

On Thu, 05 Aug 2004 21:36:03 -0400, Bruce Lilly <[email-masked]>  
wrote:

>
> Charles Lindsey wrote:
>
>> However, if you want to see what the actual authors of RFC 2606
>> intended  it to mean, you might care to look at
>> <http://purl.net/net/msgid/[email-masked]>,

>> which not only confirms its appropriateness within Netnews,
>
> There is also nothing there which says exactly what use is referred  
> to by
> "the kind of use you are talking about", or indeed any reference to  
> what
> message that might have been in response to.

He says:
    I don't see any problem in using .invalid in news messages to
    indicate an invalid address.

which seems clear enough to me as to "what use is referred to".

-- 
Charles H. Lindsey ---------At  
Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 06 Aug 2004 23:23:08 +0100
From: "Charles Lindsey" <[email-masked]>
To: "ietf-822mailing list" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

On Thu, 05 Aug 2004 21:39:21 -0400, Bruce Lilly <[email-masked]>  
wrote:

>
> More than two weeks have passed with no substantive objection that I  
> can
> see to the principle of making the From field optional in the  
> successor to
> RFC 2822.  If there are any such objections, please speak up soon.
>
Well I have provided strong and substantive objections for a start.

Ah no! I forgot! They cannot possibly be "substantive" because you are  
the sole determiner of that.

And I note that there have been no voices in support of the idea. You  
need more than lack of objections, substantive or otherwise, to change  
something that is already on the standards-track.

-- 
Charles H. Lindsey ---------At  
Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 06 Aug 2004 15:52:19 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

Laird Breyer wrote:
> On Aug 05 2004, Bruce Lilly wrote:

>>Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
>>         Mon, 14 Jun 2004 16:13:25 -0700
> 
> 
> Is this format known to be fairly stable (ie using from,by,with etc. as
> delimiters?) If so, it could form the basis for an extra "Anti-RFC" format ;)
> I'd better get to writing more code...

"from", "by", "with", "via", "id", and "for" are protocol elements
as specified in RFCs 821, 822, and 2821.  The first two are used with
domain names. The keywords used with "with" (protocols) and "via"
(links) used to be specified in the Assigned Numbers RFC (most recently
RFC 1700) before it was superseded by an online database
(see RFC 3232). However, RFC 2821 specifies "TCP" as a link value,
but is has not yet made its way into the database (after 3+ years).
On the other hand, new protocol values have recently been added based
on RFC 3848.  "id" was an area of discrepancy between RFCs 821 (a string)
and 822 (a msg-id); while RFC 1123 noted the discrepancy, it only added
to the confusion. RFCs 2821 and 2822 attempted to correct that, but there
is still a minor discrepancy (RFC 2821 "String" isn't provided for in
2822).  "for" is another area where there is a discrepancy; RFC 2821 permits
a mix of addr-specs and angle-addrs, whereas 2822 does not.  "from" and
"by" are required by RFCs 821 and 2821, as is the semicolon and
time stamp (the line was initially referred to as a "time stamp line").
RFC 2822 provides very liberal grammar for parsing, such that
   Received:
(i.e. nothing beyond the field name and colon) is considered
RFC 2822-legal -- that's a time stamp line with no time stamp.

Among the problems with the example above generated by faulty software
are:
1. "mail" is not a domain name
2. there is no "pickup" keyword
3. "Microsoft" is not a valid protocol
4. "SMTPSVC" is not a keyword
5. there is no value associated with "SMTPSVC", so even the
   very liberal RFC 2822 name-val pair syntax is violated by
   the example
6. The mandatory space before the semicolon is missing
7. RFC 821 does not permit day-of-week in the time stamp

The illegal syntax is generated by Microsoft Windows 2000 (and
probably its successors) and was reported to Microsoft years ago
(shortly after Windows 2000 was introduced).  As of Windows 2000
SP3 it was not fixed, so I suppose one could say that the bug
responsible for the broken format is stable...

Date: Fri, 6 Aug 2004 10:17:05 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

> There's a funny connection between these types of protocol elements,
> spam and statistical learning. 

only in the short term.  eventually spammers learn to make their message
headers look "just like" legitimate messages.

these days, spammers seem to learn more quickly, and upgrade their 
software more often, than MUA and MTA writers.

Date: Fri, 6 Aug 2004 14:01:54 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: RFC validation samples?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 05 2004, Bruce Lilly wrote:
> Laird Breyer wrote:
> 
> > Is your code/library thread safe?
> 
> It is intentionally and necessarily reentrant.
> 
> > The incompatibilities are actually a possible "feature".  For
> > instance, would it be reasonable to expect that a message, none of
> > whose Received: lines follow 2821 correctly, hasn't passed through an
> > internet SMTP server? I.e. I'm assuming that the majority of internet
> > SMTP servers are RFC 821/2821 compliant and follow the grammars therein
> > correctly.
> 
> Bad assumption. There exist servers using software from a certain
> vendor which generate Received field content which is not legal per
> any of RFCs 821, 822, 2821, or 2822; for example:
> 
> Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
>          Mon, 14 Jun 2004 16:13:25 -0700

Is this format known to be fairly stable (ie using from,by,with etc. as
delimiters?) If so, it could form the basis for an extra "Anti-RFC" format ;)
I'd better get to writing more code...

-- 
Laird Breyer.

Date: Fri, 6 Aug 2004 13:49:41 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: RFC validation samples?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 05 2004, Bruce Lilly wrote:
> Laird Breyer wrote:
> 
> > This kind of datamining has subtleties, because some types of data are
> > misleading. For example, the date-time element in some headers is a
> > bad statistical indicator. If you learn messages which contain many
> > "Jul" tokens, then come August the statistical model will start to
> > wonder where the "Jul" tokens went.
> 
> Beware that structured header fields aren't simply "text" (see RFCs 2277
> and 1958 for "text" vs. "names"); protocol elements such as month names
> are typically registered or otherwise well-defined (Jan, Feb, ... Nov,
> Dec) are valid month "names", "Bob" is not) and are almost always case
> insensitive (jAn, jAN, jaN, JaN, and jan are valid month "names").

There's a funny connection between these types of protocol elements,
spam and statistical learning. 

The statistical learner picks up tokens, whether
true "text" or a defined protocol element, and associates observed
frequency counts. So month names such as Jan, Feb etc. become
important (for the statistical model) solely because of their observed
frequency. 

Coincidentally, the relative frequencies of such tokens over several
emails are normally high because they are machine generated during
transport, and transport often passes through the same small
collection of MTAs close to the destination.

On the other hand, there is much more variety in such tokens at the
spammer end, and some spam sending software may well include "jAn"
instead of "Jan" in a header, for some unknown reason. When this
happens frequently, the statistical learner may associate "jAn" with
spam from this spammer's software, even though the RFC protocol itself does not
distinguish "Jan" from "jAn". 

Sometimes, this case sensitive distinction is a win, meaning it helps
the statistical classifier make fewer mistakes, and sometimes it isn't
- it's difficult to predict. In the example of the date-time protocol element,
my experience is that including an email's date stamps in the learning data is
on the whole counterproductive. However, for tracing message originators and path forgeries, date-time discrepancies are strong indicators of spam.

> 
> > The second type of goal is to see if there are enough indicators in
> > the headers to detect forgery and/or trace the message origin. For
> > example, there are some simple rules which can be used, such as: the
> > topmost Received: line is always authentic. I'd like to explore if
> > there are others, e.g. can we analyse whether a message passed through
> > the internet (ie at least one compliant SMTP server outside the local
> > network), Maybe this can't be detected, but perhaps its negation can
> > (ie this message never left the LAN).
> 
> In theory, one could examine Received header fields to establish a
> consistent set of time stamps and "from" vs. "by" domains. In
> practice the Received field is so frequently malformed as to make such
> analysis impractical.

I believe statistics can somewhat help with this, or at least it's worth
giving it a shot ;-) 

When analysing a large collection of mail with the 
same destination, some Received: lines and their idiosyncracies are more
frequent than others. Suppose you have a test which shows a header was forged,
but this test is only 60% accurate. You may have several such tests, and
statistics can combine them into a preponderance of evidence scheme, 
kind of like SpamAssassin already does. 

But SpamAssassin is network wide, its rules are chosen to work for
many users simultaneously, and because they are published spammers
study them and find ways to bypass them. I believe that rules based
directly on published RFCs may be more universal and less "noisy",
and of course a statistical learner can combine them in ways specific
to the destination, which makes it much harder for spammers to get through.

> 
> > I started out parsing only 2822 messages, but quickly realized that
> > 2821 is just slightly different, and even so some sample header lines
> > I have don't appear to comply with either (e.g. in 2821, domains must
> > contain at least one period, so localhost is not allowed but
> > localhost.localdomain is).
> 
> RFC 2821 requires fully-qualified domain names, RFC 2822 does not.
> In practice an RFC 2822-compliant message might be sent with partially
> qualified domain names to an RFC 2476 submission server which qualifies
> those domain names before sending via SMTP.

Such a submission server would show up in the trace fields, no? 
> 
> > I believe the easiest solution is to parse
> > each line four times or more, strictly according to 2822,2821,822,821
> > and special variations, as the code can refer to existing documents
> > rather than a single complex combination of these four documents which
> > incorporates many subtleties and variations.
> 
> If performance and maintainabilty are not considerations, that might be
> practical.  Liberal parsing with RFC-specific detection of issues is
> an alternative approach.

Performance is not a strong worry, at this stage. The code I have in
dbacl is pretty fast, even if I say so myself (it can handle about
100-200 typical mails per second depending on options, on a 500MHz
Pentium 3 Debian box, not counting process startup costs, and based on
classifying thousands of mails sequentially for cross-validation
simulations), and even so it probably spends half of its time parsing
mail bodies.

-- 
Laird Breyer.

Date: Thu, 05 Aug 2004 23:05:17 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

Laird Breyer wrote:

> Is your code/library thread safe?

It is intentionally and necessarily reentrant.

> The incompatibilities are actually a possible "feature".  For
> instance, would it be reasonable to expect that a message, none of
> whose Received: lines follow 2821 correctly, hasn't passed through an
> internet SMTP server? I.e. I'm assuming that the majority of internet
> SMTP servers are RFC 821/2821 compliant and follow the grammars therein
> correctly.

Bad assumption. There exist servers using software from a certain
vendor which generate Received field content which is not legal per
any of RFCs 821, 822, 2821, or 2822; for example:

Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
         Mon, 14 Jun 2004 16:13:25 -0700

Date: Thu, 05 Aug 2004 22:37:53 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

Laird Breyer wrote:

> This kind of datamining has subtleties, because some types of data are
> misleading. For example, the date-time element in some headers is a
> bad statistical indicator. If you learn messages which contain many
> "Jul" tokens, then come August the statistical model will start to
> wonder where the "Jul" tokens went.

Beware that structured header fields aren't simply "text" (see RFCs 2277
and 1958 for "text" vs. "names"); protocol elements such as month names
are typically registered or otherwise well-defined (Jan, Feb, ... Nov,
Dec) are valid month "names", "Bob" is not) and are almost always case
insensitive (jAn, jAN, jaN, JaN, and jan are valid month "names").

> The second type of goal is to see if there are enough indicators in
> the headers to detect forgery and/or trace the message origin. For
> example, there are some simple rules which can be used, such as: the
> topmost Received: line is always authentic. I'd like to explore if
> there are others, e.g. can we analyse whether a message passed through
> the internet (ie at least one compliant SMTP server outside the local
> network), Maybe this can't be detected, but perhaps its negation can
> (ie this message never left the LAN).

In theory, one could examine Received header fields to establish a
consistent set of time stamps and "from" vs. "by" domains. In
practice the Received field is so frequently malformed as to make such
analysis impractical.

> I started out parsing only 2822 messages, but quickly realized that
> 2821 is just slightly different, and even so some sample header lines
> I have don't appear to comply with either (e.g. in 2821, domains must
> contain at least one period, so localhost is not allowed but
> localhost.localdomain is).

RFC 2821 requires fully-qualified domain names, RFC 2822 does not.
In practice an RFC 2822-compliant message might be sent with partially
qualified domain names to an RFC 2476 submission server which qualifies
those domain names before sending via SMTP.

> I believe the easiest solution is to parse
> each line four times or more, strictly according to 2822,2821,822,821
> and special variations, as the code can refer to existing documents
> rather than a single complex combination of these four documents which
> incorporates many subtleties and variations.

If performance and maintainabilty are not considerations, that might be
practical.  Liberal parsing with RFC-specific detection of issues is
an alternative approach.

Date: Thu, 05 Aug 2004 22:06:15 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822imcorg list <[email-masked]>
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: ietf-822imcorg list <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Charles Lindsey wrote:
> 

> Moreover, the use of the TLD ".invalid" in munged addresses is already 
> fairly widespread, and is already imposing less load on the DNS system 
> than the use of other arbitrary bogus domains - a practice which is 
> currently even more widespread than the use of ".invalid".

N messages beating up the .com servers, plus M messages beating up on
the .net servers, plus P messages beating up on the .edu servers, plus
Q messages beating up on the .uk servers etc. is quite a different
matter from N + M + P + Q + ... messages resulting in beating up on the
root name servers.  As Keith has pointed out, that's a show-stopper.

Date: Fri, 6 Aug 2004 12:00:29 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: RFC validation samples?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 05 2004, Bruce Lilly wrote:

> If you are going to classify message content, you'll need to be able
> to parse the message body in addition to the header.  For MIME
> messages, you'll also need to be able to handle MIME-part headers,
> boundary delimiters, and body sections.  Including base64 and
> quoted-printable encoded body content.

That is correct. In my reply to Keith Moore, I described that I
already do this kind of thing, not perfectly but sufficiently well for
statistical analysis I think. At this point, I'd like to concentrate
on header parsing and data mining, which is where a lot of interesting
information is buried.

> There is an open-source validating parser available at
> http://users.erols.com/blilly/mparse.  It may do what you want,
> and if not, the package does contain several dozen test messages,
> many of them taken directly from the relevant RFCs (with
> correction per the errata in most cases).

Argh! I could have used this if I hadn't written most of my parser already! 
This is very nice work! Is your code/library thread safe? It does much more
than I need, but looks extremely complete. I shall certainly play with those
test messages.

> 
> > I'm really a newby on mail headers, but I've come across some
> > discrepancies in the RFC2821/RFC2822 grammars which led me to this
> > group.
> 
> There are indeed a few differences; Some are because RFC 2821 is
> SMTP-specific whereas RFC 2822 is intended to be a general message
> format (so, for example, RFC 2822 has a very loose definition of
> "domain-literal" and of the Received header field).  A very few
> are genuine incompatibilities (likely to be corrected in the next
> revisions, due soon).

I believe the easiest (also most maintainable) approach for me is to 
parse each line several times, according to "pure" 2822, 2821, etc.

The incompatibilities are actually a possible "feature".  For
instance, would it be reasonable to expect that a message, none of
whose Received: lines follow 2821 correctly, hasn't passed through an
internet SMTP server? I.e. I'm assuming that the majority of internet
SMTP servers are RFC 821/2821 compliant and follow the grammars therein
correctly.

> There are a few dozen relevant RFCs (depending on what types of
> messages you want to be able to handle, at what level of detail,
> whether or not you want to be able to handle "old" messages, and
> if so, how old, etc.).  Also, don't forget to look at the RFC
> Errata page (http://www.rfc-editor.org/errata.html), as there
> are some errors in published RFCs.

Good advice, thanks.
-- 
Laird Breyer.

Date: Fri, 6 Aug 2004 11:37:41 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: Re: RFC validation samples?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Aug 05 2004, Keith Moore wrote:
> My suggestion is that you ask yourself - do you want a parser that 
> validates email or do you want a parser that is useful for email 
> readers or do you want a parser that can _correct_ malformed email? It 
> can be difficult to do more than one of these at the same time.

That's a good point. I guess I should explain my goals more
comprehensively (warning: a bit longwinded).

I already have a statistical text/email classifier which reads mail
messages. Normally, I point it to sets of mbox files and it creates a
word model. If I point it to a single mail message and give it a few
models, it attempts to pick the best model among alternatives.

I've written this in C and it accepts a hodge podge of RFC related
rules, decodes HTML and MIME structures. These capabilities have
evolved over time as random improvements, for example, MIME structures
are not decoded recursively, instead anything that looks like a MIME
separator is considered putative and if MIME attachment headers follow
then appropriate flags are set (e.g. decode QP until next separator).

The headers in my code are mostly treated as ordinary text lines, and
parsed for word tokens exactly like body lines. Some miscellaneous
rules are applied, such as decoding non-ascii encodings, but again
it's rather ad-hoc, and probably not fully standards compliant as a whole.

For statistical analysis, if parsing is correct 90%+ of the time,
that's good enough, in my experience.

So why do I want to parse headers only? There are two types of goals
I'd like to try my hand on, both of which require an actual
understanding of the elements within the headers, hence parsing more
or less properly (to repeat, so far I simply censor some fields such
as Received: and treat others such as Subject: as ordinary text)

The first type of goal is to data mine headers in a meaningful way, to
improve classification accuracy. This could be as simple as
preprocessing headers and adding a bogus MIME section with some
descriptive comments, which the statistical classifier can pick up on
as ordinary text tokens when it learns/classifies.

This kind of datamining has subtleties, because some types of data are
misleading. For example, the date-time element in some headers is a
bad statistical indicator. If you learn messages which contain many
"Jul" tokens, then come August the statistical model will start to
wonder where the "Jul" tokens went.

The second type of goal is to see if there are enough indicators in
the headers to detect forgery and/or trace the message origin. For
example, there are some simple rules which can be used, such as: the
topmost Received: line is always authentic. I'd like to explore if
there are others, e.g. can we analyse whether a message passed through
the internet (ie at least one compliant SMTP server outside the local
network), Maybe this can't be detected, but perhaps its negation can
(ie this message never left the LAN).

But these are long term goals, and people on this list have likely
thought about these problems very deeply (I would enjoy discussing
these questions on this list). Right now I have a basic parser which
is functional but buggy, which is why I'm looking for test cases.

I started out parsing only 2822 messages, but quickly realized that
2821 is just slightly different, and even so some sample header lines
I have don't appear to comply with either (e.g. in 2821, domains must
contain at least one period, so localhost is not allowed but
localhost.localdomain is). I believe the easiest solution is to parse
each line four times or more, strictly according to 2822,2821,822,821
and special variations, as the code can refer to existing documents
rather than a single complex combination of these four documents which
incorporates many subtleties and variations.

> Or maybe people here could contribute to a list of common email
> format errors.

That would be very useful. If it's appropriate, I'm happy to discuss
on this list any format errors I come across.

-- 
Laird Breyer.

Date: Thu, 05 Aug 2004 21:39:21 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822mailing list <[email-masked]>
Organization: Bruce Lilly
To: ietf-822mailing list <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

More than two weeks have passed with no substantive objection that I can
see to the principle of making the From field optional in the successor to
RFC 2822.  If there are any such objections, please speak up soon.

Date: Thu, 05 Aug 2004 21:36:03 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

Charles Lindsey wrote:

> However, if you want to see what the actual authors of RFC 2606
> intended  it to mean, you might care to look at
> <http://purl.net/net/msgid/[email-masked]>,

I see no "authors" (N.B. your plural).

> which not only confirms its appropriateness within Netnews,

There is also nothing there which says exactly what use is referred to by
"the kind of use you are talking about", or indeed any reference to what
message that might have been in response to.

> No, that is exactly what RFC 3696 does _not_ say. You have already 
> presumed to understand what the authors of RFC 2606 intended to say,
> and  been proved wrong. Do you want me to write to John Klensin and ask
> him  what he really meant in RFC 3696?

Knock yourself out. Go ahead and ask him if he thinks hard-coding
recognition of ".invalid" is a good idea or not.

Cc: Keith Moore <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: RFC validation samples?
Date: Thu, 5 Aug 2004 21:26:13 -0400
To: [email-masked]
Sender: [email-masked]

>> I find it helpful to think of [2]822/MIME (including 2047) as a 
>> single format.
>
> Ah, but then you need to also consider the errata (for 2047) and RFC 
> 2231
> (which amends 2047), and the errata again (for 2231).

absolutely.

Date: Thu, 05 Aug 2004 20:56:11 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

Keith Moore wrote:
> 
>>The other thing a "useful" library for email readers would need to do 
>>is co-exist nicely with tools/library that understands MIME headers. 
>>The MIME headers are often mis-formatted.
> 
> 
> I find it helpful to think of [2]822/MIME (including 2047) as a single format.

Ah, but then you need to also consider the errata (for 2047) and RFC 2231
(which amends 2047), and the errata again (for 2231).

Date: Thu, 05 Aug 2004 20:37:29 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Nick Ing-Simmons <[email-masked]>
CC: [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

Nick Ing-Simmons wrote:

> My guess is very little mail will be strictly RFC2822 yet.
> (SPAM is often less compliant than real mail - this can use used 
>  as a filter :-)).

Sadly, very little mail is RFC 821/822 compliant (the required space
character preceding the semicolon that delimits the time stamp in a
Received field, as required by RFC 821 is a common error).

Date: Thu, 05 Aug 2004 20:27:13 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

Keith Moore wrote:
> 
> My suggestion is that you ask yourself - do you want a parser that
> validates email or do you want a parser that is useful for email readers
> or do you want a parser that can _correct_ malformed email? It can be
> difficult to do more than one of these at the same time.

I believe that correcting errors first requires detecting said errors; i.e.
correcting errors has as a prerequisite a validating parser.

> If you want the latter, you generally need to parse things according to
> RFC 822 rather than 2822, because 822's grammar is simpler and more
> permissive and more representative of what is out there.

In some cases, yes. In other case, RFC 2822 is more permissive. A
disadvantage of RFC 822 is its ambiguities. For example, to take
date-time specification as an example, it could be argued that
RFC 822 permits
   1Jan8412:34UT
And it can be argued that it does not permit that.

> For instance, dates are often malformed (in a wide variety of
> ways), and "." often appears in a phrase before an address.

RFC 2822's parse grammar specifically provides for handling an
unquoted dot in a phrase, and is an example of more permissive
parsing under RFC 2822.

Date: Thu, 05 Aug 2004 20:08:52 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

Laird Breyer wrote:

 > I'm new to this list. I've started implementing a mail header
> scanner/parser which will eventually be released under the GPL, as
> part of a wider mail classification package I'm working on (homepage
> on sourceforge: http://dbacl.sourceforge.net).

If you are going to classify message content, you'll need to be able
to parse the message body in addition to the header.  For MIME
messages, you'll also need to be able to handle MIME-part headers,
boundary delimiters, and body sections.  Including base64 and
quoted-printable encoded body content.

> I'm really a newby on mail headers, but I've come across some
> discrepancies in the RFC2821/RFC2822 grammars which led me to this
> group.

There are indeed a few differences; Some are because RFC 2821 is
SMTP-specific whereas RFC 2822 is intended to be a general message
format (so, for example, RFC 2822 has a very loose definition of
"domain-literal" and of the Received header field).  A very few
are genuine incompatibilities (likely to be corrected in the next
revisions, due soon).

> My current subgoal is to validate each header line separately
> according to the four standards 821/288/2821/2822 and any other
> relevant ones. So each header line will be marked by all the standards
> which apply.

There are a few dozen relevant RFCs (depending on what types of
messages you want to be able to handle, at what level of detail,
whether or not you want to be able to handle "old" messages, and
if so, how old, etc.).  Also, don't forget to look at the RFC
Errata page (http://www.rfc-editor.org/errata.html), as there
are some errors in published RFCs.

> What I would like to know is if there are any publicly available
> validation sample messages which I can use to check correctness of my
> parser.  Apologies if this has been discussed on the list before, I
> have only skimmed the archives. Any other comments and pointers welcome.

There is an open-source validating parser available at
http://users.erols.com/blilly/mparse.  It may do what you want,
and if not, the package does contain several dozen test messages,
many of them taken directly from the relevant RFCs (with
correction per the errata in most cases).

Date: Thu, 5 Aug 2004 12:38:35 -0400
From: Keith Moore <[email-masked]>
To: Nick Ing-Simmons <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: RFC validation samples?
Sender: [email-masked]

> The other thing a "useful" library for email readers would need to do 
> is co-exist nicely with tools/library that understands MIME headers. 
> The MIME headers are often mis-formatted.

I find it helpful to think of [2]822/MIME (including 2047) as a single format.

Cc: [email-masked], [email-masked]
Subject: Re: RFC validation samples?
To: [email-masked]
Date: Thu, 05 Aug 2004 17:15:24 +0100
From: Nick Ing-Simmons <[email-masked]>
Reply-To: Nick Ing-Simmons <[email-masked]>
Sender: [email-masked]

Keith Moore <[email-masked]> writes:
>My suggestion is that you ask yourself - do you want a parser that 
>validates email or do you want a parser that is useful for email 
>readers or do you want a parser that can _correct_ malformed email? It 
>can be difficult to do more than one of these at the same time.

The last two can be combined to some extent.

>
>If you want the latter, you generally need to parse things according to 
>RFC 822 rather than 2822, because 822's grammar is simpler and more 
>permissive and more representative of what is out there.  And you need 
>to look not just at the specifications but also at common kinds of 
>errors.  For instance, dates are often malformed (in a wide variety of 
>ways), and "." often appears in a phrase before an address.
>
>One way to get an idea of common kinds of errors would be to use a 
>strict email syntax checker to validate a large body of stored email 
>(say, from various mailing list archives).  Then you could look at the 
>discrepancies you find and use that information to write a looser 
>parser for use by email readers.  Or maybe people here could contribute 
>to a list of common email format errors.

Those are all sensible suggestions. There are several perl modules 
that do this kind of thing that are widely used by SpamAssassin etc.
that have been "trained" (i.e. manually tweaked until they stopped
complaining) on lots of mail. (They are now very robust.)
If you can read perl their code would give hints on common problems.

My guess is very little mail will be strictly RFC2822 yet.
(SPAM is often less compliant than real mail - this can use used 
 as a filter :-)).

The other thing a "useful" library for email readers would need to do 
is co-exist nicely with tools/library that understands MIME headers. 
The MIME headers are often mis-formatted.

>
>Keith
>
>> I'm new to this list. I've started implementing a mail header
>> scanner/parser which will eventually be released under the GPL, as
>> part of a wider mail classification package I'm working on (homepage
>> on sourceforge: http://dbacl.sourceforge.net).
>>
>> I'm really a newby on mail headers, but I've come across some
>> discrepancies in the RFC2821/RFC2822 grammars which led me to this
>> group. My current subgoal is to validate each header line separately
>> according to the four standards 821/288/2821/2822 and any other
>> relevant ones. So each header line will be marked by all the standards
>> which apply.
>>
>> What I would like to know is if there are any publicly available
>> validation sample messages which I can use to check correctness of my
>> parser.  Apologies if this has been discussed on the list before, I
>> have only skimmed the archives. Any other comments and pointers 
>> welcome.
>>
>> Regards,
>> -- 
>> Laird Breyer.
>>

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: RFC validation samples?
Date: Thu, 5 Aug 2004 10:52:54 -0400
To: [email-masked]
Sender: [email-masked]

My suggestion is that you ask yourself - do you want a parser that 
validates email or do you want a parser that is useful for email 
readers or do you want a parser that can _correct_ malformed email? It 
can be difficult to do more than one of these at the same time.

If you want the latter, you generally need to parse things according to 
RFC 822 rather than 2822, because 822's grammar is simpler and more 
permissive and more representative of what is out there.  And you need 
to look not just at the specifications but also at common kinds of 
errors.  For instance, dates are often malformed (in a wide variety of 
ways), and "." often appears in a phrase before an address.

One way to get an idea of common kinds of errors would be to use a 
strict email syntax checker to validate a large body of stored email 
(say, from various mailing list archives).  Then you could look at the 
discrepancies you find and use that information to write a looser 
parser for use by email readers.  Or maybe people here could contribute 
to a list of common email format errors.

Keith

> I'm new to this list. I've started implementing a mail header
> scanner/parser which will eventually be released under the GPL, as
> part of a wider mail classification package I'm working on (homepage
> on sourceforge: http://dbacl.sourceforge.net).
>
> I'm really a newby on mail headers, but I've come across some
> discrepancies in the RFC2821/RFC2822 grammars which led me to this
> group. My current subgoal is to validate each header line separately
> according to the four standards 821/288/2821/2822 and any other
> relevant ones. So each header line will be marked by all the standards
> which apply.
>
> What I would like to know is if there are any publicly available
> validation sample messages which I can use to check correctness of my
> parser.  Apologies if this has been discussed on the list before, I
> have only skimmed the archives. Any other comments and pointers 
> welcome.
>
> Regards,
> -- 
> Laird Breyer.
>

Date: Thu, 5 Aug 2004 12:52:06 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Subject: RFC validation samples?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Hi everybody,

I'm new to this list. I've started implementing a mail header
scanner/parser which will eventually be released under the GPL, as
part of a wider mail classification package I'm working on (homepage
on sourceforge: http://dbacl.sourceforge.net).

I'm really a newby on mail headers, but I've come across some
discrepancies in the RFC2821/RFC2822 grammars which led me to this
group. My current subgoal is to validate each header line separately
according to the four standards 821/288/2821/2822 and any other
relevant ones. So each header line will be marked by all the standards
which apply.

What I would like to know is if there are any publicly available
validation sample messages which I can use to check correctness of my
parser.  Apologies if this has been discussed on the list before, I
have only skimmed the archives. Any other comments and pointers welcome.

Regards,
-- 
Laird Breyer.

Date: Mon, 02 Aug 2004 21:54:46 -0700 (PDT)
From: [email-masked]
Subject: Re: Has IANA gone mad?
To: [email-masked]
Cc: [email-masked]
Sender: [email-masked]

> [email-masked]  wrote on 29.07.04 in <[email-masked]>:

> > > Kai Henningsen wrote:
> >
> > > > I found out about the datatracker about a week ago, either from
> > > > something here or possibly on [email-masked] - I don't remember exactly.
> > > >
> > > > It may have been discussed at length on various lists, but that
> > > > obviously wasn't enough except for someone who was following one of
> > > > those lists at that time. Which, it seems, didn't include me, or Bruce,
> > > > or persumable a large number of other people.
> >
> > > It's certainly been discussed in some of the plenaries. Some ADs also
> > > make a point of letting you know about it when your I-D gets to the
> > > point of being tracked.
> >
> > And also during apps area sessions, if memory serves. I'm sure it has been
> > covered in WG chair training as well, although I haven't been there to see
> > it.
> >
> > Document authors receive automatic notification of tracker state changes
> > for their documents. This may have been extended to WG chairs by now - I
> > think that was the plan some time back.
> >
> > It was also announced on the main IETF announcement list back on 4-Nov-2002.
> > Discussion has subsequently occured on that list and elsewhere.

> Which pretty much proves my point. It explicitely gets told to a small
> number of people, and was discussed in a small number of places.

An IETF participant, you're supposed to be on the main IETF announcement list.
The list has very intentionally been restructured to facilitate having everyone
on it at least twice (first to split off the IETF discussions from the
announcments, which people shouldn't be required to see, and then to split off
draft announcements, which should be an atom feed anyhow...).

> As a result, one should *expect* the general public to be uninformed.

I expect the general public to be completely uninformed. I have higher
expectations of IETF participants.

				Ned

Date: Tue, 3 Aug 2004 00:12:56 -0400
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

>  But, in this particular case, nothing  actually breaks.

as long as you don't consider pounding the root servers "breakage".

> Moreover, the use of the TLD ".invalid" in munged addresses is already  
> fairly widespread

that's no reason to make it worse.

I think this is what is known as a "showstopper".

Date: Mon, 02 Aug 2004 23:58:32 +0100
From: "Charles Lindsey" <[email-masked]>
To: [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

On Sat, 31 Jul 2004 17:37:29 -0400, Bruce Lilly <[email-masked]> wrote:

>
> Charles Lindsey wrote:

>> It says
>>
>>       ".invalid" is intended for use in online construction of domain
>>       names that are sure to be invalid and which it is obvious at a
>>       glance are invalid.
>
> That is not a recommendation for anything -- it is a statement of intent;
> intent for use "online" and not via hard-coding -- and is certainly not
> a recommendation for use in address fields.

It is a statement of what that TLD is intended to be used for, with the  
expectation that it may well turn up online - i.e. in actual  
communications using assorted internet protocols.

However, if you want to see what the actual authors of RFC 2606 intended  
it to mean, you might care to look at
<http://purl.net/net/msgid/[email-masked]>,
which not only confirms its appropriateness within Netnews, quoting the  
vary same texts I have been quoting at you, but also indicates that  
Netnews seems to have been one of the protocols particularly in mind when  
the .invalid TLD was invented.

>
> You have utterly missed the point of RFC 3696. The point is that
> hard-coding domain names is foolhardy

No, that is exactly what RFC 3696 does _not_ say. You have already  
presumed to understand what the authors of RFC 2606 intended to say, and  
been proved wrong. Do you want me to write to John Klensin and ask him  
what he really meant in RFC 3696?

>  ...  Now,
> IANA doesn't control TLDs, ICANN does; whether you have more or less
> faith in ICANN vs. IANA is another matter...

I have not that much faith in ICANN, but I cannot imagine that they would  
be so stupid as ever to create a real working TLD with the name ".invalid".

-- 
Charles H. Lindsey ---------At  
Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 02 Aug 2004 23:58:36 +0100
From: "Charles Lindsey" <[email-masked]>
To: "ietf-822imcorg list" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

On Sun, 1 Aug 2004 17:13:01 -0400, Keith Moore <[email-masked]> wrote:

> If we were talking about a completely new protocol, I might agree with  
> you.
> But Usenet and email are both widely deployed, and it's not reasonable
> to expect the installed base of Usenet and email software to change to
> avoid the impact on the root servers.

There is a lot that is wrong with Usenet, which is precisely why a new  
standard in needed. Yes, new features in the new standard will take time  
to settle down, and there may be a little pain in the meantime, as is  
always the case when things change. But, in this particular case, nothing  
actually breaks.

Moreover, the use of the TLD ".invalid" in munged addresses is already  
fairly widespread, and is already imposing less load on the DNS system  
than the use of other arbitrary bogus domains - a practice which is  
currently even more widespread than the use of ".invalid".

-- 
Charles H. Lindsey ---------At  
Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Sun, 1 Aug 2004 17:13:01 -0400
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Mandatory From field, anonymity, and hacks
Sender: [email-masked]

> >> The TLD ".invalid" already has the required
> >> property, and is registered with IANA as being guaranteed never to  
> >> resolve to anything.
> >
> > that doesn't do a thing to keep the root servers from getting hit.
> 
> It does if user agents that receive ill-considered requests to send mail  
> to such addresses recognize that particular case and don't waste time  
> looking it up.

If we were talking about a completely new protocol, I might agree with you.
But Usenet and email are both widely deployed, and it's not reasonable
to expect the installed base of Usenet and email software to change to 
avoid the impact on the root servers.  

(for the same reason, the .local TLD as used by a certain poorly designed
local name lookup system is also a bad idea.)