Subject: Re: Format=Flowed/RFC 2646 Bis (-01)
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Date: Thu, 26 Jun 2003 19:34:35 +0000
List-Id: <ietf-822.imc.org>

Randall Gellens <[email-masked]> wrote:

> <ftp://ftp.pensive.org/Public/Randy/draft-gellens-format-bis-01.txt>.

Overall it looks good, but I'm confused by some statements in section
5.5:

    A sequence of quoted lines of the same quote depth SHOULD be encoded
    as a paragraph, with the last line generated as fixed and prior
    lines generated as flowed.

    Consecutive lines with the same quoting depth are considered one
    paragraph and are reformatted together.

The following example seems to refute both statements:

>The quick brown fox jumps over*
>the lazy dog.#
>#
>Four score and seven years ago*
>our fathers brought forth on*
>this continent a new nation.#

That's a sequence of six consecutive lines, all at the same quote depth,
but it's clearly two paragraphs and one blank line, not one paragraph.

I think all this section is really trying to say is that whenever two
adjacent lines have different quote depths, senders should ensure that
the earlier line is fixed (does not end in a space), and receivers
should treat the earlier line as fixed regardless of whether it ends
with a space.  The additional remarks about paragraphs and sections and
sequences of lines are misleading (for me at least).

By the way, the phrase "quote depth" occurs seven times, and the phrase
"quoting depth" occurs twice.  You might want to make those consistent.

AMC

Date: Thu, 26 Jun 2003 10:58:39 -0700
To: [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Format=Flowed/RFC 2646 Bis (-01)
Sender: [email-masked]

An updated draft which is intended to replace RFC 2646 has been sent 
in; because of the crush of last-minute submissions, there may be a 
delay before the announcement appears.  During this time it is 
available at 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-format-bis-01.txt>.

The changes from the -00 version are a few fixes as noted by Jutta 
and Dan Kohn (thank you!) .

I believe this version is ready for IETF last call.

This draft specifies the DelSp parameter to solve the non-Western 
language problem.  Despite the kludge level, this appeared (during 
discussions last year) to be the least problematic solution.  I 
believe it's already been implemented in Apple's Mail application and 
Eudora.

I've also incorporated all the corrections/bugs that were sent to me 
since RFC 2646 was published.  See Appendix A for a list of changes 
from RFC 2646.

This message is BCCd to those who participated in the discussions 
which led to DelSp, and also to those who sent in corrections or bugs.

Abstract

     This specification establishes two parameters to be used with the
     Text/Plain media type, and, in the presence of these parameters, the
     use of trailing whitespace to indicate flowed lines.  This results
     in an encoding which appears as normal Text/Plain in older
     implementations, since it is in fact normal Text/Plain, yet provides
     for superior wrapping, flowing, and quoting.

     This standard supersedes the one specified in RFC 2646, "The
     Text/Plain Format Parameter", and adds the DelSp parameter to
     accommodate languages/coded character sets in which ASCII spaces are
     not used or appear rarely.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
All the world's a stage and most of us are desperately
unrehearsed.                            --Sean O'Casey

Date: Mon, 23 Jun 2003 17:22:58 -0700 (PDT)
From: [email-masked]
Subject: Re: RFCs 822/2882 and backslash-quoted CRLF
To: Bruce Lilly <[email-masked]>
Cc: IETF-822 Mailing List <[email-masked]>, Pete Resnick <[email-masked]>
Sender: [email-masked]

> RFC 822 mentions backslash-quoted CRLF in sections 3.4.3 (comments)
> and 3.4.5 (quoted-strings). While the EBNF for quoted-pair in 822
> permits only a single octet after the backslash, it is clear from
> the normative text in the above cited sections that the interpretation
> of a backslash followed by CR followed by LF in the respective
> contexts is interpreted as a quoted CRLF pair.

> RFC 2822 does not mention backslash-quoted CRLF at all. The ABNF
> for quoted-pair and obs-qp each allow for only a single octet
> following the backslash. Note that 2822 doesn't mention backslash-
> quoted CRLF in section 4 (obsolete syntax) or in Appendix B (summary
> of changes).

> It appears that a backslash-quoted CRLF would be interpreted by an
> RFC 2822 parser as a (deprecated) backslash-quoted CR followed by
> a (deprecated) lone LF.

> That's a different semantic interpretation from 822.

> Given the lack of mention in section 4 and Appendix B, my first
> question is: is the intent of 2822 to prohibit generation of
> backslash-quoted CRLF as described in RFC 822?

I would certainly hope this is the case. The chances of this being widely
interoperable are remote in the extreme.

> Second question: is the different semantic interpretation
> by 2822 parsers intentional?

This is largely irrlevant given the answer to the first question.

				Ned

Date: Sun, 22 Jun 2003 21:10:45 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: IETF-822 Mailing List <[email-masked]>
CC: Pete Resnick <[email-masked]>
Subject: RFCs 822/2882 and backslash-quoted CRLF
Sender: [email-masked]

RFC 822 mentions backslash-quoted CRLF in sections 3.4.3 (comments)
and 3.4.5 (quoted-strings). While the EBNF for quoted-pair in 822
permits only a single octet after the backslash, it is clear from
the normative text in the above cited sections that the interpretation
of a backslash followed by CR followed by LF in the respective
contexts is interpreted as a quoted CRLF pair.

RFC 2822 does not mention backslash-quoted CRLF at all. The ABNF
for quoted-pair and obs-qp each allow for only a single octet
following the backslash. Note that 2822 doesn't mention backslash-
quoted CRLF in section 4 (obsolete syntax) or in Appendix B (summary
of changes).

It appears that a backslash-quoted CRLF would be interpreted by an
RFC 2822 parser as a (deprecated) backslash-quoted CR followed by
a (deprecated) lone LF.

That's a different semantic interpretation from 822.

Given the lack of mention in section 4 and Appendix B, my first
question is: is the intent of 2822 to prohibit generation of
backslash-quoted CRLF as described in RFC 822?

Second question: is the different semantic interpretation
by 2822 parsers intentional?

Best regards,
   Bruce Lilly

Date: Sun, 22 Jun 2003 09:46:08 -0700
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked]
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

Arnt,

AG> A lot of clients seem to put random junk into references and 
AG> in-reply-to. Why would their "threads" generation be any more 
AG> trustworthy?

Good point.  Use of the data needs to be robust against incomplete or
incorrect information, where possible.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

Date: Sun, 22 Jun 2003 09:45:07 -0700
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Graham Klyne <[email-masked]>
CC: Nathaniel Borenstein <[email-masked]>, ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

Graham,

GK> is there scope
GK> here for collapsing burgeoning historic thread information into a hash that
GK> can be used to confirm the sending agent's knowledge of the threads in
GK> progress.

although the typical case is for threads to be short, it is not that
uncomon to have threads that are extremely long.  I think we should
attend to that occurrence.

Whether we do it through a special label (eg, hash) or through a
syntactic way of eliding some of the thread references, I don't know.

GK> Maybe that's overkill:  if an agent supplies just recent generation thread
GK> history in any message it sends, I think the complete thread structure can
GK> be assembled from the various messages.

Perhaps it would be sufficient to have the thread syntax permit saying
something like a sequence:

          <first message in thread>
          <notation saying messages are being skipped>
          <context for tail of thread>
          <message being replied to>?

So, the first two items would be the equivalent of the hash, but without
extra computation.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

From: Arnt Gulbrandsen <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Cc: [email-masked], "D. J. Bernstein" <[email-masked]>
Date: Fri, 20 Jun 2003 15:03:08 +0200
Sender: [email-masked]

Nathaniel Borenstein writes:
> Presumably trn does this by inference when possible, but I can't 
> believe it is *always* possible, depending on which parts of the 
> thread remain available.  One of the questions here is how general a 
> solution we are trying to build.  -- Nathaniel

It isn't always possible, but IMO that's mostly because many clients 
generate contradictory threading information. For example, one 
well-known e-mail client is fond of writing references fields like this 
one:

References: <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]>

Generating correct references isn't hard (in the reply-to-one-message 
case), and people still get it wrong. It seems to me that a suggested 
improvement would have to be significantly simpler, or else people will 
get it wrong, too. And how can something be much simpler than the 
references scheme?

--Arnt

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Multi-message reply and "References:"
Date: Fri, 20 Jun 2003 14:08:22 +0200
Sender: [email-masked]

Nathaniel Borenstein writes, quoting Steven Rohall:
> As one who has been involved in thread "calculation" for some time, I 
> think the biggest advantage of such a scheme is that we can take the 
> guesswork out of threads. In particular, if we define a Threads 
> header that contains all of the lineage information known at message 
> composition, then UAs don't need to calculate threads in some 
> arbitrary fashion at message reception; instead, the threads are 
> precisely known.

A lot of clients seem to put random junk into references and 
in-reply-to. Why would their "threads" generation be any more 
trustworthy?

--Arnt

Date: 19 Jun 2003 17:07:56 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

Nathaniel Borenstein writes:
> Presumably trn does this by inference when possible, but I can't
> believe it is *always* possible

Obviously it isn't always possible. For example, if you don't have the
most recent message in the thread, trn won't include that message in its
picture of the thread.

But none of the proposed More-Thread-Info fields will change this fact,
no matter what trn might do to support them. More-Thread-Info can't say
anything about messages that don't exist yet!

People keep saying that new fields will help message readers do a better
job. I'm asking for evidence to support this claim. References is not
quite _all_ of the available information, but this doesn't mean that
there's any other _useful_ information.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Thu, 19 Jun 2003 11:15:34 -0400
From: Cyrus Daboo <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>, "D. J. Bernstein" <[email-masked]>
cc: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

Hi Nathaniel,

--On Thursday, June 19, 2003 7:54 AM -0400 Nathaniel Borenstein 
<[email-masked]> wrote:

| Presumably trn does this by inference when possible, but I can't believe
| it is *always* possible, depending on which parts of the thread remain
| available.  One of the questions here is how general a solution we are
| trying to build.  -- Nathaniel

The IMAP THREAD extension does a reasonable job of dealing with incomplete 
message threads. Its certainly worthwhile to look at the algorithm 
described there to understand how references can be used.

The point to remember is that if you have a complete message thread (i.e. 
all messages in the thread are in your mailstore) then you don't need 
References at all to build the thread tree - you can simply rely on 
In-Reply-to. Note that multiple message-ids in In-Reply-to will allow you 
to map out multiple parents in the thread tree.

References comes into its own when you have incomplete threads where there 
are message 'holes'. References does not have to be complete - i.e. it does 
not have to go back all the way to the original root message (which is what 
will happen when the header gets too big and has to be truncated) - 
provided that enough messages in the thread are available to allow you to 
jump back over missing messages to form the complete thread tree. 
References effectively provides 'snapshots' of a section of the thread tree 
and those snapshots can be put together in Hockney-esque fashion to form 
the entire thread tree.

I agree with the point that we should not change the current References 
behaviour to try and cope with multiple parents. I think having a new set 
of headers is probably the right way to go.

-- 
Cyrus Daboo

Date: Thu, 19 Jun 2003 09:26:05 +0100
To: Nathaniel Borenstein <[email-masked]>, ietf-822 <[email-masked]>
From: Graham Klyne <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

At 20:20 18/06/03 -0400, Nathaniel Borenstein wrote:
>The only downside I can see to this proposal (and indeed, it is
>potentially a problem with Pete's original proposal) is that the length
>of the Threads header (or the References header in Pete's case) can get
>long.  Is there going to be a problem with MTAs truncating the header?
>Maybe, but I think I'd take a wait and see attitude.  A recent study of
>ours (of ~42000 messages across 8 users) shows that 8010021331513f email
>threads (old-fashioned ones, not Pete's fancy ones) are 5 or fewer
>nodes.  The 97 0x4046acfcoint was at 16 nodes.

Just a random thought...

if one assumes that some of the thread information may be available from 
earlier messages, and that an application that seriously uses this 
information will also have access to the earlier messages, is there scope 
here for collapsing burgeoning historic thread information into a hash that 
can be used to confirm the sending agent's knowledge of the threads in 
progress.

Maybe that's overkill:  if an agent supplies just recent generation thread 
history in any message it sends, I think the complete thread structure can 
be assembled from the various messages.

Now I'll return to my normal programming...

#g

-------------------
Graham Klyne
<[email-masked]>
PGP: 0FAA 69FF C083 000B A2E9  A131 01B9 1C7A DBCA CB5E

Subject: Re: Multi-message reply and "References:"
From: Nathaniel Borenstein <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: ietf-822 <[email-masked]>
Organization: 
Date: 19 Jun 2003 07:54:48 -0400
Sender: [email-masked]

Presumably trn does this by inference when possible, but I can't believe
it is *always* possible, depending on which parts of the thread remain
available.  One of the questions here is how general a solution we are
trying to build.  -- Nathaniel

On Wed, 2003-06-18 at 21:53, D. J. Bernstein wrote:
> Nathaniel Borenstein writes:
> > deleted messages are a real problem for post hoc thread computation
> > and force UAs to rely on subject-based matching
> 
> False. The trn program does a fine job of merging the References fields
> from many messages into a coherent thread picture, even when messages
> have been deleted.
> 
> Maybe putting more information into each message would make trn's job
> easier---but maybe not. Some sort of merging is obviously necessary as
> soon as there are two independent replies to a single message; I see no
> evidence that this software would be simplified by any of the proposals
> here.
> 
> ---D. J. Bernstein, Associate Professor, Department of Mathematics,
> Statistics, and Computer Science, University of Illinois at Chicago

Subject: Re: Multi-message reply and "References:"
From: Nathaniel Borenstein <[email-masked]>
To: ietf-822 <[email-masked]>
Organization: 
Date: 19 Jun 2003 07:53:18 -0400
Sender: [email-masked]

I'm not quite sure how it happened, but in my last message two percent
signs got corrupted, perhaps when I was cutting and pasting.  In any
event, the last two sentences should be changed from:

> A recent study of ours (of ~42000 messages across 8 users) shows that
> 8010021331513f email threads (old-fashioned ones, not Pete's fancy
> ones) are 5 or fewer nodes.  The 97 0x4046acfcoint was at 16 nodes.

to:

> A recent study of ours (of ~42000 messages across 8 users) shows that
> 80% of email threads (old-fashioned ones, not Pete's fancy ones) are 5
> or fewer nodes.  The 97% point was at 16 nodes.

I do hope that came out right this time around...  -- Nathaniel

Date: 19 Jun 2003 01:53:28 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

Nathaniel Borenstein writes:
> deleted messages are a real problem for post hoc thread computation
> and force UAs to rely on subject-based matching

False. The trn program does a fine job of merging the References fields
from many messages into a coherent thread picture, even when messages
have been deleted.

Maybe putting more information into each message would make trn's job
easier---but maybe not. Some sort of merging is obviously necessary as
soon as there are two independent replies to a single message; I see no
evidence that this software would be simplified by any of the proposals
here.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Subject: Re: Multi-message reply and "References:"
From: Nathaniel Borenstein <[email-masked]>
To: ietf-822 <[email-masked]>
Organization: 
Date: 18 Jun 2003 20:20:20 -0400
Sender: [email-masked]

I've been quietly consulting with my colleagues here at Lotus, and what
I've heard back pretty much matches my intuition, which is that we're
better off with a new header that spells out the thread map
unambiguously.  Anyway, herewith the comments from Lotus' thread
specialist.  -- Nathaniel

From: Steven Rohall <[email-masked]>
Subject: Re: Multi-message reply and "References:"

Nathaniel,

Thanks for forwarding this thread (no pun intended) of messages to me.
Basically, I have to agree with Dave.

Although I think that Pete's original proposal could be made to work, I
feel that the addition of semantics on the References header and the
goal of remaining backward compatible are at odds.  As Dave points out,
References would need to be augmented with duplicate message IDs or
special "token" message IDs (e.g., left-p@ren and right-p@ren).  This is
ugly and error-prone.  And, if a separate "ThreadsHead" header is
necessary to make it work, "smart" UAs now need to correlate data from
two headers to compute the threads.  If a separate header is necessary,
why not go all the way and define a Threads header that really does what
is needed:  represents DAGs in a clear an concise manner.

As one who has been involved in thread "calculation" for some time, I
think the biggest advantage of such a scheme is that we can take the
guesswork out of threads.  In particular, if we define a Threads header
that contains all of the lineage information known at message
composition, then UAs don't need to calculate threads in some arbitrary
fashion at message reception; instead, the threads are precisely known.
If a user composes a "new" message, then that message is the root of a
new thread tree and the Threads header simply contains that message's
message-id.  Replies to that message simply extend the Threads header
with their own information.  Indeed the threads are known despite a user
having deleted messages (or not having saved sent messages)--deleted
messages are a real problem for post hoc thread computation and force
UAs to rely on subject-based matching, for example.  (Subject-based
matching in a UA may be very useful, but it is not the same thing as a
thread.)  Subsequent replies simply augment the Threads header without
regard to which messages may or may not still be in the user's mail
database.

The only downside I can see to this proposal (and indeed, it is
potentially a problem with Pete's original proposal) is that the length
of the Threads header (or the References header in Pete's case) can get
long.  Is there going to be a problem with MTAs truncating the header?
Maybe, but I think I'd take a wait and see attitude.  A recent study of
ours (of ~42000 messages across 8 users) shows that 8010021331513f email
threads (old-fashioned ones, not Pete's fancy ones) are 5 or fewer
nodes.  The 97 0x4046acfcoint was at 16 nodes.
                               -Steve.

Steven L. Rohall
Software Architect, IBM Research
One Rogers Street, 5120S, Cambridge, MA 02142
phone: (617)693-1840, fax: (617)693-5551
[email-masked]

Date: Sun, 15 Jun 2003 15:15:46 -0700
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: "Adam M. Costello" <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

Adam,

AMC>   Here is the first idea that comes to my mind:
...
AMC> References: *( [parents] message-ID )
...
AMC> Each message-ID is optionally preceeded by a parents marker, which is a
...
AMC> It might be better to use a new header field for this.  The References
AMC> field was not originally intended to be a reply history, and certainly
AMC> not intended to be this structured.

I think this topic is best pursued in two parts. The part involving an
abstraction is defining a way to notate a forest of thread trees
(remembering that there may be independent origins.)  I do not see how
to do this reasnably, for the References definition in 2822, since it
lacks an internal syntax, and intuiting breakpoints is risky, at best.

I find myself fearing that References is now suffering mightily from the same fate as
Reply-To.  It is a very reasonable field, for its original goal, but is
now suffering from overloading of a second goal.

The problem with the examples that Pete gave in his original posting is
that examples with message id's that are sequential numbers seduces us
into thinking that message id's are related, and of course they are not.

So the current definition relies on repeating some earlier message id
and/or inventing a meta-syntax for References.  I'd class the proposal
to have a Thread-heads header that somehow correlates with References as
seriously complicated.

You get what you need by defining a Threads header that has a real
meta-syntax for laying out a forest of threads in a sequentail
representation.

At best, References should be a flat list of "relevant" messages.  If
you want to a thread meta-structure, define it separate.  If everyone
can be convinced that this meta-structure can be retrofitted to
References, without breaking any existing software, then fine.
Otherwise, use a new header.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

Date: Tue, 10 Jun 2003 11:17:38 -0700 (PDT)
From: [email-masked]
Subject: Re: RFC 2046 message/partial (sect. 5.2.2) clarification
To: [email-masked]
Cc: Bart Schaefer <[email-masked]>, IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

> > Message fragmentation seems pretty hairy.  Does anything actually use
> > it?

Small correction:

> According to my logs, I've received just 100 fragmented messages in the
                                           ^over
> past year.

				Ned

Date: Tue, 10 Jun 2003 10:09:16 -0700 (PDT)
From: [email-masked]
Subject: Re: RFC 2046 message/partial (sect. 5.2.2) clarification
To: Bart Schaefer <[email-masked]>
Cc: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

> Message fragmentation seems pretty hairy.  Does anything actually use
> it?

According to my logs, I've received just 100 fragmented messages in the
past year.

My software silently reassembles such messages. I only become aware that
messages were fragmented at some point when there are reassembly problems. I
don't recall the last time I've had a problem with message reassembly on my
personal setup.

The number of customer questions we get about message reassembly indicates to
me this feature of our software is still in active use at some sites. Probably
not many, but some.

While I would welcome clarifying text about message/partial for a future MIME
update, I do note that there is a big difference between issues that are
largely pedantic in nature and issues that actually cause real world failures.
The operational reality is that message/partial has achieved a significant
measure of operational interoperability with the present specification, warts
and all. And while clarifications are fine, I would be extremely reluctant
to add text that invalidates existing implementations in any way.

				Ned

From: Bart Schaefer <[email-masked]>
Date: Tue, 10 Jun 2003 16:00:14 +0000
Comments: In reply to "Adam M. Costello" <[email-masked]> "Re: RFC 2046 message/partial (sect. 5.2.2) clarification" (Jun  9, 10:54am)
To: IETF RFC-822 list <[email-masked]>
Subject: Re: RFC 2046 message/partial (sect. 5.2.2) clarification
Sender: [email-masked]

On Jun 9, 10:54am, Adam M. Costello wrote:
}
} Message fragmentation seems pretty hairy.  Does anything actually use
} it?

I don't know of anything currently in common use, but back around 1995
when Z-Mail was bundled on SGI workstations as MediaMail, it included
support for sending large media files as message/partial.  A separate
utility (called, cleverly, the "reassembler") was invoked from sendmail
to stash the parts in a temp directory until they'd all arrived, and
then put the message back together, extract the attachment into a new
file, rewrite the message to refer to the file as an external-body part,
and finally drop the result into the user's spool.

Date: Tue, 10 Jun 2003 07:52:18 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: IETF RFC-822 list <[email-masked]>
Organization: Bruce Lilly
To: IETF RFC-822 list <[email-masked]>
Subject: Re: RFC 2046 message/partial (sect. 5.2.2) clarification
Sender: [email-masked]

Adam M. Costello wrote:

> But for new message header fields, if they try to define themselves as
> "F fields" (fields that should go in the enclosed header), existing
> implementations won't know that, and will put them in the enclosing
> header instead.  RFC 2298 provides an example of new header fields
> needing to go in the enclosed header.  If it happened once, it can
> happen again.

Any new MIME extension fields (viz. those with field names beginning
with "Content-" -- barring inappropriate use as in some early MIXER
fields) should be handled correctly. But for others, that is an issue
that occasionally pops up -- RFC 561/724/733/822/2822 message header
fields aren't self-describing and don't have a regimented fixed format.
That means that for certain operations, an implementation must carry
hard-coded information about how to process specific fields, and changes
or additions require updated code (or may lead to interoperability
issues).  Simply parsing header fields requires some hard-coded
information; for example one cannot simply assume that in all cases
parentheses delimit comments -- that doesn't apply to unstructured
fields (or unstructured parts of fields that mix structure and
unstructured content), MIXER fields (where comments are forbidden and
parentheses are used for other constructs), fields that use RFC 2533
"features" specifications (which use parentheses for grouping), or
fields which may contain URIs (parentheses -- not necessarily balanced --
are legal characters in URIs).

> Message fragmentation seems pretty hairy.  Does anything actually use
> it?

I can't say for certain. I haven't seen any actual use of message/partial,
although ad-hoc approaches to splitting large messages and recombining
are in widespread use in Usenet binary newsgroups.  Somebody claimed (a
few months ago) that message/partial constituted a "new" exploit of
anti-virus software, but in reality it turned out that certain AV software
that claimed to function as an email gateway wasn't correctly handling
message/partial (and most likely message/external-body) as documented in
the decade-old RFC 1344.

Incidentally, there's an error in the fragmentation/reassembly example
in RFC 2046 (and as repeated and modified here): the order of Message-ID
and Subject fields is reversed in the reassembled message as compared to
the enclosed header in the first piece, in violation of the "in order"
part of rule 3.

Date: Mon, 9 Jun 2003 10:54:18 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: RFC 2046 message/partial (sect. 5.2.2) clarification
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> (the text was copied and modified from rule 2, so the same objections
> apply to rules 2 and 3).

Good point.  The fragmentation & reassembly rules should be consistent
with each other, and I guess it won't hurt if they are stricter about
order than RFC 822/2822, but it won't really help either, since you
can't assume that the order that is being so carefully preserved is
meaningful in the first place.  And as you pointed out, some orders are
impossible to preserve, because the F and non-F fields get segregated.

> What I had in mind was the pieces and non-F fields.  If those are to
> be preserved through the fragmentation/reassembly process, they have
> to appear in the initial piece message header, and that precludes the
> fragmentation agent from adding its own instance of any such field.

Okay, sorry, I misunderstood the point you were making.  I get it now.

I think I agree with your new recommendation about copying the original
header (except for the F fields) into the header of every fragment.
MIME fragmentation seems very analogous to IPv6 fragmentation, which
specifies that the fragmenter copy all the non-enclosed headers into
every fragment, and that the reassembler keep the first enclosing header
and drop the others.

I notice one key difference, though.  In IPv6, even as new options are
defined, which are unknown to existing implementations of fragmentation
and reassembly, those options will be handled correctly, because they
will appear inside one of a few known header types that indicate whether
they should be outside or inside the fragmentation enclosure (hop-by-hop
and routing headers go outside, others go inside).

IPv4 options have a flag indicating whether they are copied into all
fragments or just the first fragment, to address the same concern.

But for new message header fields, if they try to define themselves as
"F fields" (fields that should go in the enclosed header), existing
implementations won't know that, and will put them in the enclosing
header instead.  RFC 2298 provides an example of new header fields
needing to go in the enclosed header.  If it happened once, it can
happen again.

Message fragmentation seems pretty hairy.  Does anything actually use
it?

AMC

Date: Sun, 08 Jun 2003 22:12:50 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

[update based on review of message/partial]

Bruce Lilly wrote:
> Message-ID: 1
> Body contains header of original message, including the Message-ID: X
> and any References fields (N.B. these are in the *body* of message 1).

The References field (if present) from the original (unfragmented)
message would appear in the header rather than body, however the
Message-ID of the original is in the body of message 1.

> Message-ID: 2
> References: 1
> 
> Message-ID: A
> Body contains fields of fragment 3, which has been further split,
> including (in the *body*) Message-ID: 3 and References: 2.

As above. The header here contains
References: 2

[...]

> A UA which is intended to thread messages by following References and
> In-Reply-To header fields will have to do several things:
> 
> 1. It needs to determine whether a given References field is part of a
>    reply thread or part of a series of partial message fragments. If
>    the message has a Content-Type field specifying a media type of
>    message/partial, it is the latter.
> 
> 2. It needs to (at least virtually) reassemble fragmented messages in
>    order to find the message header (which is encapsulated in the first
>    part of the message/partial fragment series), which in turn is necessary
>    in order to follow Message-ID, In-Reply-To, and References fields. It's
>    also necessary in order to display the correct information for the
>    original message (as opposed to uninteresting information on a series
>    of fragments).

In-Reply-To and References would appear in the header of the first piece.
Message-ID still requires reassembly (or examination of the first piece body).

Date: Sun, 08 Jun 2003 21:53:50 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: IETF RFC-822 list <[email-masked]>
Subject: Re: RFC 2046 message/partial (sect. 5.2.2) clarification
Sender: [email-masked]

Adam M. Costello wrote:

>>    (5)   All of the header fields from the enclosed message,
> 
> 
> You mean the message to be enclosed; it hasn't yet been enclosed.

Yes.

> 
>>          except those that start with "Content-" and the specific
>>          header fields "Subject", "Message-ID", "Encrypted", and
>>          "MIME-Version", MUST be copied, in order, to the header of
>>          the initial enclosing message.  Fields so copied MAY be
>>          elided from the enclosed message fields which appear in the
>>          body of the initial piece,
> 
> 
> That looks like a correct deduction.  Well, I'm not so sure that the
> order MUST be preserved.  They MUST be copied, and based on RFC 2822
> ordering requirements, the order SHOULD be preserved, and the order of
> trace and resent fields MUST be preserved.

OK (the text was copied and modified from rule 2, so the same objections
apply to rules 2 and 3).  I also assumed it's clear that the RFC 2298
rules still apply, i.e. the 3 fields mentioned in 2298 are in the same
category as Message-ID above.  This also raises another transparency
issue: if the original message (all fields to be preserved) has fields

  Subject
  Date
  Message-ID
  From
  MIME-Version
  To
  Content-Type
  References
  Disposition-Notification-To

in that order, the reassembled message will have the F and non-F fields
segregated rather than in the original order; the reassembled message
field order might be

  Date
  From
  To
  References
  Subject
  Message-ID
  MIME-Version
  Content-Type
  Disposition-Notification-To

I.e. if the MUST be copied in order requirement is in support of
transparency, it's inadequate to truly preserve transparency. Weakening
the MUST to a SHOULD certainly wouldn't help :-)  Not a catastrophe, though.

>>          and those which are mandatory in a message SHOULD be copied
>>          to the header of subsequent pieces.
> 
> 
> That might be a good recommendation, but it would be new; it's not
> implied by the existing reassembly spec.  For example, when a message is
> fragmented, the Date field is copied into the first enclosing header.
> The Date fields of the other enclosing headers could be copies of that
> one, or they could indicate the time that the fragmentation took place.
> Either behavior seems reasonable to me, and both result in the correct
> reassembled message.

OK, but there's a slightly related issue that 2046 doesn't address, viz.
transport of the pieces after fragmentation.  Obviously they should be
sent with the same forward and reverse path and mechanism as the
unfragmented original would have used had fragmentation not taken place.
It seems desirable that likewise the (mandatory) originator and
recipient header fields which each piece must have should be copied from
the original. Specifically, it seems unreasonable for a fragmentation
agent to use bogus originator or recipient information in order to
satisfy the requirement that originator and recipient header fields be
present in each piece's header.

Various applications have different mandatory field requirements. For
example, a Path header field is required for a Usenet article (RFC 1036).
The Path field is rather important, and it would be unreasonable for a
fragmentation agent in a Usenet environment to use any Path field for
fragments other than that of the original.

There is also the issue of reassembly, which might have to be handled by
a UA rather than by some transport process. Using the same header fields
in subsequent pieces as in the first piece will tend to group the pieces
in UA display, making reassembly somewhat simpler.

Technically, per RFC 2822 section 3.6.1 the Date field in the pieces
should be the date-time when the piece was prepared (by the fragmentation
agent) for transport, but that precludes preserving the original message
date through the fragmentation/reassembly process.  And the UA grouping
issue also applies to the date-time.  If Date were in the same list as
Message-ID, this wouldn't be an issue; the RFC 2822 sect. 3.6.1 semantics
would be fine.  RFC 822 (the current Standard and the basis for 2046)
doesn't have normative text corresponding to 2822 sect. 3.6.1.  That
looks like it will be an issue for 2822 as it advances on the Standards
Track and/or for a 2046 revision.

>>          in particular, if the message format limits the number of
>>          instances of a field type in a message and that number
>>          of instances appeared in the enclosed message, an entity
>>          generating pieces MUST NOT exceed the limit by producing
>>          another instance of that field type in the initial enclosing
>>          message header.
> 
> 
> The "in particular" connector doesn't make sense, because this is
> talking about ensuring the correct formatting of the reassembled
> message, not the pieces.
> 
> More importantly, it's overstated.  For example, the message format
> limits the number of Date fields to 1, but an entity generating pieces
> MAY put instances of Date in both the enclosed header and the first
> enclosing header, because the instance in the enclosed header will be
> ignored.  The restriction you describe applies only to F fields.

What I had in mind was the pieces and non-F fields.  If those are
to be preserved through the fragmentation/reassembly process, they
have to appear in the initial piece message header, and that precludes
the fragmentation agent from adding its own instance of any such
field. Whether such a field also appears in the enclosed message
header (i.e. in the body of the initial piece) is irrelevant, because,
as you say, it will be ignored on reassembly, and (being in the body)
is ignored during transport of the first piece. Date actually is a good
example because of the issue mentioned above: to preserve the original it
has to be placed in the initial piece message header, which violates
RFC 2822 section 3.6.1, and the fragmenting agent can't comply with
that provision of 2822 by adding another Date field because there can be
only one Date field. The same applies to other fields with limits, e.g.
References.  I wanted to guard against agents that might perform
fragmentation as well as some other process (e.g Usenet moderation,
mailing list expansion, gateways of various types) adding an instance
of a field as well as copying (or moving) an instance of the same field
type from the original message where that would result in violating a limit.

> By the way, the assembled message will have Received fields reflecting
> the path taken by the first part, but the paths taken by the other parts
> will be lost.  Hmmm...

Yes, I thought of that. It seems to be unavoidable and mostly harmless.
If there's a need to examine trace fields (Return-Path as well as
Received), that would have to be done on the individual pieces
before reassembly.

Date: Sun, 8 Jun 2003 23:15:29 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF-822 Mailing List <[email-masked]>
Subject: Re: RFC 2046 message/partial (sect. 5.2.2) clarification
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> However the first piece body does contain X-Weird-Header-1 and
> X-Weird-Header-2 which are not in F but have neither been copied to
> the enclosing message header nor elided.

I think the purpose of those fields in the example is to illustrate that
putting non-F fields into the enclosed header is pointless, because they
won't survive into the assembled message.  If you want a non-F field to
appear in the assembled message, the only way to accomplish that is to
put it in the first enclosing header.  If you want an F field to appear
in the assembled message, the only way to accomplish that is to put it
in the enclosed header.

Duplicating an F field in both the enclosed and first enclosing header
will cause it to be duplicated in the assembled message, which is
probably not what you want.

Duplicating a non-F field in both the enclosed and first enclosing
header is harmless, but the copy in the enclosed header is just a waste
of space, because it is completely ignored.

Fields in non-first enclosing headers have no effect on the assembled
message, but could affect the handling of the outer message before
assembly.

Although the sections of the spec are called "Message Fragmentation
and Reassembly" and "Fragmentation and Reassembly Example", they don't
actually specify or exemplify fragmentation; they specify and exemplify
reassembly (and give one constraint on fragmentation--the line boundary
rule).

But given the reassembly spec, the fragmentor can deduce what it needs
to do in order to cause any desired result at the other end.

>     (5)   All of the header fields from the enclosed message,

You mean the message to be enclosed; it hasn't yet been enclosed.

>           except those that start with "Content-" and the specific
>           header fields "Subject", "Message-ID", "Encrypted", and
>           "MIME-Version", MUST be copied, in order, to the header of
>           the initial enclosing message.  Fields so copied MAY be
>           elided from the enclosed message fields which appear in the
>           body of the initial piece,

That looks like a correct deduction.  Well, I'm not so sure that the
order MUST be preserved.  They MUST be copied, and based on RFC 2822
ordering requirements, the order SHOULD be preserved, and the order of
trace and resent fields MUST be preserved.

>           and those which are mandatory in a message SHOULD be copied
>           to the header of subsequent pieces.

That might be a good recommendation, but it would be new; it's not
implied by the existing reassembly spec.  For example, when a message is
fragmented, the Date field is copied into the first enclosing header.
The Date fields of the other enclosing headers could be copies of that
one, or they could indicate the time that the fragmentation took place.
Either behavior seems reasonable to me, and both result in the correct
reassembled message.

>           All pieces MUST meet the message format requirements;

Of course.

>           in particular, if the message format limits the number of
>           instances of a field type in a message and that number
>           of instances appeared in the enclosed message, an entity
>           generating pieces MUST NOT exceed the limit by producing
>           another instance of that field type in the initial enclosing
>           message header.

The "in particular" connector doesn't make sense, because this is
talking about ensuring the correct formatting of the reassembled
message, not the pieces.

More importantly, it's overstated.  For example, the message format
limits the number of Date fields to 1, but an entity generating pieces
MAY put instances of Date in both the enclosed header and the first
enclosing header, because the instance in the enclosed header will be
ignored.  The restriction you describe applies only to F fields.

By the way, the assembled message will have Received fields reflecting
the path taken by the first part, but the paths taken by the other parts
will be lost.  Hmmm...

AMC

Date: Sun, 08 Jun 2003 16:21:48 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: IETF-822 Mailing List <[email-masked]>
Subject: RFC 2046 message/partial (sect. 5.2.2) clarification
Sender: [email-masked]

RFC 2046 section 5.2.2.1 contains the following text:

    The semantics of a reassembled partial message must be those of the
    "inner" message, rather than of a message containing the inner
    message.  This makes it possible, for example, to send a large audio
    message as several partial messages, and still have it appear to the
    recipient as a simple audio message rather than as an encapsulated
    message containing an audio message.  That is, the encapsulation of
    the message is considered to be "transparent".

    When generating and reassembling the pieces of a "message/partial"
    message, the headers of the encapsulated message must be merged with
    the headers of the enclosing entities.  In this process the following
    rules must be observed:

     (1)   Fragmentation agents must split messages at line
           boundaries only. This restriction is imposed because
           splits at points other than the ends of lines in turn
           depends on message transports being able to preserve
           the semantics of messages that don't end with a CRLF
           sequence. Many transports are incapable of preserving
           such semantics.

     (2)   All of the header fields from the initial enclosing
           message, except those that start with "Content-" and
           the specific header fields "Subject", "Message-ID",
           "Encrypted", and "MIME-Version", must be copied, in
           order, to the new message.

     (3)   The header fields in the enclosed message which start
           with "Content-", plus the "Subject", "Message-ID",
           "Encrypted", and "MIME-Version" fields, must be
           appended, in order, to the header fields of the new
           message.  Any header fields in the enclosed message
           which do not start with "Content-" (except for the
           "Subject", "Message-ID", "Encrypted", and "MIME-
           Version" fields) will be ignored and dropped.

     (4)   All of the header fields from the second and any
           subsequent enclosing messages are discarded by the
           reassembly process.

and section 5.2.2.2 gives an example:

    If an audio message is broken into two pieces, the first piece might
    look something like this:

      X-Weird-Header-1: Foo
      From: [email-masked]
      To: [email-masked]
      Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST)
      Subject: Audio mail (part 1 of 2)
      Message-ID: <[email-masked]>
      MIME-Version: 1.0
      Content-type: message/partial; id="[email-masked]";
                    number=1; total=2

      X-Weird-Header-1: Bar
      X-Weird-Header-2: Hello
      Message-ID: <[email-masked]>
      Subject: Audio mail
      MIME-Version: 1.0
      Content-type: audio/basic
      Content-transfer-encoding: base64

        ... first half of encoded audio data goes here ...

    and the second half might look something like this:

      From: [email-masked]
      To: [email-masked]
      Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST)
      Subject: Audio mail (part 2 of 2)
      MIME-Version: 1.0
      Message-ID: <[email-masked]>
      Content-type: message/partial;
                    id="[email-masked]"; number=2; total=2

        ... second half of encoded audio data goes here ...

    Then, when the fragmented message is reassembled, the resulting
    message to be displayed to the user should look something like this:

      X-Weird-Header-1: Foo
      From: [email-masked]
      To: [email-masked]
      Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST)
      Subject: Audio mail
      Message-ID: <[email-masked]>
      MIME-Version: 1.0
      Content-type: audio/basic
      Content-transfer-encoding: base64

        ... first half of encoded audio data goes here ...
        ... second half of encoded audio data goes here ...

RFC 2298 amends 2046 by adding Original-Recipient, Disposition-Notification-To,
and Disposition-Notification-Options to the list of fields including Message-ID.
For convenience, I'll refer to the set of fields named above (as amended) plus
MIME-content fields as F.

The RFC 2046 text isn't exactly clear about fragmentation and original
message header fields, and the example doesn't show the original message
fields.

Presumably the original message was RFC 822 compliant and therefore had a
Date field and at least one originator field and at least one recipient field.
Those fields are not in F, nor do they appear in the "fields" in the first
piece's message body. However the first piece body does contain X-Weird-Header-1
and X-Weird-Header-2 which are not in F but have neither been copied to the
enclosing message header nor elided. So the first question is: which fields
from the original message (if any) are to be elided from the "fields" in the
first piece body when fragmenting a large message?

Rule 2 refers to "the initial enclosing message" and to "the new message",
which in the case of generating the first piece appear to refer to the
same message and therefore that rule has a null effect in that case. Rule
3 provides for copying the set of fields F (and no others) from the
original message to the new message. That raises a second question: is
there a specific rule for the source of mandatory fields which must appear
in the header of each piece? [obviously the Content-Type, MIME-Version, and
Message-ID fields will be generated by the entity performing the fragmentation;
given the rules for reassembly, the originator (From, Sender, Reply-To, etc.),
recipient, and Date fields had better be copied from the original message, or that
information will be lost (i.e. the fragmentation/reassembly process won't be
transparent -- but there doesn't appear to be a rule that says that that should
be done, and one interpretation of rule 3 is that copying To, From, Date, etc.
(i.e. fields not in F) to the first piece header is forbidden]

Transparency raises a third issue: the process as demonstrated by the
example given *isn't* transparent; any semantics conveyed by X-Weird-Header-1
and/or X-Weird-Header-2 have been lost in the fragmentation/reassembly process.
Any such semantics (in the specific case of X- fields) would have to be agreed
among the sender and recipients, but might be unknown to intermediate entities,
including those that might fragment the original message or one of its pieces
while in transit.  Something seems to be missing from the rules if transparency
is to be maintained as stated in the first paragraph of 5.2.2.1 (Rule 3
implies that if transparency is to be maintained, all fields not in the set F
that are in the original message should be copied (or moved) to the header of
the initial enclosing message when fragmenting -- but clearly that hasn't been
done in the example w.r.t. the X-Weird-Header-* fields, and there doesn't seem
to be an explicit rule covering this situation).  Apart from the X- fields in the
example, this issue also affects standard and extension fields not in the set F,
such as Comments, Keywords, References, In-Reply-To, Resent- fields, etc.

I believe that an additional rule resolves most of these issues rationally:

     (5)   All of the header fields from the enclosed message,
           except those that start with "Content-" and the
           specific header fields "Subject", "Message-ID",
           "Encrypted", and "MIME-Version", MUST be copied, in
           order, to the header of the initial enclosing message.
           Fields so copied MAY be elided from the enclosed
           message fields which appear in the body of the initial
           piece, and those which are mandatory in a message
           SHOULD be copied to the header of subsequent pieces.
           All pieces MUST meet the message format requirements;
           in particular, if the message format limits the number
           of instances of a field type in a message and that
           number of instances appeared in the enclosed message,
           an entity generating pieces MUST NOT exceed the limit
           by producing another instance of that field type in
           the initial enclosing message header.

In addition, rule 3 could be clarified by appending "by the reassembly
process" to the end of the last sentence.

That changes the example to:

    If an audio message is broken into two pieces, the first piece might
    look something like this:

      X-Weird-Header-1: Foo
      X-Weird-Header-1: Bar
      X-Weird-Header-2: Hello
      From: [email-masked]
      To: [email-masked]
      Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST)
      Subject: Audio mail (part 1 of 2)
      Message-ID: <[email-masked]>
      MIME-Version: 1.0
      Content-type: message/partial; id="[email-masked]";
                    number=1; total=2

      X-Weird-Header-1: Bar
      X-Weird-Header-2: Hello
      Message-ID: <[email-masked]>
      Subject: Audio mail
      MIME-Version: 1.0
      Content-type: audio/basic
      Content-transfer-encoding: base64

        ... first half of encoded audio data goes here ...

    and the second half might look something like this:

      From: [email-masked]
      To: [email-masked]
      Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST)
      Subject: Audio mail (part 2 of 2)
      MIME-Version: 1.0
      Message-ID: <[email-masked]>
      Content-type: message/partial;
                    id="[email-masked]"; number=2; total=2

        ... second half of encoded audio data goes here ...

    Then, when the fragmented message is reassembled, the resulting
    message to be displayed to the user should look something like this:

      X-Weird-Header-1: Foo
      X-Weird-Header-1: Bar
      X-Weird-Header-2: Hello
      From: [email-masked]
      To: [email-masked]
      Date: Fri, 26 Mar 1993 12:59:38 -0500 (EST)
      Subject: Audio mail
      Message-ID: <[email-masked]>
      MIME-Version: 1.0
      Content-type: audio/basic
      Content-transfer-encoding: base64

        ... first half of encoded audio data goes here ...
        ... second half of encoded audio data goes here ...

    Which is the original massage plus fields provided by the fragmenting
    entity and transport (in this case the "X-Weird-Header-1: Foo" field).

Date: Sun, 08 Jun 2003 14:06:23 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
CC: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

D. J. Bernstein wrote:
> Bruce Lilly writes:
>   [ ignoring newer USENET docs, and apparently misrepresenting RFC 2822 ]
> 
>>While RFCs 1036 and 2822 suggest ordering of the msg-ids in the References
>>field, there is no requirement to use a specific ordering.  In particular,
>>UAs should not presume that the order of appearance of msg-ids in a 
>>References field is significant.

There are no "newer USENET docs" with any status as Standards. And RFC 2822
(a Proposed Standard) is not misrepresented; RFC 822 (the current message
format Standard) does not require any specific ordering of msg-ids in
References (or In-Reply-To for that matter), and compliance with any RFC
2822 provisions (especially any that aren't specified as MUST or MUST NOT
requirements per RFC 2119) is purely voluntary unless, of course, compliance
with 2822 is claimed.

Date: Sat, 7 Jun 2003 21:22:35 +0000
From: "Adam M. Costello" <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

"D. J. Bernstein" <[email-masked]> wrote:

> It's particularly important for References to start with the thread ID
> and end with the parent ID.

If a message is a reply to multiple messages then there are multiple
parents, each of which may have a different thread ID, in which case
we'd need multiple References fields in order to satisfy this rule.
RFC 822 allows multiple References fields, but RFC 2822 does not (when
creating new messages).

AMC

Date: 7 Jun 2003 17:40:39 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

Bruce Lilly writes:
  [ ignoring newer USENET docs, and apparently misrepresenting RFC 2822 ]
> While RFCs 1036 and 2822 suggest ordering of the msg-ids in the References
> field, there is no requirement to use a specific ordering.  In particular,
> UAs should not presume that the order of appearance of msg-ids in a 
> References field is significant.

In the real world, UAs do pay attention to the order inside References.
It's particularly important for References to start with the thread ID
and end with the parent ID.

My understanding is that trn---which uses References to construct a
thread picture even when some messages are missing---works best when
References is sorted. This is violated by some of the multiple-parent
proposals. Perhaps multiple parents should be handled by simply

   * creating a References field from the newest parent and
   * creating an In-Reply-To field from all the parents.

But I wouldn't support this proposal, or any other proposal, without
seeing a careful investigation of current thread-displaying software.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sat, 07 Jun 2003 12:34:22 -0400
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

> When you reply to a message, you put that message's message-id in the 
> In-Reply-To header.  You copy any References header from that message 
> into the new message, and append the original message-id to it.  So if I 
> have messages 1 (the original), 2 (a reply to 1) and 3 (a reply to 2), 
> the headers look like:
> 
> 1:
>     Message-Id: 1
> 
> 2:
>     Message-Id: 2
>     In-Reply-To: 1
>     References: 1
> 
> 3:
>     Message-Id: 3
>     In-Reply-To: 2
>     References: 1 2

Note that a References header field may appear in contexts other
than a reply: RFC 2046 section 5.2.2.2 specifically permits use
of References with message/partial media types representing
fragments of an original message.  So a large message with id X
may be split for transport into fragments:

Message-ID: 1
Body contains header of original message, including the Message-ID: X
and any References fields (N.B. these are in the *body* of message 1).

Message-ID: 2
References: 1

Message-ID: A
Body contains fields of fragment 3, which has been further split,
including (in the *body*) Message-ID: 3 and References: 2.

Message-ID: B
References: A

Message-ID: 4
References: 3

Note that if transport does not reassemble the message from fragments,
a UA may need to examine body content of messages A and 1 in order to
resolve references.  Also note that some message Y may have a field
References: W X
and that will again require the UA to examine body content (of message 1)
in order to find X in the absence of reassembly by the transport layer.

Reassembly can only reliably take place at the final stage of transport
(since fragments may take different routes, not all fragments are available
at intermediate hosts), and as noted in RFC 2821, it is not always possible
for an MTA to determine whether or not it is making final delivery. So
UAs need to cope with fragments.

> Now let's suppose we have message 4 as a reply to 1, and 5 as a reply to 
> 4.  Those messages look like:
> 
> 4:
>     Message-Id: 4
>     In-Reply-To: 1
>     References: 1
> 
> 5:
>     Message-Id: 5
>     In-Reply-To: 4
>     References: 1 4

While RFCs 1036 and 2822 suggest ordering of the msg-ids in the References
field, there is no requirement to use a specific ordering.  In particular,
UAs should not presume that the order of appearance of msg-ids in a References
field is significant.  So in the example above,

Message-ID: 5
In-Reply-To: 4
References: 4 1

is legal. A UA wishing to construct an ordering of messages 1, 4, and 5
should examine the header fields of messages 1 and 4 as well as 5, and
it doesn't matter what order it does so.

> Now, through the magic of the content concentrator and multiple reply, 
> we reply to 3 and 5 simultaneously.  The resulting message should be 
> constructed as follows:
> 
> 6:
>     Message-Id: 6
>     In-Reply-To: 3 5
>     References: 1 2 3 1 4 5

In this case (and subsequent examples), repeating a msg-id is redundant;
ordering of the msg-ids in the References fields of messages 3 and/or 5
might not be as is assumed, and a UA will have to examine all of messages
1 through 6 to properly construct a partial ordering -- and there's no reason
to examine message 1's fields twice.  Repetition makes the field longer
than necessary (and it is known that some implementations will drop some
msg-ids in long References fields).  As noted by Adam Costello, 2822
discourages use of References when replying to multiple messages, and so
long as each message has its predecessor(s) listed in its In-Reply-To
field, a UA can construct a partial ordering by recursively examining
each message listed in an In-Reply-To field; the References field itself
may be redundant[*].

[...]
> First complication: replying to a combined thread.  Suppose message 8 is 
> a reply to 7.  We think its References header should reflect all the 
> threads it continues, and therefore should be constructed thus:
> 
> 8:
>     Message-Id: 8
>     In-Reply-To: 7
>     References: 1 2 3 7 1 4 5 7
> 
> You know that the threads have been combined because 1 appears twice, 
> and you can figure out the sequencing.

That presumes that every generator of References fields has used the same
algorithm and that no generator or transport has dropped any msg-ids. It
is known that ids are dropped by some implementations, so the implicit
presumptions are not valid.

> Second complication: replying to combined DISTINCT threads.
> 
> Suppose we have message A, B as a reply to A, and C as a reply to B. We 
> now reply to 3 and C, to form message X.  The references header would be 
> generated thus, according to the above algorithm:
> 
> X:
>     Message-Id: X
>     In-Reply-To: 3 C
>     References: 1 2 3 A B C
> 
> The problem is that there is no way to tell that A is not a reply to 3.

Of course there is: message A does not refer to message 3 in either of its
In-Reply-To or References fields.

> Here is where we will have to invent syntax.  Either we make a special 
> "new thread" message-id and insert it:
> 
>     References: 1 2 3 iamspecial A B C

That would be a serious problem. It changes the semantics, and current
UAs wouldn't use the changed semantics. It also won't work if (as is known
to be the case) msg-ids are dropped and/or if some generator uses a
different means of ordering msg-ids or a different "marker" id,

> Or we add a new header:
> 
>     Thread-Heads: 1 A
>     References: 1 2 3 A B C
> 
> Either option would preserve the information that 1 2 3 and A B C are 
> distinct threads.

Unnecessary.

In-Reply-To: 3 C

suffices (the UA of course needs to examine the headers of messages 3 and C
to find 2 and B, then 1 and A; and an implementation could cache the
relationship information, so that need not be expensive) if the intermediate
messages are available. If a References field is also supplied, a UA can
still produce a partial ordering even if some messages are unavailable.

A more complicated scenario can't be readily handled by such a scheme alone,
even with the unrealistically optimistic assumptions that all generators use
the same ordering algorithm (and marker id) and that no generator or transport
ever drops any msg-id. A UA may still need to examine intermediate message
headers in order to determine the message relationships: consider original
messages 1, A, 10, and AA

Message-ID  2
In-Reply-To: 1
References: 1

Message-ID: B
In-Reply-To: A
References: A

Message-ID: 11
In-Reply-To: 10
References: 10

Message-ID: BB
In-Reply-To: AA
References: AA

Message-ID: Q
In-Reply-To: 11 BB
References: 10 11 AA BB

One might eventually get to:

Message-ID: R
In-Reply-To: P Q
...

which could apply equally well to (at least 2) different relationships:

1.

Message-ID: X
In-Reply-To: 2 B
...

Message-ID: P
In-Reply-To: X 10
...

or, 2.

Message-ID: X
In-Reply-To: B
...

Message-ID: P
In-Reply-To: 2 X 10
...

If I understand the algorithm correctly, either case will result in the
same References field (and Thread-Heads if used) for message R.

Determining whether or not message X is a response to message 2 cannot be
determined by looking at the header of message R alone; it is necessary to
examine the header of at least one other message (the obvious preference
being message X if it's available)..

[...]

A UA which is intended to thread messages by following References and
In-Reply-To header fields will have to do several things:

1. It needs to determine whether a given References field is part of a
    reply thread or part of a series of partial message fragments. If
    the message has a Content-Type field specifying a media type of
    message/partial, it is the latter.

2. It needs to (at least virtually) reassemble fragmented messages in
    order to find the message header (which is encapsulated in the first
    part of the message/partial fragment series), which in turn is necessary
    in order to follow Message-ID, In-Reply-To, and References fields. It's
    also necessary in order to display the correct information for the
    original message (as opposed to uninteresting information on a series
    of fragments).

3. It appears to be necessary to examine the Message-ID, In-Reply-To, and
    References fields of each message (including reassembled fragmented
    ones) in order to construct a relationship graph of the messages. At
    least it is not always possible to unambiguously determine the
    relationships using the proposed methods (one could construct a graph
    if the References field contained all of the embedded relationships,
    e.g. if it consisted of ordered pairs of related msg-ids, which would
    lead to quite long fields).  Examiming each message header is not a big
    deal; it is obviously necessary in any event for the UA to be able to
    display subject, date, sender, etc. for each message.

4. The message relationships will yield a partial ordering; the UA might
    use additional information to produce a repeatable full ordering for
    presentation.

----------
* unfortunately, there is some software that so badly botches In-Reply-To
fields that they become unparseable. E.g.:

References: <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]> <[email-masked]>
X-Mailer: Mutt 1.0.1i
In-Reply-To: <[email-masked]>; from [email-masked] on Sat, Jul 14, 2001 at 10:04:00PM -0400

is a relatively mild example. Even allowing phrases in In-Reply-To,
unquoted semicolon, @, commas, and colons aren't valid in a phrase. In
this case, the References field carries redundant information which
provides backup for the botched In-Reply-To field. References also
accounts for cases where some intermediate messages might not be
available to the UA; however repetition of msg-ids is still unnecessary
and because of dropped ids and ordering variation a UA will still need
to examine the available messages to correctly construct an ordering.

Date: Sat, 7 Jun 2003 06:27:49 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Multi-message reply and "References:"
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

"D. J. Bernstein" <[email-masked]> wrote:

> (I have no idea what Costello thinks his proposal is supposed to
> accomplish.)

Me neither, actually.  Someone asked how to represent a message's
ancestry when a message can have multiple parents, and I had an idea, so
I volunteered it.  I don't know what the intention was for using that
ancestry information.

AMC

Date: 7 Jun 2003 06:11:01 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Multi-message reply and "References:"
Sender: [email-masked]

RFC 2822 is incorrect when it suggests that References has all of the
message's ancestors. What References actually has is the start of the
thread, followed by the most recent ancestors, ending with the parent.
Ancient ancestors other than the start of the thread are often missing.

(This is explained in various pieces of USENET documentation, and was
pointed out in the discussions that led up to RFC 2822, so there's
really no excuse for RFC 2822 error.)

Threaded readers reconstruct the thread by looking at References in many
messages. This would obviously be necessary even if the naive RFC 2822
model of References were accurate: a message doesn't know anything about
other branches of the same thread, particularly branches that don't even
exist yet. (I have no idea what Costello thinks his proposal is supposed
to accomplish.)

As for multiple parents: This is the only time that I'd recommend
actually putting In-Reply-To into a message. The reader, after seeing
the multiple parents in In-Reply-To, will realize that References is a
mix of ancestor information from those parents. End of problem.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sat, 7 Jun 2003 06:08:02 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Filtering outgoing messages on DNS names or local-part structure
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

I wrote:

> I never noticed what a nightmare RFC 2368 is in this respect.
>
> This requirement is complete nonsense.

Wow, I really could have found a more tactful way to say that.  I must
have been in an obnoxious mood at that moment, and now I'm embarrassed.
My apologies to the authors.  Speaking of which, I should really glance
at the author list when reading an RFC.  Hi Paul.  :)

Switching to a more constructive mode... if any of the authors of 2368
are reading this, can you remember what the requirement in question was
aiming for?

AMC

Date: Sat, 7 Jun 2003 04:03:54 +0000
From: "Adam M. Costello" <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: Multi-message reply and "References:"
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Pete Resnick <[email-masked]> wrote:

> We've got a new feature we're implementing in Eudora: Replying to
> multiple messages.

Mutt does this.

> Forming the "In-Reply-To:" is easy. You take the message-id's of all
> of the messages to which you're replying and put them in.

That's what mutt does, and what RFC 2822 specifies.

> But "References:" is a different problem.

Agreed.  Mutt constructs some list of message-IDs, but it's not obvious
how, and I can't find any documentation (let alone justification) for
the behavior.  RFC 2822 discourages the creation of a References field
for a reply that has multiple parents.

> As it is used in usenet and elsewhere now, everything assumes a single
> rooted thread.  But when replying to multiple messages, you can end up
> with a multi-rooted thread. And if you reply to two messages that have
> a common root, you end up with branches re-joining.

If every message is a reply to at most one message, then the full graph
is a tree, and the ancestry of any one message is a linear chain, which
is easily represented by the References field.

But if every message can be a reply to several other messages, then the
full graph is a directed acyclic graph (DAG), and the ancestry of any
one message is also a DAG.

If you want to represent that ancestry graph in the References field
(RFC 2822 notwithstanding), I can't imagine how it could be done with
just a list of message-IDs (unless you have lots of duplicates).  I
think you're going to need additional markers.  If additional markers
are acceptable, then you can use them to eliminate all duplicates.  Here
is the first idea that comes to my mind:

References: *( [parents] message-ID )

Each message-ID is optionally preceeded by a parents marker, which is a
hyphen-separated list of integers.  0 refers the previous message-ID in
the list, 1 refers to the message-ID before that, and so on.  A hyphen
with no integers means no parents (the following message-ID was not a
reply to anything).  If the parents marker is absent, it defaults to
"0", which is equivalent to the current convention.

The list can be constructed by doing a reverse depth-first traversal of
the DAG:

    visit(msgid):
      if msgid is not in the references list then begin
        for each parent of msgid do visit(parent)
        let pset = {}
        for each parent of msgid do begin
          find parent in the references list
          add its position to pset
        end
        if pset is {} then let parents = "-"
        else if pset is {0} then let parents = ""
        else let parents = join(pset, "-")
        endif
        append to the references list: parents msgid
      end

    start with an empty references list
    visit(my_id)

It might be better to use a new header field for this.  The References
field was not originally intended to be a reply history, and certainly
not intended to be this structured.

AMC

Date: Sat, 7 Jun 2003 02:20:42 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Filtering outgoing messages on DNS names or local-part structure
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

John C Klensin <[email-masked]> wrote:

> I recently posted draft-klensin-name-filters-01.txt as an I-D.
>
> Comments welcome.

Good idea.  I'd love to see web applications stop mangling the plus-sign
in my email address.

Section 3 says:

         Abc\@[email-masked]

    is a valid form of an email address.

    Conventional double-quote characters may be used to surround strings.

         "Abc@def"@example.com

I think the first form is not valid (but the second is).  If I'm not
mistaken, RFCs 821, 822, 2821, and 2822 all agree that quoted-pairs are
allowed in quoted-strings, but not in atoms.

The character you called "acute accent" is actually a grave accent.

Section 4 says "The syntax for URLs (Uniform Resource Locators) is
specified in [4]" (where [4] maps to RFC 1738).  RFC 1738 specifies
the syntax of some URLs, but others (including http and mailto) are
currently specified by other documents.  See the IANA URI scheme
registry (http://www.iana.org/assignments/uri-schemes).  Most of the URI
schemes listed there are URL schemes, but it's not always easy to tell
which ones.  I have been unable to find any formal definition of URL.

You might want to just avoid the term URL, and say that generic URI
syntax is specified by RFC 2396, and scheme-specific URI syntax is
specified by other documents: RFC 2616 for http, RFC 2368 for mailto,
RFC 1738 for several others; refer the reader to the IANA registry for
the complete list.

Section 4.3:

It looks like some lists of characters were lost.

In the first example, "Joe" has become "joe".  This brings up another
point that would be good to make in this document: filters must not
capitalize or decapitalize letters in local parts, nor in URIs (except
for the host field).

Note 2 says 'There is actually some uncertainty as to whether or not the
"+" characters requires escaping in MAILTO URLs (the standards are not
precisely clear).'  Wow, I never noticed what a nightmare RFC 2368 is in
this respect.  It says:

    Following the syntax conventions of RFC 1738 [RFC1738], a "mailto"
    URL has the form:

    mailtoURL  =  "mailto:" [ to ] [ headers ]
    ...

    Note that all URL reserved characters in "to" must be encoded: in
    particular, parentheses, commas, and the percent sign ("%"), which
    commonly occur in the "mailbox" syntax.

Huh?  Neither parentheses nor comma nor percent are listed as reserved
by RFC 1738 (RFC 2396 lists comma but not the others).  Percent must be
escaped, but not because it is reserved (RFC 1738 calls it "unsafe", and
RFC 2396 calls it "delims", but both explain that it needs to be escaped
because it is the escape character).  RFC 1738 (which is the citation
here) explicitly lists parentheses and comma as characters that need not
be escaped.  But at-signs *are* listed as reserved in both RFC 1738 and
RFC 2396.  So at-signs must be escaped?  But the examples in RFC 2368
show unescaped at-signs.  This requirement is complete nonsense.  If I
were trying to implement mailto URIs, my best guess would be to ignore
it.

A little later, RFC 2368 says 'Within mailto URLs, the characters
"?", "=", "&" are reserved.'  Now that I can believe, because those
characters are indeed used to delimit the components of the mailto URI.

Getting back to section 4.3 of your draft, the third example escapes
the slash but not the equal-sign, which is backwards.  The equal-sign
almost certainly needs to be escaped, because it is explicitly listed as
reserved by RFC 2396, and is actually used as a delimiter in mailto URIs
(in the headers part).  But the slash is in the same boat as plus-sign.
The only thing hinting that slash or plus-sign might need to be escaped
is that crazy sentence that would also imply that at-sign needs to be
escaped, which we know from the examples is wrong.

There is another escaping-related gotcha that often breaks mail
addresses containing plus-signs:

Some server-side programs, when generating HTML containing URIs
containing query-strings containing previous user-supplied data, forget
to apply the x-www-form-urlencoded escaping to that data.  The result
is corruption of any data that contained a plus-sign, percent-sign,
ampersand, or equal-sign.

For example, Yahoo mail is able to reply to email addresses containing
these characters, but if such an address is stored in the address book,
new mail cannot be sent to it.  Hotmail has a similar problem.

Even if x-www-form-urlencoded encoding and decoding is done at all the
right places, there is another potential gotcha for plus-signs, because
the specification of application/x-www-form-urlencoded is broken.  HTML
2.0 and 4.01 both say:

    Space characters are replaced by `+', and then reserved characters
    are escaped as described in RFC 1738: non-alphanumeric characters
    are replaced by `%HH'...

Which is it, reserved characters or non-alphanumeric characters?  Either
way, the specified process is not reversible, because it perfoms %HH
escaping *after* changing spaces to plus-signs.  For example, the values
"foo+bar" and "foo bar" map to the same thing, either "foo+bar" (if
plus-sign is not escaped), or "foo%2Bbar" (if plus-sign is escaped).

As far as I know, browsers always violate the spec and do something
sane instead: they do the %HH escaping *before* changing spaces to
plus-signs, and they include plus-sign in the set of characters to be
escaped.  That way, the server can distinguish between "foo%2Bbar"
(which means "foo+bar") versus "foo+bar" (which means "foo bar").

AMC

Date: Fri, 6 Jun 2003 15:15:39 -0500
To: ietf-822 <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Multi-message reply and "References:"
Sender: [email-masked]

[I am sending this message to the [email-masked] mailing list and I 
am Bcc'ing the USEFOR list. Please post all follow-ups to the 
IETF-822 list; it is a discussion that USEFOR people will be 
interested, but certainly doesn't belong on that list.]

Though I am not working on Eudora full-time anymore, occasionally I 
still do write code and discuss new features. We've got a new feature 
we're implementing in Eudora: Replying to multiple messages. With 
this feature, you can select multiple messages and hit "Reply", and 
Eudora will address the reply to all of the senders of the original 
message (Cc'ing all of the recipients if you say "Reply-to-All"), 
quote the text of those messages in the reply (in some fancy ways 
that have to do with another feature that's being worked on), and 
allow you to edit your reply. That's all well and good.

When the reply is formed, two additional fields appear in the header 
of the message you send: "In-Reply-To:" and "References:". Forming 
the "In-Reply-To:" is easy. You take the message-id's of all of the 
messages to which you're replying and put them in.

But "References:" is a different problem. As it is used in usenet and 
elsewhere now, everything assumes a single rooted thread. But when 
replying to multiple messages, you can end up with a multi-rooted 
thread. And if you reply to two messages that have a common root, you 
end up with branches re-joining. I've got an idea of what a 
"References": field should look like, but I thought others might want 
to give their input.

Below is a message from Steve Dorner with the idea that he and I 
worked out. We'd like a little feedback on the proposal, including 
how badly it's going to screw up current implementations. (Screwing 
up current implementations is not a show-stopper, but we'd consider 
reasonable workarounds if you can come up with some.) All of the 
below is perfectly within the 822/2822 semantics, but it's definitely 
not something that anything can deal with now.

Comments?

pr

--- begin forwarded text

Date: Thu, 29 May 2003 11:23:12 -0700
To: Eudora-Losers:;
From: Steve Dorner <[email-masked]>
Subject: multi reply and references

Pete and I spent some time discussing how to build proper 
In-Reply-To: and References: headers for replying to more than one 
message at a time.

Review:

When you reply to a message, you put that message's message-id in the 
In-Reply-To header.  You copy any References header from that message 
into the new message, and append the original message-id to it.  So 
if I have messages 1 (the original), 2 (a reply to 1) and 3 (a reply 
to 2), the headers look like:

1:
	Message-Id: 1

2:
	Message-Id: 2
	In-Reply-To: 1
	References: 1

3:
	Message-Id: 3
	In-Reply-To: 2
	References: 1 2

Now let's suppose we have message 4 as a reply to 1, and 5 as a reply 
to 4.  Those messages look like:

4:
	Message-Id: 4
	In-Reply-To: 1
	References: 1

5:
	Message-Id: 5
	In-Reply-To: 4
	References: 1 4

Now, through the magic of the content concentrator and multiple 
reply, we reply to 3 and 5 simultaneously.  The resulting message 
should be constructed as follows:

6:
	Message-Id: 6
	In-Reply-To: 3 5
	References: 1 2 3 1 4 5

Now let's assume message 7 is a reply to messages 1-5.  It should be 
constructed thus:

7:
	Message-Id: 7
	In-Reply-To: 1 2 3 4 5
	References: 1 2 3 1 4 5

How did we come up with that?  Well, the theory behind the references 
header is that it should trace the entire thread.  In our case, we're 
combining threads, and so we want to represent each thread uniquely 
and distinctly, but not redundantly.  The algorithm is as follows:

- Make a list of all messages being replied to.

- Take all the message-id's from the messages in the list and put 
them into the new In-Reply-To:.

- Aggregate all the references from all the messages in the list. For 
each message in the list, search for that message's message-id in the 
aggregated references.  If it appears, remove it from the list 
(because it is being referred to by another message, it's clear

- For each message left in the list, take its references, append its 
own message-id, and add it to the references for the new message.

There are two more complications.

First complication: replying to a combined thread.  Suppose message 8 
is a reply to 7.  We think its References header should reflect all 
the threads it continues, and therefore should be constructed thus:

8:
	Message-Id: 8
	In-Reply-To: 7
	References: 1 2 3 7 1 4 5 7

You know that the threads have been combined because 1 appears twice, 
and you can figure out the sequencing.  (What a doctrinaire reference 
threading UI does with this, I neither know nor care, but at least 
the information is there.)

Second complication: replying to combined DISTINCT threads.

Suppose we have message A, B as a reply to A, and C as a reply to B. 
We now reply to 3 and C, to form message X.  The references header 
would be generated thus, according to the above algorithm:

X:
	Message-Id: X
	In-Reply-To: 3 C
	References: 1 2 3 A B C

The problem is that there is no way to tell that A is not a reply to 
3.  Here is where we will have to invent syntax.  Either we make a 
special "new thread" message-id and insert it:

	References: 1 2 3 iamspecial A B C

Or we add a new header:

	Thread-Heads: 1 A
	References: 1 2 3 A B C

Either option would preserve the information that 1 2 3 and A B C are 
distinct threads.  Probably we would revise message 8 to follow 
whatever explicit syntax we use to solve this problem, even though 
it's not strictly necessary:

8':
	Message-Id: 8
	In-Reply-To: 7
	References: 1 2 3 7 iamspecial 1 4 5 7

8'':
	Message-Id: 8
	In-Reply-To: 7
	Thread-Heads: 1
	References: 1 2 3 7 1 4 5 7

--- end forwarded text

-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Fri, 06 Jun 2003 16:02:50 -0400
From: John C Klensin <[email-masked]>
To: [email-masked]
Subject: Filtering outgoing messages on DNS names or local-part structure
Sender: [email-masked]

Hi.

I recently posted draft-klensin-name-filters-01.txt as an I-D. 
It addresses the issues raised by web sites thinking they know 
all of the top-level domain names (so that, e.g., forms that 
specify domains of ".museum" or ".aero" are rejected as invalid) 
and by various applications that are convinced they know the 
valid formats of email local-parts and hence reject 
foo+bar@domain, bar/baz@domain, and a series of other things as 
invalid.

It is just an analysis and reprise of existing standards, and 
doesn't propose any new standards.  The hope is that, by drawing 
things together into one place, we might have something easy to 
point to and get a bit more conformance.

Comments welcome.

    john