Subject: Message-ID persistence in gateways

From: Jacob Palme <[email-masked]>
Date: Tue, 8 Dec 1998 12:57:33 +0100

draft-ietf-drums-msg-fmt-06.txt specifies:

    The "Message-ID:" field provides a unique message identifier that refers
    to a particular version of a particular message. The uniqueness of the
    message identifier is guaranteed by the host that generates it (see
    below). This message identifier is intended to be machine readable and
    not necessarily meaningful to humans. A message identifier pertains to
    exactly one instantiation of a particular message; subsequent revisions
    to the message each receive new message identifiers.

    Note: When messages are introduced into the transport system, they are
    often prepended with additional header fields such as trace fields
    (described in section 3.6.7) and resent fields (described in section
    3.6.6). Even though the addition of these fields "changes" the message
    in some sense, such additional fields do not require changing the
    "Message-ID:" field of the message. It is, in effect, the same message
    to which transport trace information has been prepended.

This text seems to imply that when you add any other header
than "Received:" AND "Return-Path:", then the Message-ID
should be changed. Is this really what we mean? For example:

(1) Should a gateway from e-mail to news, which adds a
    "Newsgroups:" header to a message, cause the Message-ID to
    be changed?

(2) Same for a gateway from news to e-mail.

(3) Should a mailing list expander, which adds headers like
    List-Unsubscribe (RFC 2369) require the Message-ID to be
    changed?

(4) Should the addition of Resent-headers cause the
    Message-ID to be changed?

In order to decide when the Message-ID should be changed,
we should discuss how it is used. The main uses of the
Message-ID is

(a) To remove duplicates of the same message.

(b) To keep reply threads together.

To do this, Message-ID should certainly be changed if the
body of a message is changed, or if fields like "Subject:",
"Comment:" or "Keywords:" are changed.

In my opinion, the addition of "Resent-" headers should not
change the original Message-ID, but a "Resent-Message-ID:"
can of course be added. Gateways should usually not change
the "Message-ID:" even if the gateways add or remove
headers.

The text of the msgfmt documents should do one of two things:

(A) Specify exactly which changes in headers should and
    should not cause a Message-ID to be changed, or

(B) Use wording which clearly shows that Message-ID can be
    kept when changing anything except the body, "Subject:",
    "Comment:" and "Keywords:" headers, and that gateways
    should, if possible, preserve Message-ID.

Proposal, add the following text to the msgfmt document:

Changes in the body, "Subject:", "Comment:" and "Keywords:"
headers SHOULD cause the message to get a new Message-ID.
Addition of "Resent-" headers or gatewaying SHOULD, if
possible, not cause the message to get a new Message-ID.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

---

From: Pete Resnick <[email-masked]>
Date: Tue, 8 Dec 1998 13:03:05 -0500

> draft-ietf-drums-msg-fmt-06.txt specifies:
[...]
>     often prepended with additional header fields *such as* trace fields
>     in some sense, *such* additional fields do not require changing the

[My emphasis added above.]

> should be changed.

No. That's no what it says. It's very careful to say that fields 
*such as* trace and resent fields that are added during transport 
don't "change" the message from one message into another. So, 
anything that is added in the act of transport (or perhaps similar 
situations which I will discuss below) which don't change the 
identity of the message don't get a new message ID.

> Is this really what we mean? For example:

>     Message-ID to be changed?

That's clear from the text. No.
>     List-Unsubscribe (RFC 2369) require the Message-ID to be

These are all effectively gateway issues. Though this document is 
going to avoid these issues like the plauge, note that in each case 
the gateway is effectively acting as transport and adding trace 
information either relating to how the message was transported, or in 
the last case information about the pipe through which the message 
came. In each case, I would say that these are transport trace fields 
being added to the message and therefore do not require the 
Message-ID to be changed.

> In order to decide when the Message-ID should be changed, we should 
> discuss how it is used.

No, we shouldn't. The semantics of a Message-ID are simply a unique 
identifier for *this* message. If the thing moving the message along 
thinks that what comes in should no longer be treated as the same 
message (for any use) as what is going out, then the Message-ID 
should change. Otherwise it should remain the same.

>     should, if possible, preserve Message-ID.

I disagree completely, and as I remember this has been discussed 
before. Particular fields in the message do not make or break the 
"identity" of a particular message. Someone may add a Keywords field 
for indexing purposes or add trailing advertisements to the body of a 
message through some gatewaying system and that would not constitute 
(for that system) a substantive change to the identity of the 
message. It must be on a case by case basis depending entirely on the 
semantics the sending system wishes to express.

pr
--
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated
Work: (217)337-6377 or (619)651-4478
Fax: (217)337-1980 or (619)651-1102

---

From: Russ Allbery <[email-masked]>
Date: 08 Dec 1998 16:49:48 -0800

> "identity" of a particular message. Someone may add a Keywords field for
> indexing purposes or add trailing advertisements to the body of a

Yup, we've talked about this before.  I agree with the above and stand by
my position that there is no way of formulating any consistent, exact
rules for when the message ID should change.  What one wants to do with
moderation approvals is different than a straight gateway is different
than more complicated gateways, and it also varies by what traffic is
being gatewayed.

-- 
Russ Allbery ([email-masked])         <URL:http://www.eyrie.org/~eagle/>

---

From: Tony Hansen <[email-masked]>
Date: Wed, 09 Dec 1998 01:24:15 -0500

>     to which transport trace information has been prepended.

A related issue is that there is no text that I could find saying that
Message-Id: headers must not change when different "versions" of the message are
sent by the MUA with and without Bcc: headers. Obviously they must use the same
Message-Id: header, but I couldn't find anywhere where it's stated or I thought
it was made clear.

    Tony Hansen
    [email-masked]

---

From: Jacob Palme <[email-masked]>
Date: Wed, 9 Dec 1998 12:56:07 +0100

> identity of the message don't get a new message ID.

If you say that changes in certain fields should *not*
change the Message-ID, it is very easy to misunderstand
your text to imply that other fields *should* change the
Message-ID. At least, you should add a sentence "which
changes in other header-fields will require a change in the
Message-ID is not specfied in this standard" to avoid such
misunderstanding.

It is dangerous to say in a standard that certain header
fields should not change the Message-ID, without mentioning
anything about whether other header fields should change
the Message-ID or not.

In general, I dislike standards which say that something is
"undefined". But it is better to explicitly say that things
are undefined than to use wordings which may be interpreted
differently by different readers. If we absolutely think
that this is undefined for other header fields than trace
and resent fields, then we should at least say so. And also
add something about Bcc fields.

The reason why it is bad so say that things are undefined,
is that this may cause problems in co-working between
different implementations which interpret this in different
ways.

For example, Usenet News uses the Message-ID for loop
control, and this has the effect that if you repost a
message to a new newsgroup with the same Message-ID, it
will not be forwarded correctly. But software which uses
In-Reply-To and References to build threads, will have
problems if different variants of a message has different
Message-ID.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

---

From: Pete Resnick <[email-masked]>
Date: Wed, 9 Dec 1998 07:26:23 -0500

On 12/9/98 at 12:56 PM +0100, Jacob Palme wrote:

> If you say that changes in certain fields should *not* change the 
> Message-ID...

No. That's not what it says. It says that changes in certain fields 
(for example, the addition of Resent-* during reintroduction or 
Received and Return-Path during transport) do not change the message 
into a different message. Whether or not to change the Message-ID is 
not a syntactic issue at all; it's a semantic issue. If the message 
is different, change the Message-ID field. If the message is the 
same, don't change it. But the Message-ID field does not change (or 
fail to change) on the basis which other fields are added or 
subtracted; you can't figure that out from the syntax of the message 
alone.

> ...it is very easy to misunderstand your text to imply that other 
> fields *should* change the Message-ID.

I may be able to clarify by parenthesizing the example and labeling 
it as "for example" instead of "such as". I'll see what I can do.

> At least, you should add a sentence "which changes in other 
> header-fields will require a change in the Message-ID is not 
> specfied in this standard" to avoid such misunderstanding.

Absolutely not. There is nothing unspecified about when to change 
Message-ID; you change it when the message is not functionally the 
same message. How to determine whether two messages are functionally 
the same is tricky, but I think is reasonably well explained by the 
example given.

pr

---

From: Pete Resnick <[email-masked]>
Date: Wed, 9 Dec 1998 16:07:32 -0500

> it as "for example" instead of "such as". I'll see what I can do.

During the WG session, it was suggested that I simply rewrite the 
Note: in section 3.6.4 to clarify this. Here's my attempt for your 
approval (or, more importantly, your disapproval; if there aren't big 
problems with this, it's going in):

Note: There are many instances when messages are "changed", but those 
changes do not constitute a new instantiation of that message, and 
therefore the message would not get a new message identifier. For 
example, when messages are introduced into the transport system, they 
are often prepended with additional header fields such as trace 
fields (described in section 3.6.7) and resent fields (described in 
section 3.6.6). The addition of such header fields does not change 
the identity of the message and therefore the original "Message-ID:" 
field is retained. In all cases, it is the meaning that the sender of 
the message wishes to convey (i.e., whether this is the same message 
or a different message) that determines whether or not the 
"Message-ID:" field changes, not any particular syntactic difference 
that appears (or does not appear) in the message.

pr

---

From: Antony Bowesman <[email-masked]>
Date: Thu, 10 Dec 1998 09:09:22 +0900

Pete Resnick wrote:
> 
>                                          Here's my attempt for your

I thought the original text was pretty clear, however the above adds
extra clarification up to the phrase

> it is the meaning that the sender of the message wishes to convey
> (i.e., whether this is the same message or a different message)'

'sender' is used ambiguously in the draft and if taken here to be the
human, implies they have control over the message-id which should not be
true.

How about,

+ In all cases, the "Message-ID:" references the original mail
+ item in the Originator's mailbox.  A new instance of a message
+ in a mailbox should should determine whether or not the
+ "Message-ID:" field changes, not any particular syntactic
+ difference that appears (or does not appear) in the message.

Rgds
-- 
Antony Bowesman
[email-masked]

---

From: Pete Resnick <[email-masked]>
Date: Thu, 10 Dec 1998 10:36:38 -0500

> human, implies they have control over the message-id which should not be

There is nothing stopping a human from entering a message by hand. I 
don't see that this is a problem in and of itself.

> + item in the Originator's mailbox.

No, that's wrong. First of all, we're not going to talk about "mail 
items" in someone's "mailbox"; those aren't terms used anywhere in 
this document. Second, there may or not be any messages in the 
Originator's "mailbox" when this decision is made, and the existence 
of such may or may not indicate the need for a new message id. If I'm 
forwarding a message with some edits in it, it might be a good idea 
to use a new message-id, even if I preserve the originator and date 
fields for the convenience of my recipient.

pr

---

From: Jacob Palme <[email-masked]>
Date: Fri, 11 Dec 1998 11:30:08 +0100

> that appears (or does not appear) in the message.

Your new text certainly solves the problem I was complaining about.
But Bcc should perhaps be explicitly mentioned?

And one does, however, wonder whether computers are capable
of making these difficult semantic decisions. When I read
your text, I envisage a user interface which forces a user
to specify whether a new sending of a message should get a
new Message-ID or not. But most users will certainly not
want to do this, and will be very confused if the computer
asks them "change Message-ID or not?".

Perhaps change the text:

    In all cases, it is the meaning that the sender of the
    message wishes to convey (i.e., whether this is the same
    message or a different message) that determines whether or
    not the "Message-ID:" field changes, not any particular
    syntactic difference that appears (or does not appear) in
    the message.

to:

    This standard does not specify exactly that certain
    syntactic elements require a new Message-ID. This
    is a semantic decision which is beyond the scope of
    this standard.

Explanation: The reason, why I think this is so important,
is that I believe more mail systems in the future will
have more advanced support for threads. And changing the
Message-ID of a message will break a thread. If we want
to send the same message again with a new Message-ID,
then the old version should be put in the "References"
header or in a new header. This will reduce the risk
of broken threads. However, the new version of the message
might be sent to other recipients than the first version,
and then all recipients will not get enough information
to be able to re-create the thread in their mail data
bases.

Perhaps this should be put into the drums document?
The following text:

    If a message is submitted several times with different
    Message-ID, it is RECOMMENDED to put the Message-ID-s of
    the different submissions into the "References:" header
    field of each other.

---

From: [email-masked] (Kai Henningsen)
Date: 12 Dec 1998 18:08:00 +0200

> of making these difficult semantic decisions. When I read

They aren't. Programmers, admins, and sometimes even end users are,  
however.

> your text, I envisage a user interface which forces a user

Whereas I, at most, envisage a configuration option for a mailing list.

> asks them "change Message-ID or not?".

Don't do that, then.

>     the different submissions into the "References:" header

Sounds good.

MfG Kai

---

From: Simon Lyall <[email-masked]>
Date: Sun, 13 Dec 1998 14:18:48 +1300 (NZDT)

>     the different submissions into the "References:" header

Shouldn't you use the "Supersedes: " or "Replaces: " header for this?

An article which has another in the "References: " header is a specific
followup not a more recent (possibly updated) version.

Of course you could always use "Replaced-By: " and versioning message-ids
but I don't think you want to start _that_ thread.

-- 
Simon J. Lyall.  |   Very  Busy  |   Mail: [email-masked]
"To stay awake all night adds a day to your life" - Stilgar | MT.

---

From: Antony Bowesman <[email-masked]>
Date: Sun, 13 Dec 1998 13:35:48 +0900

> Shouldn't you use the "Supersedes: " or "Replaces: " header for this?

Doesn't follow, e.g. copies to different recipients.  Supersedes is not
an MUA/Gateway decision.

Rgds
-- 
Antony Bowesman
[email-masked]

---

From: Jacob Palme <[email-masked]>
Date: Wed, 16 Dec 1998 10:10:25 +0100

> Shouldn't you use the "Supersedes: " or "Replaces: " header for this?

No, I do not think so. "Supersedes:" or "Replaces:" says
that the new message is a replacement for the old, and
often means that the old message is invalid. If you send a
message more than once with different Message-ID-s, this
need not be because the new message overrides the old
message, they may both be equally valid. (In general, I
would prefer to keep the same Message-ID in this case, but
some people seems to prefer to change the Message-ID, and
this paragraph is for them.)
> followup not a more recent (possibly updated) version.

I understand the new definition of "References:" in the
msg-fmt to mean that the "References" field should list
all, or as many as possible, of the messages in a current
thread. It could typically be used by a receiving client in
order to correlate the set of messages within a thread. And
since two versions of the same message certainly belong to
the same thread (exceptions may occur), the "References:"
header is a way of showing this.