Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
From: SM <[email-masked]>
To: [email-masked]
Date: Tue, 30 Nov 2004 22:46:05 -0800
List-Id: <ietf-822.imc.org>

Hi Steve,
At 15:36 30-11-2004, Steve Dorner wrote:
>I just spoke to a fellow with a very interesting email proxy system that 
>unfortunately works by using such []'ed cruft in subject fields. I 
>suppressed my gag reflex, because I understood why he was doing it. Most 
>current MUA's hide all headers except a specific few, so if you want to 
>communicate anything to the end user, the only way you have to do it is by 
>scribbling on those few visible fields.  Or in the body, which is even 
>worse.  Gone are the days when you could add a header and expect anyone to 
>see it.  So, list processors aside, I think we're going to see more of 
>this and not less.

Tags are usually inserted because of the limitations of a widely used 
MUA.  The tags can be used to convey information for filtering as the MUA 
does not have the capability to set rules based upon the List-Id or X headers.

We are going to see more of this because "everyone" does it and believe 
that is the norm.

>Could this be generalized as a recommendation to MUA's?  I dunno.  It 
>seems like a feature that's useful a lot of the time, and harmful 
>occasionally (eg, when there is real information inside []'s for some reason).

The tags that are inserted in the Subject header break DomainKeys.  They 
can also be confusing when you have a list message forwarded to another 
lists as you see multiple tags in the Subject header then.

Regards,
-sm 

Date: Wed, 1 Dec 2004 16:28:22 +1000
From: Laird Breyer <[email-masked]>
To: Steve Dorner <[email-masked]>
Cc: Peter Koch <[email-masked]>, [email-masked]
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Reply-To: [email-masked]
Mail-Followup-To: Steve Dorner <[email-masked]>, Peter Koch <[email-masked]>, [email-masked]
Sender: [email-masked]

On Nov 30 2004, Steve Dorner wrote:
> 
> I just spoke to a fellow with a very interesting email proxy system 
> that unfortunately works by using such []'ed cruft in subject fields. 
> I suppressed my gag reflex, because I understood why he was doing it. 
> Most current MUA's hide all headers except a specific few, so if you 
> want to communicate anything to the end user, the only way you have 
> to do it is by scribbling on those few visible fields.  Or in the 
> body, which is even worse.  Gone are the days when you could add a 
> header and expect anyone to see it.  So, list processors aside, I 
> think we're going to see more of this and not less.
> 

This sounds like something that POPFile also does (which is a statistical
classifier proxy for POP3 and SMTP, reads incoming mail and tags it
according to one of a number of categories, then passes it on. The 
MUA reads the tag and performs actions on the message).

FWIW, I happen to know that the reason [] is added to the subject in that 
particular case is because Outlook Express doesn't allow automatic filing 
of messages based on arbitrary header fields, but can trigger rules
based on words in the subject. Other mail readers are supposed to use
X-POPFile-something. I believe OE is the only braindead MUA of this kind.
 
> My MUA ignores such text when doing subject comparisons.
> 
> Re: foo
> foo
> Re[2]: foo
> [mydumblist] foo
> 
> are all considered equal.  A further step that would probably be 
> interesting would be to simply remove such strings from subjects 
> displayed in message lists.
> 
> Could this be generalized as a recommendation to MUA's?  I dunno.  It 
> seems like a feature that's useful a lot of the time, and harmful 
> occasionally (eg, when there is real information inside []'s for some 
> reason).
> 

You can never distinguish a machine added [] tag from a parenthesis 
inserted by a human. In fact, POPFile is a good example, because the 
contents of the [] can be literally any string the user wants to insert
as a category label. So your MUA would have to query POPFile for the 
list of category strings used or ask the user etc.
 
> I personally wouldn't be opposed to amending 2822 to declare that [] 
> are comment indicators in subject fields.  I know that sins against 
> the holy writ, but oh well...

If a MUA did consider [] as a comment, wouldn't that break "Re[2]:
foo" parsing ?

Laird Breyer.

Date: Wed, 1 Dec 2004 06:55:48 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Cc: Steve Dorner <[email-masked]>, Peter Koch <[email-masked]>
Sender: [email-masked]

Let me quote selectively.

Steve Dorner writes:
> I just spoke to a fellow with a very interesting email proxy system 
> that unfortunately works by using such []'ed cruft in subject fields. 
> I suppressed my gag reflex, because I understood why he was doing it. 
> Most current MUA's hide all headers except a specific few, so if you 
> want to communicate anything to the end user, the only way you have 
> to do it is by scribbling on those few visible fields.
...
> A further step that would probably be interesting would be to simply 
> remove such strings from subjects displayed in message lists.
>
> Could this be generalized as a recommendation to MUA's?  I dunno.  It 
> seems like a feature that's useful a lot of the time, and harmful 
> occasionally (eg, when there is real information inside []'s for some 
> reason).

Sounds like an arms race.

1. Proxy adds header fields.
2. MUA learns to show only a select few fields.
3. Proxy sticks information into subject using []
4. MUA learns to suppress [].

In step 5, the proxy must stop using [] and switch to something it's 
more difficult to suppress.

The same situation applies to disclaimer/advert footers, I suppose.

Arnt

Date: Wed, 1 Dec 2004 06:03:26 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Attempts at establishing harmful conventions
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
> proxies and filters should not be writing in the same portions of the 
> message that the author wrote in.

Oh, fine. I misunderstood your earlier messages, then.

Arnt

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Wed, 1 Dec 2004 00:02:33 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>> IMHO - if the subject line is in Klingon, any annotations should also
>> be in Klingon.
>
> Uh. Steve Dorner mentioned �proxies and filters that all would like to
> communicate with the recipient�. Do you think such proxies and filters
> should be required to detect the language used and insert their
> thingammajigs in the detected language?

proxies and filters should not be writing in the same portions of the 
message that the author wrote in.

Date: Wed, 1 Dec 2004 05:47:39 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Attempts at establishing harmful conventions
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
>> How would you handle foreign-language abbreviations?
>
> IMHO - if the subject line is in Klingon, any annotations should also 
> be in Klingon.

Uh. Steve Dorner mentioned �proxies and filters that all would like to 
communicate with the recipient�. Do you think such proxies and filters 
should be required to detect the language used and insert their 
thingammajigs in the detected language?

Arnt

Date: Wed, 1 Dec 2004 12:46:46 +1000
From: Laird Breyer <[email-masked]>
To: Steve Dorner <[email-masked]>
Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Reply-To: [email-masked]
Mail-Followup-To: Steve Dorner <[email-masked]>, Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
Sender: [email-masked]

On Nov 30 2004, Steve Dorner wrote:
> 
> At 7:10 PM -0500 11/30/04, Keith Moore wrote:
> >users are lazy - they will sometimes reply to a message just as an 
> >easy way to get the same recipient list, even if the topics are 
> >different.  often they don't even change the subject.
> 
> Barring Bayesian analysis, I think we're SOL if they don't change the subject.
> 
> But if they do change the subject, and one slavishly follows IRT 
> anyway, how is that A Good Thing?

This is somewhat complex. Bayesian analysis has the most clues to work
with if it's allowed to examine the full message body. 

Headers tend to be too similar across threads, so detecting a new
thread simply on the basis of header statistics would probably have
low success rate (contrast with spam filtering, where the similarity
of headers actually helps to distinguish interesting messages from
uninteresting ones).

Unfortunately, retrieving/analysing full bodies are expensive
operations, for example think of an IMAP server where the MUA works
with headers only and downloads bodies only if absolutely necessary.

So Bayesian analysis for the purpose of thread idetification is not
something that can be plugged in easily, it needs a support framework
(ie inexpensive availability of the full message, caching of computed
scores etc). 

Laird.

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 21:52:24 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

> How would you handle foreign-language abbreviations?

IMHO - if the subject line is in Klingon, any annotations should also 
be in Klingon.

Cc: Steve Dorner <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: MoreOn: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 21:50:58 -0500
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

>> I see (at least) two distinct kinds of badness associated with 
>> [listname] tags.  One is that they get in the way of searches, 
>> comparisons, display, etc. so there's a temptation to remove or 
>> ignore them just as with Re:, Fwd: and similar tags.
>
> I'm not sure this is a problem.  Is it a problem if a search engine 
> builds in a smart spelling algorithm, so that searching for "colour" 
> finds "color" and "collor"?  It seems to me that a similarly "smart" 
> email system would build in heuristics based on common human email 
> conventions.  Where's the problem?

one man's crisis is another man's opportunity :)

>> The other is the harm to transparency - these tags alter the message 
>> content from what the original author intended.  Footers added by 
>> lists cause the same kinds of problems.  This begs the question - is 
>> the purpose of the list to be a transparent multicast channel or is 
>> it an original source of content?
>
> As usual, you've asked precisely the right question.  I think it 
> points to the fundamentally human element here, which is that some 
> lists have the former purpose and some the latter.  One might 
> taxonimize them as "transparent mailing lists," "header-modifying 
> mailing lists," "body-modifying mailing lists," and "non-transparent 
> mailing lists."  Any of these might reflect a reasonably legitimate 
> set of needs on the part of the list owner.  I think we just have to 
> deal with it.  It might be nice, however, if there were a way to tell 
> the difference.

Some lists definitely have a purpose at one or the other of the 
extremes.  But I'm seeing a lot of lists that are mostly transparent, 
but still want to make minor modifications - usually either to 
"enhance" the user's experience or to help pay for the list service.  
Unfortunately, short-term "enhancements" often produce dysfunctional 
behavior in the long term.  As for "legitimate needs" - I'd say there 
are desires of list owners and desires of list users and they're often 
in conflict.  For that matter, the desire of one list user is often in 
conflict with the desires of another.  But some of these desires are 
fairly benign, while others, if implemented, impair the ability of the 
mail system to evolve to better accommodate users.

>> For instance, the message-id should be changed, in-reply-to and 
>> references fields should be cleared or altered, etc.
>
> Or, perhaps, a "List-modifications" field could be defined.
>
>> As for the "dog" analogy:  we can scarcely prevent any kind of 
>> protocol violation.  But we can at least point out that some 
>> practices are undesirable and try to discourage them.
>
> Perhaps we need the equivalent of a "Curb Your Dog" sign for mail sent 
> to mailing lists?  A "please don't screw with this message" flag?  
> Would that be useful in any way?

well, most people understand that it's not acceptable to let their dogs 
urinate anywhere they want to...that's why they take them outside at 
regular intervals.  so maybe we need the equivalent of designated 
doggie walks for list owners :)

Keith

Date: Wed, 1 Dec 2004 12:33:36 +1000
From: Laird Breyer <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Reply-To: [email-masked]
Mail-Followup-To: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
Sender: [email-masked]

On Nov 30 2004, Keith Moore wrote:
> 
> I don't think this is cruft - I think it's useful information that 
> reasonably belongs in the subject field.  what's broken is something 
> else - perhaps our expectation that all messages in a thread should 
> have the same subject.

Altering the subject is a hack necessary to achieve the intended
effect in commonly used MUAs, but there is no justification beyond
this I can think of. It shouldn't be encouraged, as once the most
popular mail reading software fix this limitation, it will just
be a convention whose motivation is no longer relevant, but whose
consequences will stay. 

Since clearly tagging is somewhat valuable information (and given that
we've entered the information glut age, I think we can expect more
varied tagging in the future, not less), there is still the question
where best to put such tags. With this assumption, here are some thoughts:

There's really two places where such information can go, the body or
the header.

Bruce made the case in a previous discussion for inserting such 
extra information as an attachment, so that the original message is 
available unaltered. 

For tagging, I think it's better to put such information somewhere in the
headers. For example, if your mail resides on an IMAP server, you'll
typically have access to the headers, but the full body won't be
downloaded until absolutely necessary. So a tag which can assist in 
collating messages would be useless anywhere but in the message header.

The subject is a great hack for placing tags, because the Subject: field
is some sort of lowest common denominator among MUAs, it's nearly always
displayed to the user. So putting a tag there ensures the greatest likelihood
that the recipient's MUA will have the facilities needed to access the 
tag, and that's why it's the natural field used by taggers.

Looking towards the future, if every potential tagger adds their own
X- header field say, then MUAs will never converge to consistent 
similar behaviour, because each such tag would have to be treated 
as a special case by each MUA. What's needed is a convention for 
tagging which all taggers and MUAs can agree on. Without one, the
least common denominator stays the Subject: field, and therefore it will
continue to be misused.

So it seems to me inevitable that some tagging field (Tagged:,
Processed:, or whatever) must be devised. Perhaps the existing Comment:
field can be used for that purpose? I don't know.

Laird.

Date: Wed, 1 Dec 2004 03:12:36 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Attempts at establishing harmful conventions
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
>  where do you draw the line between what belongs in the subject and 
>  what belongs elsewhere?

Language.

On one side of that boundary, there's text in Hindi or whatever else the 
sender sent. The message recipient can be expected to understand that 
language, but neither the transport system nor the recipient's MUA.

> I don't think this is cruft - I think it's useful information that 
> reasonably belongs in the subject field.

How would you handle foreign-language abbreviations?

Arnt

Cc: Steve Dorner <[email-masked]>, ietf-822 <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: MoreOn: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 21:07:39 -0500
To: Keith Moore <[email-masked]>
Sender: [email-masked]

[Sorry, couldn't resist continuing to play with the subject line.]

On Nov 30, 2004, at 8:09 PM, Keith Moore wrote:

> I see (at least) two distinct kinds of badness associated with 
> [listname] tags.  One is that they get in the way of searches, 
> comparisons, display, etc. so there's a temptation to remove or ignore 
> them just as with Re:, Fwd: and similar tags.

I'm not sure this is a problem.  Is it a problem if a search engine 
builds in a smart spelling algorithm, so that searching for "colour" 
finds "color" and "collor"?  It seems to me that a similarly "smart" 
email system would build in heuristics based on common human email 
conventions.  Where's the problem?

> The other is the harm to transparency - these tags alter the message 
> content from what the original author intended.  Footers added by 
> lists cause the same kinds of problems.  This begs the question - is 
> the purpose of the list to be a transparent multicast channel or is it 
> an original source of content?

As usual, you've asked precisely the right question.  I think it points 
to the fundamentally human element here, which is that some lists have 
the former purpose and some the latter.  One might taxonimize them as 
"transparent mailing lists," "header-modifying mailing lists," 
"body-modifying mailing lists," and "non-transparent mailing lists."  
Any of these might reflect a reasonably legitimate set of needs on the 
part of the list owner.  I think we just have to deal with it.  It 
might be nice, however, if there were a way to tell the difference.

> For instance, the message-id should be changed, in-reply-to and 
> references fields should be cleared or altered, etc.

Or, perhaps, a "List-modifications" field could be defined.

> As for the "dog" analogy:  we can scarcely prevent any kind of 
> protocol violation.  But we can at least point out that some practices 
> are undesirable and try to discourage them.

Perhaps we need the equivalent of a "Curb Your Dog" sign for mail sent 
to mailing lists?  A "please don't screw with this message" flag?  
Would that be useful in any way?

> Humans want things to be mostly unstructured; computer programs want 
> things to be highly structured.  Humans prefer loose definitions for 
> data elements; computer programs prefer strict definitions.  And yet 
> we want computer programs that deal with human natural language 
> communications.  This seems like an inherent conflict that we're not 
> likely to resolve.

Very well put.  But a good first step is to distinguish the parts of a 
program (e.g. an MUA) that are dealing with structured protocols like 
MIME from those that are dealing with human conventions like those in 
the subject lines.  That way you can keep the heuristics in the parts 
of the software that actually need them.  Your MIME parser really 
shouldn't have to make too many guesses, but your subject line parser 
will probably be better if it makes some informed guesses about human 
conventions.  -- Nathaniel

Cc: Nathaniel Borenstein <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Company-Confidential indication
Date: Tue, 30 Nov 2004 21:03:32 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

On Nov 30, 2004, at 8:42 PM, Bruce Lilly wrote:

> A (possibly) interesting side-note to the issue of company-confidential
> indications that Nat brought up: both VPIM and MIXER use a
> Sensitivity message header field, however, while MIXER provides
> for a "Company-Confidential" value for the field body, VPIM
> does not.  Otherwise, the syntax and values appear identical.
>
> So a message with
>   Sensitivity: Company-Confidential
> might be a legal MIXER message or a VPIM message with illegal
> content; and it's not easy to determine which...

sounds like an opportunity for an addition to one or two of the RFC 
Editor's errata pages:

1. for VPIM - point out (a) that MIXER also uses the Sensitivity field 
(b) that it's possible to receive a message from a MIXER gateway that 
has such a field with the value of Company-Confidential and (c) VPIM 
implementations should not break when they see such a field.

2. for MIXER - point out that VPIM also uses the Sensitivity field, and 
(if the VPIM uses are at odds with those in MIXER) point out that MIXER 
gateways should do something reasonable if they see Sensitivity fields 
with values they don't understand.

Keith

p.s. Grumble...when you set reply-to to point to the list rather than 
you, I have to remember to leave the
  "On <date>, at <time>, Bruce Lilly wrote:"  in the message body.  I'm 
so accustomed to removing it that I do it without thinking.  I hate 
reading that cruft when it's redundant with the information in the 
message header, but without having you explicitly listed in the To 
field of the reply, there's a need for it.

Maybe this is another case where the recipient's UA should be supplying 
context based on References or In-Reply-To instead of having the reply 
author's MUA include it in the message?

Cc: Steve Dorner <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 20:43:16 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

> How many UAs make it possible, let alone easy, for authors to
> edit References or In-Reply-To fields?
>
> For that matter, how many permit a user to select multiple
> messages to compose a response citing all of the selected
> messages (including appropriate setting of In-Reply-To and
> References fields)?
>
> This is probably an area where some advice to UA authors
> is called for.

seems like we need some experiments before making such recommendations. 
  we can usually guess how a protocol will function without extensive 
real-world testing.  this doesn't work nearly as well for user 
interfaces.  humans are a lot less predictable than machines (alas :)

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Company-Confidential indication
Date: Tue, 30 Nov 2004 20:42:40 -0500
Cc: Keith Moore <[email-masked]>, Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

A (possibly) interesting side-note to the issue of company-confidential
indications that Nat brought up: both VPIM and MIXER use a
Sensitivity message header field, however, while MIXER provides
for a "Company-Confidential" value for the field body, VPIM
does not.  Otherwise, the syntax and values appear identical.

So a message with
  Sensitivity: Company-Confidential
might be a legal MIXER message or a VPIM message with illegal
content; and it's not easy to determine which...

Cc: Nathaniel Borenstein <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 20:40:39 -0500
To: Steve Dorner <[email-masked]>
Sender: [email-masked]

>> it seems like there are better ways for lists to tag their contents 
>> and for MUAs to display information about list origin to recipients.
>
> I agree.  But I don't think lists are the only problem here.

More generally, the problem is that there are lots of things that think 
they should be able to demand the recipients' attention, and there are 
only two places they can reliably do so - the subject field and the 
message body.  And for this reason both of them commonly get altered.

Now let's say (as a thought experiment) that we had the luxury of 
changing how MUAs display messages.  Then we could define a couple of 
new fields:

"Display-Header-Fields:" field-name *("," field-name)

which would go in a message header, and would say to the MUA that the 
indicated field-names are worthy of display to the recipient in 
addition to whatever other fields were normally displayed.

"Content-Added-By:" identifier

which would go in a body part header, and would indicate that the 
content was not supplied by the message author but was instead added by 
some intermediary.  Content-Disposition fields could then indicate 
whether the intermediary thinks the additional body part should be 
displayed "inline", or made available as an attachment, or what.

This would make it possible for the recipient's MUA to separate 
annotations from original content.

Now the question is - would the intermediaries be willing to mark their 
annotations in such a way that the content could easily be separated?  
Some lists would, undoubtedly.  Maybe virus filters also (though I 
think they're more likely to remove content than to add it).  But what 
about things that add advertisements?

The transparency issue seems to crop up at every layer and in every 
popular application. It's as if there is a battle between the 
intermediaries and the endpoints.  Either the intermediaries think they 
are smarter or know better than the endpoints (as in the case of 
security firewalls and virus filters), or they think that they have the 
right to impose demands on the endpoints that involve changing the 
content exchanged by the endpoints (as in the case of "walled gardens" 
and "value added services" implemented on top of what were intended as 
end-to-end protocols).  In every case the result is a arms race where 
the intermediaries and endpoints both become more and more complex over 
time -- until the system as a whole becomes so expensive and/or 
dysfunctional that there is no longer sufficient incentive to fix the 
problems.

Keith

Cc: Nathaniel Borenstein <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 20:22:16 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

>> I think we need to distinguish between human-level conventions and
>> protocol elements.  Any "harmfulness" in the former is not something
>> that can be addressed technically.
>
> Agreed that we need to make the distinction.  The harmfulness
> arises when mindless mechanical processing subverts or otherwise
> interferes with the human communication by attempting to
> overload protocol keywords onto natural language text.

It's hardly unusual for computer programs to try to interpret natural 
language text.  For instance, programs that sort English-language book 
titles know to ignore case, an initial "The", most punctuation 
characters, etc.  We find it useful to write programs that can compare 
postal addresses and know when they're the same place even when they're 
not written identically.   Similarly, programs that can pick out 
telephone numbers, meatspace addresses, email addresses, and URIs from 
natural language text are quite useful even if they're imperfect.  So 
we shouldn't be surprised when programs try to make sense of things 
like "Re:" and "[listname]".  (but we would do well to avoid enshrining 
such conventions in protocols, because protocols don't evolve as easily 
as human conventions)

Humans want things to be mostly unstructured; computer programs want 
things to be highly structured.  Humans prefer loose definitions for 
data elements; computer programs prefer strict definitions.  And yet we 
want computer programs that deal with human natural language 
communications.  This seems like an inherent conflict that we're not 
likely to resolve.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 20:14:05 -0500
Cc: Steve Dorner <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

On Tue November 30 2004 18:58, Steve Dorner wrote:

> You can bet they NEVER edit the 
> References or In-Reply-To fields before they send the mail (a few 
> members of this mailing list excepted, no doubt).

How many UAs make it possible, let alone easy, for authors to
edit References or In-Reply-To fields?

For that matter, how many permit a user to select multiple
messages to compose a response citing all of the selected
messages (including appropriate setting of In-Reply-To and
References fields)?

This is probably an area where some advice to UA authors
is called for.

Cc: Steve Dorner <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 20:09:34 -0500
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

> The "[listname]" insertion is clearly the "sore point" case.  This is 
> natural because it seems to defy the otherwise reasonable-sounding 
> rule that i myself proposed, which is that the Subject field is for 
> human beings.  However, I'm not sure that this is really as bad a 
> layering violation as it might seem.  While the list processor is 
> indeed inserting something into the "human" part of the stream, it is 
> (at least in theory) doing so in accord with a policy set by a list 
> owner.  A list owner should be able to choose to insert any strings he 
> wants to in messages that go through his list, to convey a meaning 
> that he wants to convey.  The fact that list processors make it so 
> easy to insert such strings may be simply a reflection of the fact 
> that so many list owners want to be able to do that.  But they want it 
> for its human meaning, not for any intended protocol consequences.

I see (at least) two distinct kinds of badness associated with 
[listname] tags.  One is that they get in the way of searches, 
comparisons, display, etc. so there's a temptation to remove or ignore 
them just as with Re:, Fwd: and similar tags.

The other is the harm to transparency - these tags alter the message 
content from what the original author intended.  Footers added by lists 
cause the same kinds of problems.  This begs the question - is the 
purpose of the list to be a transparent multicast channel or is it an 
original source of content?  In many cases the same content that is 
submitted to a list is also transmitted to other recipients, who will 
think they're all receiving the same message - but they're not.

While a case can probably be made that at least a few lists should be 
content sources, IMHO lists that alter subject fields or message bodies 
should also alter other portions of the message to avoid confusion with 
the message as originally written by the author.  For instance, the 
message-id should be changed, in-reply-to and references fields should 
be cleared or altered, etc.

As for the "dog" analogy:  we can scarcely prevent any kind of protocol 
violation.  But we can at least point out that some practices are 
undesirable and try to discourage them.

Date: Tue, 30 Nov 2004 17:05:23 -0800
To: Keith Moore <[email-masked]>
From: Steve Dorner <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Cc: Nathaniel Borenstein <[email-masked]>, ietf-822 <[email-masked]>
Sender: [email-masked]

At 7:52 PM -0500 11/30/04, Keith Moore wrote:
>it seems like there are better ways for lists to tag their contents 
>and for MUAs to display information about list origin to recipients.

I agree.  But I don't think lists are the only problem here.

I think there are increasing numbers of proxies and filters that all 
would like to communicate with the recipient, and are prevented from 
doing so by MUA's that show only certain fields, period.

If a solution for lists generalizes to virus scanners, security 
software, etc, etc, good.  If it's just a special case for lists, I 
don't think we've solved the problem.
-- 
Where are we going, and why are we in this handbasket?

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 20:03:27 -0500
Cc: Nathaniel Borenstein <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

On Tue November 30 2004 19:16, Nathaniel Borenstein wrote:

> I think we need to distinguish between human-level conventions and 
> protocol elements.  Any "harmfulness" in the former is not something 
> that can be addressed technically.

Agreed that we need to make the distinction.  The harmfulness
arises when mindless mechanical processing subverts or otherwise
interferes with the human communication by attempting to
overload protocol keywords onto natural language text.

> When you choose the "Reply" function in your MUA, and it gives you a 
> window in which the subject has been initialized to "Re: <old 
> subject>," this is at best a "helpful suggestion" from your MUA, 
> because you can always change it.

"at best", as you say. It can also be viewed as an obnoxious
intrusion into human-to-human communication. Few users
bother to change it; therefore it has become mere noise.

> Arguing over the "right" way to do "Re:" or "Fwd:" or  
> "[ietf-822]" is about as technically useful as arguing over the right 
> greeting or salutation for an email message, because ultimately the 
> user controls it, not the software.

*If* the user controls it -- recall that the discussion started
regarding cruft added automatically by mailing list software
which is not under control of the message author(s).

Cc: Nathaniel Borenstein <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 19:52:15 -0500
To: Steve Dorner <[email-masked]>
Sender: [email-masked]

>> I think we need to distinguish between human-level conventions and 
>> protocol elements.  Any "harmfulness" in the former is not something 
>> that can be addressed technically.
>
> Ok, human bad, protocol element good.  That's quite clear.  Pesky 
> humans.

well, the human behavior is at least somewhat influenced by the 
interfaces they're using.  improved interfaces might produce better 
behavior independently of any changes to protocol elements or how they 
are used.

> Is [nameoflist] inserted by a list processor a human-level convention?
>
> If a machine puts it there, I think it's reasonable for another 
> machine to try to take it out.  If we as an Internet community can 
> make that easier, we will have done a good thing.

perhaps.  but while we might accept an arms race ("complexity race"?) 
as a short-term necessity,  it seems like there are better ways for 
lists to tag their contents and for MUAs to display information about 
list origin to recipients.

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 19:51:34 -0500
To: Steve Dorner <[email-masked]>
Sender: [email-masked]

The "[listname]" insertion is clearly the "sore point" case.  This is 
natural because it seems to defy the otherwise reasonable-sounding rule 
that i myself proposed, which is that the Subject field is for human 
beings.  However, I'm not sure that this is really as bad a layering 
violation as it might seem.  While the list processor is indeed 
inserting something into the "human" part of the stream, it is (at 
least in theory) doing so in accord with a policy set by a list owner.  
A list owner should be able to choose to insert any strings he wants to 
in messages that go through his list, to convey a meaning that he wants 
to convey.  The fact that list processors make it so easy to insert 
such strings may be simply a reflection of the fact that so many list 
owners want to be able to do that.  But they want it for its human 
meaning, not for any intended protocol consequences.

I would certainly be unhappy with a list processor that *required* the 
insertion of such strings, but when a list administrator chooses to 
have them inserted,this is the human moderator's decision to "put his 
mark" on all messages to the list.  Sort of like a dog urinating on a 
tree -- it may not always be pretty or desirable, but it's going to be 
awfully hard to design a tree that can prevent itself from being peed 
on.  -- Nathaniel

On Nov 30, 2004, at 7:39 PM, Steve Dorner wrote:

> At 7:16 PM -0500 11/30/04, Nathaniel Borenstein wrote:
>> I think we need to distinguish between human-level conventions and 
>> protocol elements.  Any "harmfulness" in the former is not something 
>> that can be addressed technically.
>
> Ok, human bad, protocol element good.  That's quite clear.  Pesky 
> humans.
>
> Is [nameoflist] inserted by a list processor a human-level convention?
>
> If a machine puts it there, I think it's reasonable for another 
> machine to try to take it out.  If we as an Internet community can 
> make that easier, we will have done a good thing.
> -- 
> Where are we going, and why are we in this handbasket?
>
>

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Subject field hacks and machine processing of "only human-readable information"
Date: Tue, 30 Nov 2004 19:50:36 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Tue November 30 2004 18:27, Keith Moore wrote:

> what's wrong with abbreviations?

Not much per se.

> why should I have to say "reply to  
> your message: Attempts at establishing harmful conventions" instead of 
> using "Re:"?

If this were a century ago and taking place via pieces
of paper, something like "Re: your correspondence of
30 November" might be appropriate.  For quite some
time there has been provision in the message
format for something similar (via the In-Reply-To
field, which until 2822 was permitted to contain a
phrase such as the above), However, it has been
unnecessary to do so in the Subject field (that is
redundant with In-Reply-To).  In the specific
instance cited, you could simply leave the Subject
field unaltered, or change it if the topic changes
(as I have done in this message).  Unlike century-old
paper correspondence, modern electronic messages
carry with them a set of references to previous
correspondence, making explicit reference in the
Subject field unnecessary.

> more to the point, why is it a bad idea for the subject  
> to indicate that the message is a reply or some other kind of response? 

It's redundant, as noted above. That alone is a
small flaw; it gets really bad when mindless
machine processing happens with the assumption
that the string "Re: " is *always* an indication
of a response or that it is *always* redundant cruft
added to indicate that a message is a response.

>   why is it a bad idea for the subject to give some indication of 
> message topic?

It's not; but unless we're discussing Rhenium, or
musical notes, etc. "Re" isn't the topic per se.

> where do you draw the line between what belongs in the  
> subject and what belongs elsewhere?

Personally I would draw it at the point where a
human author indicates a topic (either directly
in a message, or in canned boilerplate for an
automated response).  I draw the line there because
the Subject field is defined as containing "only
human-readable content with information about
the message", and to be consistent with the
architectural principle of end-to-end (in this
case human to human) communications.  Once
software starts mindlessly prepending cruft, other
software begins mindlessly removing particular
strings under the assumption that such strings
are always cruft.   And then the humans attempting
to communicate lose.
 
> I don't think this is cruft - I think it's useful information that 
> reasonably belongs in the subject field.  what's broken is something 
> else - perhaps our expectation that all messages in a thread should 
> have the same subject.

What's broken is the assumption that the Subject
field is structured with "Re: " being some sort of
magic token which always indicates a response.
As noted above it is unnecessary, as a response
is indicated by the presence of an In-Reply-To
field (and UA authors may write software that
takes that into account when displaying a message
or message list by indicating that a message is
a response).  The same applies to mailing list
messages where insertion of cruft into the Subject
field is redundant with the presence of List- fields.
Were I a chemist searching for messages whose
topic is element 75, I should be very frustrated
indeed.  The prevalence of the particular "Re"
hack is so bad that there have been drafts
proposing that all instances of "Re" and similar
content be discarded (recursively) when searching
Subject fields or collating messages by subject.
There are implicit assumptions that given a
message whose header contains
  Subject: Re: the element with atomic number 75 
that:
1. that message is a response to some other message
2. that the "real" topic is "the element with
    atomic number 75", i.e. that "Re: " is mere cruft
3. that somewhere there is some related message
    with
      Subject: the element with atomic number 75
In reality, none of those assumptions may be
correct.

Date: Tue, 30 Nov 2004 16:39:03 -0800
To: Nathaniel Borenstein <[email-masked]>
From: Steve Dorner <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
Sender: [email-masked]

At 7:16 PM -0500 11/30/04, Nathaniel Borenstein wrote:
>I think we need to distinguish between human-level conventions and 
>protocol elements.  Any "harmfulness" in the former is not something 
>that can be addressed technically.

Ok, human bad, protocol element good.  That's quite clear.  Pesky humans.

Is [nameoflist] inserted by a list processor a human-level convention?

If a machine puts it there, I think it's reasonable for another 
machine to try to take it out.  If we as an Internet community can 
make that easier, we will have done a good thing.
-- 
Where are we going, and why are we in this handbasket?

Cc: ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 19:33:47 -0500
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

> I think we need to distinguish between human-level conventions and 
> protocol elements.  Any "harmfulness" in the former is not something 
> that can be addressed technically. [...]

I dislike intermediaries that modify the subject line.  Otherwise, I'm 
substantially in agreement with your message.

and if we start asking users to fill in multiple fields that are 
descriptive of the messages they send - one for the topic, another for 
message attributes (reply, forward, confidential, etc) the instructions 
for trying to get users to "do the right thing" with those fields are 
going to look as if Lewis Carroll wrote them.

Cc: ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 19:28:55 -0500
To: Steve Dorner <[email-masked]>
Sender: [email-masked]

>> users are lazy - they will sometimes reply to a message just as an 
>> easy way to get the same recipient list, even if the topics are 
>> different.  often they don't even change the subject.
>
> Barring Bayesian analysis, I think we're SOL if they don't change the 
> subject.

I think we're SOL if user interfaces don't evolve.

Keith

Date: Tue, 30 Nov 2004 16:20:27 -0800
To: Keith Moore <[email-masked]>
From: Steve Dorner <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

At 7:10 PM -0500 11/30/04, Keith Moore wrote:
>users are lazy - they will sometimes reply to a message just as an 
>easy way to get the same recipient list, even if the topics are 
>different.  often they don't even change the subject.

Barring Bayesian analysis, I think we're SOL if they don't change the subject.

But if they do change the subject, and one slavishly follows IRT 
anyway, how is that A Good Thing?
-- 
Where are we going, and why are we in this handbasket?

Cc: ietf-822 <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: MoreOn: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 19:16:13 -0500
To: Keith Moore <[email-masked]>
Sender: [email-masked]

Distinguished and revered colleagues,

I think we need to distinguish between human-level conventions and 
protocol elements.  Any "harmfulness" in the former is not something 
that can be addressed technically.

When you choose the "Reply" function in your MUA, and it gives you a 
window in which the subject has been initialized to "Re: <old 
subject>," this is at best a "helpful suggestion" from your MUA, 
because you can always change it.  In the case of this particular 
message, for example, I decided to change it to "MoreOn" and no harm 
was done, unless someone chose to regard it as an insulting pun.  (And 
don't tell me that your mail reader needs "Re:" for message threading, 
because if your MUA can't get thread info from the In-Reply-To field, 
it isn't trying hard enough.)

Ultimately, the user-editability of "Subject" makes it almost valueless 
as a machine-usable protocol, but highly valuable for interpersonal 
conventions.  We should not be surprised, however, if different 
conventions arise among different subcultures, perhaps with subtly 
different semantics.  In fact, this happens all the time -- some email 
communities already have informal but quite complex norms for making 
their subjects "more useful" at the human level.  (I put "more useful" 
in quotes because they aren't always more useful -- at IBM, 
confidential messages often start with an "IBM Confidential and 
Proprietary" prefix that is so long as to eclipse all information in 
the subject, but it is a convention which is almost mandatory within 
the company.)  This is not a problem of protocol design -- it is simply 
a reality of evolving human norms and etiquette.

Thus, if you want to put data into a message that is intended to convey 
meaning reliably to a piece of software, you shouldn't put it in the 
Subject field.  Arguing over the "right" way to do "Re:" or "Fwd:" or 
"[ietf-822]" is about as technically useful as arguing over the right 
greeting or salutation for an email message, because ultimately the 
user controls it, not the software.  I have the right to greet you all 
as "distinguished and revered colleagues" if I'm so inclined, right?

Y.O.S.,

nsb

On Nov 30, 2004, at 6:27 PM, Keith Moore wrote:

>
> Bruce Lilly writes:
>
>>> I do think there's a significant difference between conventions for 
>>> the
>>> Subject field that the message author (or the author's MUA) uses to
>>> indicate information about the content, and alterations to the 
>>> Subject
>>> field that are done by intermediaries.
>>
>> There may be a distinction in the mind of a message author or
>> UA software author.  But once the message is sent, there's no
>> way to reliably differentiate "a short string identifying the topic
>> of the message" from any cruft that may have been added (as
>> demonstrated by the various "Re: " examples).
>
> what's wrong with abbreviations?  why should I have to say "reply to 
> your message: Attempts at establishing harmful conventions" instead of 
> using "Re:"?  more to the point, why is it a bad idea for the subject 
> to indicate that the message is a reply or some other kind of 
> response?  why is it a bad idea for the subject to give some 
> indication of message topic?  where do you draw the line between what 
> belongs in the subject and what belongs elsewhere?
>
> I don't think this is cruft - I think it's useful information that 
> reasonably belongs in the subject field.  what's broken is something 
> else - perhaps our expectation that all messages in a thread should 
> have the same subject.
>
> Keith
>
>
>

Cc: ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 19:10:13 -0500
To: Steve Dorner <[email-masked]>
Sender: [email-masked]

>> what's broken is something else - perhaps our expectation that all 
>> messages in a thread should have the same subject.
>
> Users often reply to mail in order to start a new thread with the same 
> set of folks in it.  Sometimes they change the subject when they do 
> this, sometimes they don't.  You can bet they NEVER edit the 
> References or In-Reply-To fields before they send the mail (a few 
> members of this mailing list excepted, no doubt).
>
> So if you don't consider subject when doing threads, you lose one 
> thing that users actually DO occasionally use to mark a new topic.

a changed subject is not a reliable indicator of a new thread.

just because the author of a reply changes the subject from that of the 
message being replied to doesn't mean that the reply is unrelated to 
the message being replied to, and it doesn't mean that recipients won't 
want to see the relationships between the messages.

users are lazy - they will sometimes reply to a message just as an easy 
way to get the same recipient list, even if the topics are different.  
often they don't even change the subject.    (but how many MUAs give 
users a better way to do this?)

Keith

Date: Tue, 30 Nov 2004 15:58:48 -0800
To: Keith Moore <[email-masked]>
From: Steve Dorner <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

At 6:27 PM -0500 11/30/04, Keith Moore wrote:
>what's broken is something else - perhaps our expectation that all 
>messages in a thread should have the same subject.

Users often reply to mail in order to start a new thread with the 
same set of folks in it.  Sometimes they change the subject when they 
do this, sometimes they don't.  You can bet they NEVER edit the 
References or In-Reply-To fields before they send the mail (a few 
members of this mailing list excepted, no doubt).

So if you don't consider subject when doing threads, you lose one 
thing that users actually DO occasionally use to mark a new topic.
-- 
Where are we going, and why are we in this handbasket?

Date: Tue, 30 Nov 2004 15:36:08 -0800
To: Peter Koch <[email-masked]>
From: Steve Dorner <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Cc: [email-masked]
Sender: [email-masked]

I just spoke to a fellow with a very interesting email proxy system 
that unfortunately works by using such []'ed cruft in subject fields. 
I suppressed my gag reflex, because I understood why he was doing it. 
Most current MUA's hide all headers except a specific few, so if you 
want to communicate anything to the end user, the only way you have 
to do it is by scribbling on those few visible fields.  Or in the 
body, which is even worse.  Gone are the days when you could add a 
header and expect anyone to see it.  So, list processors aside, I 
think we're going to see more of this and not less.

My MUA ignores such text when doing subject comparisons.

Re: foo
foo
Re[2]: foo
[mydumblist] foo

are all considered equal.  A further step that would probably be 
interesting would be to simply remove such strings from subjects 
displayed in message lists.

Could this be generalized as a recommendation to MUA's?  I dunno.  It 
seems like a feature that's useful a lot of the time, and harmful 
occasionally (eg, when there is real information inside []'s for some 
reason).

I personally wouldn't be opposed to amending 2822 to declare that [] 
are comment indicators in subject fields.  I know that sins against 
the holy writ, but oh well...
-- 
Where are we going, and why are we in this handbasket?

From: Keith Moore <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 18:27:55 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

Bruce Lilly writes:

>> I do think there's a significant difference between conventions for 
>> the
>> Subject field that the message author (or the author's MUA) uses to
>> indicate information about the content, and alterations to the Subject
>> field that are done by intermediaries.
>
> There may be a distinction in the mind of a message author or
> UA software author.  But once the message is sent, there's no
> way to reliably differentiate "a short string identifying the topic
> of the message" from any cruft that may have been added (as
> demonstrated by the various "Re: " examples).

what's wrong with abbreviations?  why should I have to say "reply to 
your message: Attempts at establishing harmful conventions" instead of 
using "Re:"?  more to the point, why is it a bad idea for the subject 
to indicate that the message is a reply or some other kind of response? 
  why is it a bad idea for the subject to give some indication of 
message topic?  where do you draw the line between what belongs in the 
subject and what belongs elsewhere?

I don't think this is cruft - I think it's useful information that 
reasonably belongs in the subject field.  what's broken is something 
else - perhaps our expectation that all messages in a thread should 
have the same subject.

Keith

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Tue, 30 Nov 2004 17:50:01 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Tue November 30 2004 12:06, Keith Moore wrote:

> For completeness we should also include the FTC's mandated
> SEXUALLY-EXPLICIT: tag.

Sure.  And the various proposals for "Adv", etc.

> I do think there's a significant difference between conventions for the
> Subject field that the message author (or the author's MUA) uses to 
> indicate information about the content, and alterations to the Subject
> field that are done by intermediaries.

There may be a distinction in the mind of a message author or
UA software author.  But once the message is sent, there's no
way to reliably differentiate "a short string identifying the topic
of the message" from any cruft that may have been added (as
demonstrated by the various "Re: " examples).  Once cruft gets
added, all the king's horses and all the king's men can't separate
cruft from topic.

Date: Tue, 30 Nov 2004 12:06:27 -0500
From: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Attempts at establishing harmful conventions
Sender: [email-masked]

> While we're bashing such harmful attempts at establishing
> ill-advised conventions, let's not forget the "Auto: " Subject
> field hack, which is in the same boat as the "Re: " hack which
> was being discussed.

For completeness we should also include the FTC's mandated
SEXUALLY-EXPLICIT: tag.

I do think there's a significant difference between conventions for the
Subject field that the message author (or the author's MUA) uses to 
indicate information about the content, and alterations to the Subject
field that are done by intermediaries.
--
Blessed are the billionaires, for they shall inherit without estate taxes.

To: LIST: [email-masked];
Xref: clerew local.mime:4206
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Tue, 30 Nov 2004 10:54:10 GMT
Lines: 37
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Mon November 22 2004 06:07, Charles Lindsey wrote:

>> Yes, but you are being somewhat selective in your quotations from RFC
>> 2822,

>Yes, RFC 2822 covers a lot of ground; I selected the appropriate
>parts concerning the defined syntax and semantics of the field
>under discussion.

>> because it _does_ give some sort of approval to the "Re:�" hack. 

>No, (as is sometimes the case where "MAY" is used) it states
>an obvious fact;  true the Subject field body may start with
>"Re: " -- it may also start with "Blurfl" or any other content
>consistent with the defined (viz. unstructured) syntax.  Once
>a message has been sent, there is no defined meaning (in a
>protocol sense) to any part of a Subject field; users (or UA
>authors) can make guesses, but guesses are sometimes wrong
>and play no role in a well-designed protocol.

But for some inexplicable reason the words "the field body may start with
the string "Blurfl" do not appear in RFC 2822, which suggests, to me at
least, that the string "Re:�" was intended to have some special
significance which does not apply to the string "Blurfl".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Attempts at establishing harmful conventions
Date: Mon, 29 Nov 2004 08:15:18 -0500
Cc: Charles Lindsey <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

On Mon November 22 2004 06:01, Charles Lindsey wrote:

> In <[email-masked]> Keith Moore <[email-masked]> writes:

> >is that anything like defining a new convention for group syntax?

> > I have already accepted that a new convention for group syntax was not a
> pretty solution to that particular problem

It isn't a solution; it is itself a problem (for several independent reasons).

> It still remains the only solution on the table 
> for picking the list addresses out of a message sent using a Reply-to-All
> feature, or for detecting cross-posts to multiple lists in the absence of
> any indication added by the sender.

First, there is no "table".  Second, comparing List-Post field content
to recipient address field content could be used to identify list
posting mailboxes (though it is by no means 100% reliable). That
is certainly a great deal better than using a common surname
as it it were a magic incantation turning a human-readable
word into a protocol element.

While we're bashing such harmful attempts at establishing
ill-advised conventions, let's not forget the "Auto: " Subject
field hack, which is in the same boat as the "Re: " hack which
was being discussed.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Mon, 29 Nov 2004 08:27:14 -0500
Cc: "Charles Lindsey" <[email-masked]>, LIST: [email-masked];
Sender: [email-masked]

On Mon November 22 2004 06:07, Charles Lindsey wrote:

> Yes, but you are being somewhat selective in your quotations from RFC
> 2822,

Yes, RFC 2822 covers a lot of ground; I selected the appropriate
parts concerning the defined syntax and semantics of the field
under discussion.

> because it _does_ give some sort of approval to the "Re:�" hack. 

No, (as is sometimes the case where "MAY" is used) it states
an obvious fact;  true the Subject field body may start with
"Re: " -- it may also start with "Blurfl" or any other content
consistent with the defined (viz. unstructured) syntax.  Once
a message has been sent, there is no defined meaning (in a
protocol sense) to any part of a Subject field; users (or UA
authors) can make guesses, but guesses are sometimes wrong
and play no role in a well-designed protocol.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Mon, 29 Nov 2004 08:03:20 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Fri November 19 2004 07:22, Charles Lindsey wrote:

> Generally speaking, meaningful tags in Subject fields

By definition, there are no "tags" in the Subject field; it is an
unstructured field.

> (specifically any 
> which may need to be globally recognized by MUAs)

The only content that *needs* to be recognized are illegal
content (when generating) and RFC 2047 encoded-words,
and only for two specific reasons:
1. to present content to humans in human-readable form
2. to comply with specific requirements when generating
    fields

> And yet software
> needs to recognize at least the "Re:�" form,

There is no such "need", and "Re: " has no defined
special significance in any unstructured field.  "Need"
implies some role in protocol; there is none in this case.
A role in protocol as implied by "need" further implies
that some protocol element (in a properly-designed
protocol) is unambiguous.  That is certainly not the
case with the "Re: " hack (which is why it is a "hack");
see below for some well-known examples.  There is a
defined, long-standing, standardized indication that
a message is a response to another message; that is
the presence of an In-Reply-To or References field.

> for purposes of presenting 
> lists of messages in a sensible order

Nonsense.  The Subject field has no inherent role in order
of presentation; sure, it can be used for collating messages
at the user's request, but "Re: " should play no specific role
in that process (unless specified by the user).  Otherwise,
Subjects such as:
  Subject: Re: the note between Do and Mi
  Subject: Re: the element with atomic number 75
  Subject: RE: Raoul Edwards strikes again!
  Subject: Re: the prefix signifying back, again, against, anew
  Subject: Re: the Egyptian sun god
etc. may be treated improperly (including by the methods
in the IMAP sorting extensions draft).  Likewise for
localized variants of "Re: ", such as "Ant: ".

> The USEFOR WG
> has taken the position that the only one worthy of any official
> recognition at all is a single "Re:�"

No, the USEFOR WG has not taken any such position. There
is a wide variety of opinion, and what is agreed is that the
"Re: " hack should not be considered part of the field syntax
or semantics (per RFC 2822 and predecessors).  Even that
straightforward issue took literally years to resolve, and there
are some (ahem) who still attempt to reverse that consensus-
based decision.

Cc: [email-masked]
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
To: [email-masked]
Date: Wed, 24 Nov 2004 09:06:30 +0000
From: Nick Ing-Simmons <[email-masked]>
Reply-To: Nick Ing-Simmons <[email-masked]>
Sender: [email-masked]

Peter Koch <[email-masked]> writes:
>Dear ``email people'',
>
>motivated by a discussion on the main IETF list a couple of weeks ago I
>just submitted this Internet Draft
>
>        Title           : Subject: [tags] Considered Harmful
>        Author(s)       : P. Koch
>        Filename        : draft-koch-subject-tags-considered-00.txt
>        Pages           : 8
>        Date            : 2004-11-18
>        
>Various mailing lists modify the Subject: header field of messages
>   sent to the list to contain a kind of identifier enclosed in square
>   brackets.  Every now and then this 'feature' is also requested for
>   the main IETF list.  This document collects pros and cons of this
>   approach, tries to identify the real issue and will, in a later
>   stage, try to give a recommendation.
>
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-koch-subject-tags-considered-00.txt

As others have said cluttering up suposed human readable Subject:
line with these things are a pain. e.g. there are a lot of mails 
to Alsa-devel list which have two tags:

Subject: [Alsa-devel] [ALSA - driver 0000659]: ...

It only needs a few Re: on front and no part of real subject is visible 
in my mailer's window.

>--
>
>The draft is just a request for discussion, I'm not sure which direction (if
>any at all) it will take. Those [tags] are a matter of fact and "banning" them
>might not lead us anywhere (well, we have BCPs, not WCPs), but collecting and
>documenting their drawbacks might still be helpful. One potential outcome could
>be a "standardized" way of handling them at the MUA level. However, that may
>come close to <http://www.mail-archive.com/[email-masked]/msg06215.html> "Drunk
>Driving on the Internet".
>
>Feedback as well as pointers to any previous discussion of this topic (none
>were found it in the archives of *this* list) are welcome!
>
>-Peter

To: LIST: [email-masked];
Xref: clerew local.mime:4201
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Mon, 22 Nov 2004 11:01:12 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> Generally speaking, meaningful tags in Subject fields (specifically any
>> which may need to be globally recognized by MUAs) are a Bad Thing. The
>> problem is that people regularly invent new conventions for same, 
>> usually
>> without full understanding of the impact elsewhere

>is that anything like defining a new convention for group syntax?

I have already accepted that a new convention for group syntax was not a
pretty solution to that particular problem, and repeatedly asked for
suggested alternatives. It still remains the only solution on the table
for picking the list addresses out of a message sent using a Reply-to-All
feature, or for detecting cross-posts to multiple lists in the absence of
any indication added by the sender.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4202
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Mon, 22 Nov 2004 11:07:08 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>The draft doesn't mention two very important considerations:

>1. the defined semantics of the field (RFC 2822):

>   These three fields are intended to have only human-readable content
>   with information about the message.  The "Subject:" field is the most
>   common and contains a short string identifying the topic of the
>   message.

>Added cruft identifying some list, etc. does not conform to the
>defined semantics ("identifying the topic of the message").

Yes, but you are being somewhat selective in your quotations from RFC
2822, because it _does_ give some sort of approval to the "Re:�" hack.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Cc: ietf-822 list <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Sat, 20 Nov 2004 12:09:35 -0500
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

> Generally speaking, meaningful tags in Subject fields (specifically any
> which may need to be globally recognized by MUAs) are a Bad Thing. The
> problem is that people regularly invent new conventions for same, 
> usually
> without full understanding of the impact elsewhere

is that anything like defining a new convention for group syntax?

To: LIST: [email-masked];
Xref: clerew local.mime:4190
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Fri, 19 Nov 2004 12:22:58 GMT
Lines: 69
Sender: [email-masked]

In <[email-masked]> Peter Koch <[email-masked]> writes:

>        Title           : Subject: [tags] Considered Harmful
>        Author(s)       : P. Koch
>        Filename        : draft-koch-subject-tags-considered-00.txt
>        Pages           : 8
>        Date            : 2004-11-18
>        
>Various mailing lists modify the Subject: header field of messages
>   sent to the list to contain a kind of identifier enclosed in square
>   brackets.  Every now and then this 'feature' is also requested for
>   the main IETF list.  This document collects pros and cons of this
>   approach, tries to identify the real issue and will, in a later
>   stage, try to give a recommendation.

>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-koch-subject-tags-considered-00.txt
>--

Hmmm! This is a minefield.

Generally speaking, meaningful tags in Subject fields (specifically any
which may need to be globally recognized by MUAs) are a Bad Thing. The
problem is that people regularly invent new conventions for same, usually
without full understanding of the impact elsewhere, and some of these
conventions then become widely used, and even find their way into
standards.

The best known example is the "Re:�" hack, which is so widely used and
recognized now that there is no turning back from it. The problem is that
it has also spawned variants such as "Re[2]:�", "Re:�Re:�" and "Sv:�", and
nobody knows which ones are 'official' and which not. And yet software
needs to recognize at least the "Re:�" form, for purposes of presenting
lists of messages in a sensible order (because you cannot rely on
threading by the References header field alone to do the whole job).

Much the same applies to "Fwd:�" (which again has variant forms), and
there is am IMAP draft <draft-ietf-imapext-sort-17.txt> which seeks to
codify some arbitrary selection of such conventions (including some of
the form "[...]"). Personally, I think this is a Bad Thing. The USEFOR WG
has taken the position that the only one worthy of any official
recognition at all is a single "Re:�", and that even that usage might be
better it it were phased out (some hope!).

Within the context of a mailing list (essentially a closed community),
there _might_ be some justification for the convention discussed in this
draft, but it still leads to many complications, not least the fact that a
message cross-posted to two mailing lists will have a different Subject in
each list, which is going to lead to threading problems if you manage to
get followups also crossposted to both lists (a desirable aim, but very
difficult to achieve, as discussed here recently in other threads).

There are a lot of problems with mailing list (again see recent threads
here) and essentially what is needed is a BCP document (perhaps with a
standard-track companion if it turns out new headers such as MFT are also
needed) to establish just what mailing list practice should be. The
question of these header tags could well have a place in such a document.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 19 Nov 2004 20:32:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 19.11.04 in <[email-masked]>:

> > Here is a Sieve language snippet for
> > identifying an ietf-822 list message:
> >
> > if anyof (address :contains ["From", "Sender", "Return-Path", "To",
> > "CC", "Bcc"] "ietf-822", header :contains ["List-Post", "List-ID"]
> > "ietf-822") {
> >         fileinto "INBOX.IETF-822";
> >         stop;
> > }
>
> arrgh.  if this is sufficient for your purposes, great - but it's not a
> good idea in general to recommend using header fields for this purpose.
> it will not do the right thing with forwarded or resent messages or
> messages sent to multiple lists to which you are subscribed.
>
> you really want to file things based on envelope information, like MAIL
> FROM or (with subaddresses) RCPT TO.

My personal experience is that, no, I do *NOT* want to sort on envelope in  
preference to List-Id - envelope information, in my experience, is  
significantly less reliable than List-Id (if there is a List-Id - and a  
pox on lists where there isn't!).

Any use of the address the list uses to reach you can have any amount of  
spam coming in, not to mention legitimate answers, completely independent  
of if you even used it in a list context or just forgot to change it to  
whatever other context you mistakenly used it in. None of those will have  
the List-Id: set, however.

And then think of all the people - still the vast majority, I suspect -  
who *can't* use list-specific addresses.

Or did you mean envelope-*from* (Return-Path:)? That breaks even more  
easily. A simple list-side mail system change that's completely unrelated  
to the list might change that!

There's a reason we have List-Id:.

I do agree that other headers are less useful - I just include envelope  
fields in that category.

MfG Kai

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Fri, 19 Nov 2004 23:08:38 +0100
Lines: 34
OpenPGP: id=0xB565716F; url=http://josefsson.org/key.txt
Cancel-Lock: sha1:FvIQ0zH5+dYU5lgaY7h52cWJMa4=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> > > if anyof (address :contains ["From", "Sender", "Return-Path", "To",
>> > > "CC", "Bcc"] "ietf-822", header :contains ["List-Post", "List-ID"]
>> > > "ietf-822") {
>> > >         fileinto "INBOX.IETF-822";
>> > >         stop;
>> > > }
>  
>> > it will not do the right thing with forwarded or resent messages or 
>> > messages sent to multiple lists to which you are subscribed.
>> 
>> That depends on one's definition of "the right thing".
>
> true.  but I suspect it doesn't even fit _your_ definition of the right 
> thing.
>
> for instance, if the same message is sent to both ietf-822 and ietf-smtp,
> do you really want them both filed into the same mailbox?

Cross-posting duplicates seem to be a subjective matter, so arguable,
some people could actually want that behaviour.

However, in this case, I suspect the server suppresses duplicates by
default, so there would not be two copies.

Even if there were two copies on the server, the MUA could be
suppressing duplicates, so the user would only read it once.

I'm not sure there is an objective "right thing" when it comes to
issues like how to manage your e-mail.

Regards,
Simon

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Fri, 19 Nov 2004 16:19:30 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Fri November 19 2004 15:31, Keith Moore wrote:

> true.  but I suspect it doesn't even fit _your_ definition of the right 
> thing.
> 
> for instance, if the same message is sent to both ietf-822 and ietf-smtp,
> do you really want them both filed into the same mailbox?

Given that there is a single (logical) message, there are several
possibilities that one might consider:

A. Whether or not to suppress received duplicates
   1. before filing
   2. per-"folder", after filing

B. Whether or not to make copies (or logical links, saving
    storage space) available via multiple "folders".

C. A hierarchy of filing for such cases.

D. Whether to program, test, debug, repeat for a script to
    handle the various possibilities, or to handle simple cases
    and deal with unusual situations by hand.

I use C and handle unusual cases manually.  If better tools
were available for testing and managing sieve scripts, I
might entertain adding complexity to the script to handle
unusual conditions such as you describe.  Given the state
of such tools, I (intentionally) have not done so [indeed,
until relatively recently, filing was handled by MUA (IMAP
client) filters which moved messages from the catch-all
INBOX to specific folders; I changed to a server-based
Sieve script to avoid dependence on a particular UA (or
having to replicate the filters in all UAs), but only with
some trepidation and after an unsuccessful search for
sieve script validation/testing/management tools].

Date: Fri, 19 Nov 2004 15:36:40 -0500
From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Sender: [email-masked]

> > you really want to file things based on envelope information, like MAIL
> > FROM or (with subaddresses) RCPT TO.
> 
> Using List-ID for this purpose seem perfectly fine to me.  Here's my
> Sieve snippet for this list [1]:
> 
> if header :contains "List-Id" "ietf-822.imc.org" {
> 	fileinto "INBOX.rfc822";
> }

that (or using return-path) doesn't fail as often.  it doesn't
necessarily do what you want if someone "resends" you a message from
the list - but that's not a very common case.

Date: Fri, 19 Nov 2004 15:31:23 -0500
From: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Sender: [email-masked]

> > > if anyof (address :contains ["From", "Sender", "Return-Path", "To",
> > > "CC", "Bcc"] "ietf-822", header :contains ["List-Post", "List-ID"]
> > > "ietf-822") {
> > >         fileinto "INBOX.IETF-822";
> > >         stop;
> > > }
 
> > it will not do the right thing with forwarded or resent messages or 
> > messages sent to multiple lists to which you are subscribed.
> 
> That depends on one's definition of "the right thing".

true.  but I suspect it doesn't even fit _your_ definition of the right 
thing.

for instance, if the same message is sent to both ietf-822 and ietf-smtp,
do you really want them both filed into the same mailbox?

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Fri, 19 Nov 2004 14:02:47 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Fri November 19 2004 12:28, Keith Moore wrote:
> > Here is a Sieve language snippet for
> > identifying an ietf-822 list message:
> > 
> > if anyof (address :contains ["From", "Sender", "Return-Path", "To",
> > "CC", "Bcc"] "ietf-822", header :contains ["List-Post", "List-ID"]
> > "ietf-822") {
> >         fileinto "INBOX.IETF-822";
> >         stop;
> > }
> 
> arrgh.  if this is sufficient for your purposes, great - but it's not a
> good idea in general to recommend using header fields for this purpose. 

It's sufficient for me.  It handles messages sent to me from the
list expander, copies of messages sent to the list, periodic subscription
notices regarding the list, etc.

> it will not do the right thing with forwarded or resent messages or 
> messages sent to multiple lists to which you are subscribed.

That depends on one's definition of "the right thing".
 
> you really want to file things based on envelope information, like MAIL
> FROM or (with subaddresses) RCPT TO.

By the time the particular sieve implementation gets ahold of a
message, sender envelope information has already been placed
in a Return-Path field and envelope recipient information is
constant (and irrelevant).  Sieve can (in general) use
envelope information (as described in RFC 3028); it's simply
not useful in the particular case from which the example was
taken.

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Fri, 19 Nov 2004 19:06:46 +0100
Lines: 41
OpenPGP: id=0xB565716F; url=http://josefsson.org/key.txt
Cancel-Lock: sha1:5VR5l1wT1s9nJvy8K1zzb7lzqaE=
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> Here is a Sieve language snippet for
>> identifying an ietf-822 list message:
>> 
>> if anyof (address :contains ["From", "Sender", "Return-Path", "To",
>> "CC", "Bcc"] "ietf-822", header :contains ["List-Post", "List-ID"]
>> "ietf-822") {
>>         fileinto "INBOX.IETF-822";
>>         stop;
>> }
>
> arrgh.  if this is sufficient for your purposes, great - but it's not a
> good idea in general to recommend using header fields for this purpose. 
> it will not do the right thing with forwarded or resent messages or 
> messages sent to multiple lists to which you are subscribed.
>
> you really want to file things based on envelope information, like MAIL
> FROM or (with subaddresses) RCPT TO.

Using List-ID for this purpose seem perfectly fine to me.  Here's my
Sieve snippet for this list [1]:

if header :contains "List-Id" "ietf-822.imc.org" {
	fileinto "INBOX.rfc822";
}

I agree that using any other header would be unreliable.

The problem with writing Sieve rules with envelope tests, is that the
envelope information might not always be available.  So if you ever
want to re-spool your mail through a Sieve script later, writing the
tests for header information, rather than envelope information, may be
more reliable.  If I understood correctly, Sieve was not designed
exclusively for initial delivery filtering.

Regards,
Simon

[1] I don't subscribe to the list these days, since I read through
gmane.org, but that's beside the point.

Date: Fri, 19 Nov 2004 12:28:02 -0500
From: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Sender: [email-masked]

> Here is a Sieve language snippet for
> identifying an ietf-822 list message:
> 
> if anyof (address :contains ["From", "Sender", "Return-Path", "To",
> "CC", "Bcc"] "ietf-822", header :contains ["List-Post", "List-ID"]
> "ietf-822") {
>         fileinto "INBOX.IETF-822";
>         stop;
> }

arrgh.  if this is sufficient for your purposes, great - but it's not a
good idea in general to recommend using header fields for this purpose. 
it will not do the right thing with forwarded or resent messages or 
messages sent to multiple lists to which you are subscribed.

you really want to file things based on envelope information, like MAIL
FROM or (with subaddresses) RCPT TO.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Fri, 19 Nov 2004 10:56:28 -0500
Cc: Peter Koch <[email-masked]>
Sender: [email-masked]

On Thu November 18 2004 16:42, Peter Koch wrote:
> 
> Dear ``email people'',
> 
> motivated by a discussion on the main IETF list a couple of weeks ago I
> just submitted this Internet Draft
> 
>         Title           : Subject: [tags] Considered Harmful
>         Author(s)       : P. Koch
>         Filename        : draft-koch-subject-tags-considered-00.txt
>         Pages           : 8
>         Date            : 2004-11-18

The draft doesn't mention two very important considerations:

1. the defined semantics of the field (RFC 2822):

   These three fields are intended to have only human-readable content
   with information about the message.  The "Subject:" field is the most
   common and contains a short string identifying the topic of the
   message.

Added cruft identifying some list, etc. does not conform to the
defined semantics ("identifying the topic of the message").

2. the defined syntax of the field (RFC 2822 again):

   The "Subject:" and "Comments:" fields are
   unstructured fields as defined in section 2.2.1, and therefore may
   contain text or folding white space.

   subject         =       "Subject:" unstructured CRLF

   Some field bodies in this standard are defined simply as
   "unstructured" (which is specified below as any US-ASCII characters,
   except for CR and LF) with no further restrictions.  These are
   referred to as unstructured field bodies.  Semantically, unstructured
   field bodies are simply to be treated as a single line of characters
   with no further processing (except for header "folding" and
   "unfolding" as described in section 2.2.3).

Aside from lack of necessity for groping through a Subject field
in a misguided attempt to find information unrelated to the topic
of the message (see below), unstructured fields (as noted above)
are intended to convey human-readable content, not machine-
parsed data.  Attempting to impose structure on an unstructured
field is inappropriate.  And section 2.7 of the draft illustrates the
folly of trying to base machine processing decisions on text
(in the RFC 2277 sense of "text") intended as "only human-readable"
(bearing in mind the fact that RFC 2047 encoded-words may only
appear in a subset of non-protocol, human-readable portions of
header field bodies).

Draft section 3 (identify a list message; three indistinct variants)
can easily and reliably be accomplished using envelope information
and List- message header fields, either with a Sieve (RFC 3028)
filter or a similar mechanism.  Here is a Sieve language snippet for
identifying an ietf-822 list message:

if anyof (address :contains ["From", "Sender", "Return-Path", "To", "CC", "Bcc"] "ietf-822", header :contains ["List-Post", "List-ID"] "ietf-822") {
        fileinto "INBOX.IETF-822";
        stop;
}

For some insight into another reason why adding cruft to the
Subject field is a problem, consider the contortions necessary
to remove such cruft for the purpose of getting to the actual
topic-identifying content, as discussed in
draft-ietf-imapext-sort-17.txt.

To: [email-masked]
Subject: New I-D: draft-koch-subject-tags-considered-00.txt
Date: Thu, 18 Nov 2004 22:42:48 +0100
From: Peter Koch <[email-masked]>
Sender: [email-masked]

Dear ``email people'',

motivated by a discussion on the main IETF list a couple of weeks ago I
just submitted this Internet Draft

        Title           : Subject: [tags] Considered Harmful
        Author(s)       : P. Koch
        Filename        : draft-koch-subject-tags-considered-00.txt
        Pages           : 8
        Date            : 2004-11-18
        
Various mailing lists modify the Subject: header field of messages
   sent to the list to contain a kind of identifier enclosed in square
   brackets.  Every now and then this 'feature' is also requested for
   the main IETF list.  This document collects pros and cons of this
   approach, tries to identify the real issue and will, in a later
   stage, try to give a recommendation.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-koch-subject-tags-considered-00.txt
--

The draft is just a request for discussion, I'm not sure which direction (if
any at all) it will take. Those [tags] are a matter of fact and "banning" them
might not lead us anywhere (well, we have BCPs, not WCPs), but collecting and
documenting their drawbacks might still be helpful. One potential outcome could
be a "standardized" way of handling them at the MUA level. However, that may
come close to <http://www.mail-archive.com/[email-masked]/msg06215.html> "Drunk
Driving on the Internet".

Feedback as well as pointers to any previous discussion of this topic (none
were found it in the archives of *this* list) are welcome!

-Peter

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: a header authentication scheme
Date: Wed, 17 Nov 2004 21:38:04 -0500
Sender: [email-masked]

On Wed November 17 2004 03:34, Laird Breyer wrote:

> If T2 is located on a LAN, would this require an open
> relay or is this not needed?

In general, an open relay is not necessary.  Since your
model presumes that T2 (etc.) handle messages destined
for the domain associated with B, open relays are not an
issue since the message is destined for B regardless of its
origin (an open relay applies when the server in question
is associated with neither the domain of the message origin
nor the domain of its destination; since the message
destination is appropriate, one of the two necessary
conditions for an "open relay" situation is not present).

> Also, can the very last SMTP server which 
> accepts the message on behalf of B be forced to *not* insert its own
> Received line?

If T2 transfers by some means other than SMTP, M3 may
very well *be* the last SMTP server involved (if indeed it uses
SMTP).

> I must admit that I was under the impression that at least that case
> is guaranteed to be unforgeable (ie if the message was transported
> through SMTP and none of the subsequent MDAs alter the message
> headers, then at least one Received line exists and the top Received
> line was written by the last SMTP server along the path).

There are many assumptions implicit in that statement, which
might not reflect reality for some specific cases.  For example,
some MDAs do insert Received fields such as:

Received: from mail.blilly.com ([unix socket])
	by marty (Cyrus v2.2.3) with LMTP; Wed, 17 Nov 2004 10:15:56 -0500

That was inserted by a Cyrus LMTP MDA process when making
"final" delivery to a mailbox.  Other MDAs may also alter message
header fields (e.g. sendmail), some auxiliary filtering processes
associated with an MDA may alter fields, and some MUAs are also
known to alter message headers.  Messages can be transferred
without SMTP (e.g. on LANs not using Internet mail protocols,
via uucp, etc.).  Received fields might be added by transfer
processes which do not involve SMTP (e.g. sendmail when
transferring via uucp).  So it's possible to have a message (w/
or w/o Received fields) which was transferred without use of
SMTP, and it's possible to have a message which has not
been transferred using SMTP contain one or more Received
fields.

> Even a "web of trust" alternative as used by PGP must still be trusted on
> arbitrary grounds, and compared to a hierarchical system it's more
> difficult for the web of trust to scale as I understand it.
[...] 
> Out of band verification such as with the telephone won't work unless
> there's already an out of band relationship to start with. If all you
> have is information within the message, you can't verify much.
> Requiring an existing relationship means this type of verification is
> not scalable to the internet, as much email communication also
> initiates new relationships.

Something like "DomainKeys" may be scalable, and does not
require a pre-existing relationship.  But now we're getting
somewhat off-topic.

> I don't see that. Since trust is entirely a quality originating from
> B, the amount of trust attributed to "T" (by B obviously) can be any
> shade of gray as well as dependent on time or message content, once B
> gets to see the message and analyse it. Of course, B probably applies
> simple heuristics wherein "T" is trusted regardless of content or time
> of day as you claim, but it's not a necessity.

As you may recall, the discussion that led to that was about
the appropriateness of having a non-filtering remote "filter"
if a local filter is required to determine whether the remote
"filter"'s assessment is reliable.  If such a local filter is required
(because one needs to analyze content in order to determine
whether or not to use a remote "filter" value judgment, etc.),
that situation applies.  Now, that isn't the case if and only if
a remote judgment's reliability is independent of message
content, etc.   With anything other than a content-invariant
black/white reliability, we get back to the inherent problems
with the proposed "processed" field -- there's no indication
of what parameters went into the claimed value judgment. 

Date: Wed, 17 Nov 2004 18:34:03 +1000
From: Laird Breyer <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: a header authentication scheme
Reply-To: [email-masked]
Mail-Followup-To: ietf-822 <[email-masked]>
Sender: [email-masked]

On Nov 13 2004, Bruce Lilly wrote:

> 
> > Arguing that an intermediate may exist which is trusted by the
> > recipient but, whether intentionally or otherwise, doesn't deserve
> > this trust, is not very profound. So what?
> 
> So the recipient trusts what is purported to be claimed by that
> site! To return to your very simplistic model:
> 
>     A -> M1 -> M2 -> M3 -> ... -> T1 -> T2 -> ... -> B
> 
> Site M3 can forge a Received field and corresponding "processed"
> field asserting insertion at T1, then bypass T1 by sending to T2
> (using a source route, using non-SMTP transport, etc.).  The
> recipient has no way of determining what has happened, and
> trusts what M3 has inserted (which purports to be from T1).

> Per your model and the scenario above, M3 is in your "untrusted"
> category, but the "incorrect" fields inserted by it are trusted.
> Q.E.D.

Well done, now we're getting somewhere! Source routing is very
interesting.  If T2 is located on a LAN, would this require an open
relay or is this not needed? Also, can the very last SMTP server which
accepts the message on behalf of B be forced to *not* insert its own
Received line?

> [...] because one cannot determine which content is inserted
> by any particular SMTP server (short of wrapping and signing at
> each stage during transport as suggested earlier in this discussion).

I must admit that I was under the impression that at least that case
is guaranteed to be unforgeable (ie if the message was transported
through SMTP and none of the subsequent MDAs alter the message
headers, then at least one Received line exists and the top Received
line was written by the last SMTP server along the path).

I'll respond to your other very interesting message separately, below
are just some minor points.

> 
> You're assuming a particular model which is not applicable to
> all signatures (it may apply to S/MIME, but not to PGP/MIME,
> since PGP/MIME does not use certificates).  

Even a "web of trust" alternative as used by PGP must still be trusted on
arbitrary grounds, and compared to a hierarchical system it's more
difficult for the web of trust to scale as I understand it.
 
> Moreover if one
> receives a message purporting to be from the XYZ company
> but which is signed by a key associated with Joe Spammer, the
> fact of that mismatch indicates that something is awry (it
> might be due to modification in transit, or due to a forgery at
> the source).

Out of band verification such as with the telephone won't work unless
there's already an out of band relationship to start with. If all you
have is information within the message, you can't verify much.
Requiring an existing relationship means this type of verification is
not scalable to the internet, as much email communication also
initiates new relationships.

> 
> > I'm saying that people may elect to trust certain mail transport
> > agents. There's no difference between that and electing to trust
> > VeriSign say.
> 
> Perhaps true, but irrelevant (except for the fact that such trust
> may lead to problems in either case).

You're arguing from the classic risk averse point of view. Other
points of view are risk neutral and risk seeking, in which the 
potential problems are outweighed by other perceived benefits.
 
> Your proposition as stated is "we always have a model of message
> transport as follows: [...] M represents computers which are
> untrusted by B, and T represents computers which are trusted by B",
> i.e. that all hosts involved in message transport are either always
> "T" (regardless of message content, invariant over time) or
> always "M"; it has no provision for degree of trust (as opposed
> to a binary "trusted"/"untrusted" distinction) -- it is a black-and-white
> model that has no provision for any shades of gray or other colors.

I don't see that. Since trust is entirely a quality originating from
B, the amount of trust attributed to "T" (by B obviously) can be any
shade of gray as well as dependent on time or message content, once B
gets to see the message and analyse it. Of course, B probably applies
simple heuristics wherein "T" is trusted regardless of content or time
of day as you claim, but it's not a necessity.

-- 
Laird Breyer.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: a header authentication scheme
Date: Sat, 13 Nov 2004 12:36:16 -0500
Sender: [email-masked]

On Sun November 7 2004 22:47, Laird Breyer wrote:

> On Nov 04 2004, Bruce Lilly wrote:
> 
> > > So we always have a model of message transport as follows:
> > > 
> > > A -> M1 -> M2 -> M3 -> ... -> T1 -> T2 -> ... -> B
> > 
> > Not necessarily.  There may well be multiple paths to B, N-1
> > of which do not pass through "T2".  "T1", "T2", etc. might
> > pass some messages w/o change, but may change some
> > fraction of messages (i.e. those messages not sympathetic
> > to the interests of the owner/operators of those systems).
> 
> I didn't expect this picture to specify a single path from A to B
> which must always be used by all messages from A to B, rather it
> represents the actual path taken by a given message. "T1" and "T2" are
> whatever makes sense for that actual path. My assumption which can be
> challenged is that all paths to B are covered by a small number of transport
> agents which can be reasonably labeled by B as being of type T.

Here is another scenario, based on real-world observed behavior
(http://lists.sans.org/pipermail/list/2002-September/054073.html)
which demonstrates that remote filtering (and the proposed cruft
added by remote "filters") is incompatible with existing Internet
mail architecture.  Understanding the scenario requires understanding
basic MIME principles (RFCs 1344, 2046) and an understanding of
the roles of gateways, a basic understanding of the role of the DNS
in mail transport, plus knowledge of some basic graph theory.

A message may be split for transport into fragments using the
MIME media type message/partial. Some types of gateways,
specifically gateways that scan content (for viruses, for spam
detection, etc.), require access to a complete message. Message
fragments may take diverse paths enroute to a recipient; a
complete message may be obtained by reassembling fragments
only at a site which is common to the paths of all fragments.
>From a graph topology point of view of the network, the host
node performing the gateway function must be on one side of
an edge which is a bridge edge (i.e. an edge which uniquely
connects two graph components which would not be connected
components without that edge) in the directed graph which
describes potential message transport paths from sender to
recipient.  Mail transport uses MX DNS records, and a minimum
of two independent MX hosts must be specified.  Therefore any
bridge edge must be located on the recipient side of all MX host
nodes for a given domain (from a theoretical point of view, it
would be possible to have a bridge edge attached to some
common node away from the recipient side of the MX host
nodes in some network topologies; aside from such a bridge
node located near the message sender (which is uninteresting
for the scenario of message reassembly for scanning) such
topologies are not representative of the Internet, and in such
topologies the MX hosts would not be truly independent).  Any
gateway activities which require message scanning must
therefore reside on the recipient side of the recipient domain's
MX hosts, and is therefore "local" rather than "remote" for all
practical purposes.

Fragmented message reassembly incorporates some fields
from the header of the first fragment and some fields from
the encapsulated message header which is conveyed in that
first fragment.  No transport-added (or any other) fields from
any other fragments are preserved by the reassembly process.
If -- contrary to RFC 1344 and the Internet mail architecture --
scanning of message fragments (rather than complete
messages) is performed, there are several predictable results,
none favorable for proponents of "remote" scanning:
1. the content which defines a message (e.g. as virus-laden,
    or as "spam") might not be present in the first fragment;
    it also might not appear in its entirety in *any* fragment.
2. reassembly involves discarding the message headers of all
    fragments but the first, and many fields from the first
    fragment header; any information conveyed in any of
    those discarded fields is irretrievably lost on reassembly.
    That includes any commentary on defining content which
    might appear in those fragments.
3. failure of gateways to take responsibility for all gateway
    functions WILL be exploited by bad guys (as has been the
    case for some products as noted in the CERT advisory
    mentioned in the linked message referenced above).

The bottom line is that there are some things that simply
cannot be done within the Internet mail architecture;
misguided efforts to break the rules usually backfire. "Remote"
scanning (i.e. inconsistent with the functions of gateways,
which necessarily must occur near the recipient) is such a
misguided effort which *has* backfired.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: a header authentication scheme
Date: Sat, 13 Nov 2004 12:50:57 -0500
Sender: [email-masked]

On Wed November 10 2004 21:50, Laird Breyer wrote:
> On Nov 10 2004, Bruce Lilly wrote:

> > No, that would be inaccurate. Message content can be signed
> > (at the originating endpoint, and the signature verified at the
> > receiving endpoint) to provide protection against undetected
> > modification in transit.
> 
> The message signature could be illegally removed in transit, and the
> body appended with some innocuous sentence such as "P.S. My PGP is
> temporarily broken", along with other subtle changes.

That would be a detectable modification, except in a statistically
insignificant number of cases (hash method collisions).  It could
be confirmed (as intentional content provided by the sender, or
as a forgery or modification) by an out-of-band method (e.g. a
telephone conversation).

> The message could be modified and resigned with another private key
> under the purview of the same certificate authority.

You're assuming a particular model which is not applicable to
all signatures (it may apply to S/MIME, but not to PGP/MIME,
since PGP/MIME does not use certificates).  Moreover if one
receives a message purporting to be from the XYZ company
but which is signed by a key associated with Joe Spammer, the
fact of that mismatch indicates that something is awry (it
might be due to modification in transit, or due to a forgery at
the source).

> You simply cannot claim that message modifications are detectable
> without some measure of trust in external entities separate from the
> sender and receiver.

Sure I can; modifications are detectable (statistically insignificant
caveat noted above) with zero trust in third parties.  It is in
fact blind trust in third parties that may lead to problems.  Failure
to trust third parties will never result in failure to detect
modification.

> I'm saying that people may elect to trust certain mail transport
> agents. There's no difference between that and electing to trust
> VeriSign say.

Perhaps true, but irrelevant (except for the fact that such trust
may lead to problems in either case).

> > And as I pointed out:
> > > > You are assuming an extraordinarily simplistic model which
> > > > simply doesn't conform to reality; a model where there are
> > > > only "M" and "T" and where each system is either one or the
> > > > other, invariant with message content and over time, with no
> > > > message transport paths through other mechanisms.
> 
> You persist in addressing a non-issue. The sentence preceding the one you
> quoted states
> 
>        I didn't expect this picture to specify a single path from A to
>        B which must always be used by all messages from A to B, rather it
>        represents the actual path taken by a given message. "T1" and
>        "T2" are whatever makes sense for that actual path.

It's an issue because your model assumes that trust is dependent
only upon host name, and is invariant over time and with different
message content, and that it is independent on all other sites that
may be involved in message transport.

> My proposition isn't, I don't know which birds you are metaphorically 
> referring to.

Your proposition as stated is "we always have a model of message
transport as follows: [...] M represents computers which are
untrusted by B, and T represents computers which are trusted by B",
i.e. that all hosts involved in message transport are either always
"T" (regardless of message content, invariant over time) or
always "M"; it has no provision for degree of trust (as opposed
to a binary "trusted"/"untrusted" distinction) -- it is a black-and-white
model that has no provision for any shades of gray or other colors.

> Arguing that an intermediate may exist which is trusted by the
> recipient but, whether intentionally or otherwise, doesn't deserve
> this trust, is not very profound. So what?

So the recipient trusts what is purported to be claimed by that
site! To return to your very simplistic model:

    A -> M1 -> M2 -> M3 -> ... -> T1 -> T2 -> ... -> B

Site M3 can forge a Received field and corresponding "processed"
field asserting insertion at T1, then bypass T1 by sending to T2
(using a source route, using non-SMTP transport, etc.).  The
recipient has no way of determining what has happened, and
trusts what M3 has inserted (which purports to be from T1).

> Show me a counterexample 
> where the intermediate isn't trusted, but the incorrect Processed
> payload is still accepted.

Per your model and the scenario above, M3 is in your "untrusted"
category, but the "incorrect" fields inserted by it are trusted.
Q.E.D.

> Perhaps you would find the following analogy useful: Just as for
> cryptographic signing, trust is placed in third party certificate
> authorities, here trust is placed in SMTP servers writing Received
> lines.

Your analogy fails because when one verifies a certificate, one
knows where one has gone to obtain verification (short of widespread
failure of DNS *and* IP routing); when one looks at a Received
field, one cannot be certain where it was inserted.

> In cryptographic signing, certificates can be revoked, and here too trust
> in particular SMTP servers can be stopped.

Moot point, because one cannot determine which content is inserted
by any particular SMTP server (short of wrapping and signing at
each stage during transport as suggested earlier in this discussion).

> Similarly, if 
> the Received line's id is "unguessable" (as one example of the
> scheme), then a Processed line which quotes this id is very unlikely
> to have been written before the associated Received line was written.

Another moot point; both fields can be forged without detection.
 
> You've already noted that sometimes a Received line doesn't contain
> an id, and there are difficulties with parsing. Those are good
> points, while requiring a recipient to trust an untrustworthy intermediate
> just so they can mislead the recipient is not I think. 

A recipient may believe that he is using information supplied by
a "trusted intermediate" when in fact he is unknowingly using
forged information.  *I* am imposing no requirement for trust;
I am merely pointing out that a bad guy may exploit such
trust (whether the bad guy is the trusted party or a different
party).

> > Use of multiple filters does not require that any of them be
> > "remote".  Having a local filter and a remote filter does not
> > guarantee different "strengths" (for any definition of that
> > rather nebulous term).  
> 
> Unless, for example, one of the filters has access to a relevant
> database which other filters, residing in a different network
> location, don't have.

I will concede that from an abstract point of view there may be
a theoretical difference in capabilities, but not that that
necessarily implies *greater* "strength", nor that it is a
*guarantee* of a different "strength".  Moreover, as
discussed in a separate message, any hosts capable of
performing meaningful filtering in the context of Internet mail
will necessarily be located in a similar "network location"
from a network topology point of view.

> So why again are you opposed to insertion of cruft into the headers by
> the list expander?

Because it makes matters worse; it is a "cure" that is worse than
the symptoms.  It adds more cruft on top of cruft (spam) and
adds cruft to non-cruft. It exacerbates the problems caused by
spam (wasted transmission bandwidth and storage space).

> If imc.org say performs a virus scan and inserts a 
> header detailing the result, or even changes the subject line by
> inserting a label such as [ietf-822], this should be entirely
> acceptable from the viewpoint of Internet protocols, since the message
> originating from [email-masked] is considered entirely
> independent from the message originating from [email-masked]. The
> sender of the latter can lay no claim to the contents of the former.

No, the messages are not "entirely independent".  The Subject field is
set by the author, and imc.org is not an author in this example, it is
a re-mailer.

> > Adding cruft isn't "filtering". It makes the problems (transmission
> > bandwidth and storage capacity requirements) worse.  True
> > corporate filters (as opposed to "filters" that do no filtering) can
> > and do reduce the problems; but neither they nor any other true
> > filtering method have any need to add the field that you propose.
> 
> I've already given you two examples, the sourceforge mailing list and
> a commercial service such as death2spam. Filtering cruft exists. 

You haven't demonstrated any need to add cruft.  You have simply
shown that cruft can be added, which was already known.

Date: Thu, 11 Nov 2004 12:50:03 +1000
From: Laird Breyer <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked]
Subject: Re: a header authentication scheme
Reply-To: [email-masked]
Mail-Followup-To: ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

On Nov 10 2004, Bruce Lilly wrote:
> On Sun November 7 2004 22:47, Laird Breyer wrote:
> > 
> > Ok, thanks for the clarification. Can I summarize your position in the
> > following terms? Mail transport is in essence an end-to-end service which
> > cannot be trusted by anyone due to the potential for unethical
> > intermediaries to change the content of messages against user's
> > wishes.
> 
> No, that would be inaccurate. Message content can be signed
> (at the originating endpoint, and the signature verified at the
> receiving endpoint) to provide protection against undetected
> modification in transit.

The message signature could be illegally removed in transit, and the
body appended with some innocuous sentence such as "P.S. My PGP is
temporarily broken", along with other subtle changes. That would be
"social engineering", which we know works for a surprisingly large
number of people.

The message could be modified and resigned with another private key
under the purview of the same certificate authority. Unless you've
actually exchanged public keys with the sender at some earlier date,
you'll be trusting the CA if you intend to obtain the public key and
verify the signature. CAs have been compromised before: VeriSign
signed certificates which purported to belong to "Microsoft".

http://www.pkiforum.com/resources/verisigncerts.html

You simply cannot claim that message modifications are detectable
without some measure of trust in external entities separate from the
sender and receiver.

I'm saying that people may elect to trust certain mail transport
agents. There's no difference between that and electing to trust
VeriSign say.

> 
> > My assumption which can be
> > challenged is that all paths to B are covered by a small number of transport
> > agents which can be reasonably labeled by B as being of type T.
> 
> And as I pointed out:
> > > You are assuming an extraordinarily simplistic model which
> > > simply doesn't conform to reality; a model where there are
> > > only "M" and "T" and where each system is either one or the
> > > other, invariant with message content and over time, with no
> > > message transport paths through other mechanisms.

You persist in addressing a non-issue. The sentence preceding the one you
quoted states

       I didn't expect this picture to specify a single path from A to
       B which must always be used by all messages from A to B, rather it
       represents the actual path taken by a given message. "T1" and
       "T2" are whatever makes sense for that actual path.

> 
> > Instead, your examples so far invoke institutionalized corruption by those
> > computer systems. I don't find that very convincing as a counterexample.
> 
> "Corruption" as you are using it is not a technical term.  My
> examples were intended to illustrate the fact that there may
> be cases where some party labels what the recipient might
> consider to be spam as non-spam or vice versa. I have
> given examples where that may be intentional (specifically
> to refute your implication that there is no reason for an
> intermediate to do so); it is also possible due to incompetence,
> difference in values, etc. If your proposition is that "all birds
> are either black or white", I can show that to be a false

My proposition isn't, I don't know which birds you are metaphorically 
referring to.

I've addressed your counterexamples already previously. It is
always the responsibility of the recipient to decide what to trust.
If the recipient trusts the whole world willy-nilly, then the
consequences won't be surprising. 

Arguing that an intermediate may exist which is trusted by the
recipient but, whether intentionally or otherwise, doesn't deserve
this trust, is not very profound. So what?  Show me a counterexample
where the intermediate isn't trusted, but the incorrect Processed
payload is still accepted. Arnt gave such an example, which works
about 1/10 of the times.

Perhaps you would find the following analogy useful: Just as for
cryptographic signing, trust is placed in third party certificate
authorities, here trust is placed in SMTP servers writing Received
lines.  There is no ultimate objective justification for trusting a
CA, nor is there for trusting a given MTA.

In cryptographic signing, certificates can be revoked, and here too trust
in particular SMTP servers can be stopped.

As a consequence, all documents subsequently signed by a revoked
certificate are no longer trustworthy, and similarly all Processed
lines associated with an untrusted Received line are untrustworthy.

But if the certificate is trusted, then messages signed with this
certificate are very unlikely be fully forged, because the associated
mathematical problem is extremely difficult to solve. Similarly, if
the Received line's id is "unguessable" (as one example of the
scheme), then a Processed line which quotes this id is very unlikely
to have been written before the associated Received line was written.

You've already noted that sometimes a Received line doesn't contain
an id, and there are difficulties with parsing. Those are good
points, while requiring a recipient to trust an untrustworthy intermediate
just so they can mislead the recipient is not I think. 

> 
> Use of multiple filters does not require that any of them be
> "remote".  Having a local filter and a remote filter does not
> guarantee different "strengths" (for any definition of that
> rather nebulous term).  

Unless, for example, one of the filters has access to a relevant
database which other filters, residing in a different network
location, don't have.

> 
> > > Filtering that takes place at the receiving endpoint is consistent
> > > with the Internet Architecture.
> > 
> > I agree. It's just not reflective of reality, unless you pick various
> > intermediate points and call them endpoints, such as you did with the
> > example of mailing lists. 
> 
> In a message sent from [email-masked] to [email-masked],
> the latter is unquestionably one endpoint of that communication.
> With regard to the Internet protocols, it is in no way
> "intermediate".  Likewise a message sent from
> [email-masked] to [email-masked] has those
> mailboxes as endpoints.  That some message content may be
> the same due to re-mailing doesn't change those facts.  If
> [email-masked] were to vanish, that would break the first
> path because one of the endpoints disappeared. Disappearance
> of some intermediate system (in a properly-configured system,
> e.g. with at least two MX hosts) would not break that
> communication path.

So why again are you opposed to insertion of cruft into the headers by
the list expander? If imc.org say performs a virus scan and inserts a
header detailing the result, or even changes the subject line by
inserting a label such as [ietf-822], this should be entirely
acceptable from the viewpoint of Internet protocols, since the message
originating from [email-masked] is considered entirely
independent from the message originating from [email-masked]. The
sender of the latter can lay no claim to the contents of the former.

> 
> > MUAs have had spam filtering capabilities for years, starting with 
> > keyword searches in the subject line which had to be entered by hand.
> 
> The Subject field is not designed to be a spam indicator; it is

That's irrelevant. A spam filter performs whatever analysis it is 
programmed to do. It is not bound by the meanings derived from RFCs. 

> 
> Adding cruft isn't "filtering". It makes the problems (transmission
> bandwidth and storage capacity requirements) worse.  True
> corporate filters (as opposed to "filters" that do no filtering) can
> and do reduce the problems; but neither they nor any other true
> filtering method have any need to add the field that you propose.

I've already given you two examples, the sourceforge mailing list and
a commercial service such as death2spam. Filtering cruft exists. 

-- 
Laird Breyer.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: a header authentication scheme
Date: Wed, 10 Nov 2004 09:23:35 -0500
Sender: [email-masked]

On Sun November 7 2004 22:47, Laird Breyer wrote:
> 
> Ok, thanks for the clarification. Can I summarize your position in the
> following terms? Mail transport is in essence an end-to-end service which
> cannot be trusted by anyone due to the potential for unethical
> intermediaries to change the content of messages against user's
> wishes.

No, that would be inaccurate. Message content can be signed
(at the originating endpoint, and the signature verified at the
receiving endpoint) to provide protection against undetected
modification in transit.

> If this summarizes your points,

It doesn't.

> It seems to me that the 
> underlying assumption of mail is that people do at least trust the
> transport infrastructure to faithfully replicate the sender's message
> at the destination.

Transport is designed to be transparent, but it sometimes
fails to be for various reasons; there is at this time another
discussion on the ietf-822 list related to such a possible
failure.  Moreover, your proposal relates to content added
by an intermediary, which is a completely different
matter from "the sender's message".

> My assumption which can be
> challenged is that all paths to B are covered by a small number of transport
> agents which can be reasonably labeled by B as being of type T.

And as I pointed out:
> > You are assuming an extraordinarily simplistic model which
> > simply doesn't conform to reality; a model where there are
> > only "M" and "T" and where each system is either one or the
> > other, invariant with message content and over time, with no
> > message transport paths through other mechanisms.

> Instead, your examples so far invoke institutionalized corruption by those
> computer systems. I don't find that very convincing as a counterexample.

"Corruption" as you are using it is not a technical term.  My
examples were intended to illustrate the fact that there may
be cases where some party labels what the recipient might
consider to be spam as non-spam or vice versa. I have
given examples where that may be intentional (specifically
to refute your implication that there is no reason for an
intermediate to do so); it is also possible due to incompetence,
difference in values, etc. If your proposition is that "all birds
are either black or white", I can show that to be a false
proposition by pointing to a single bird of a different color,
brown for example.  I do not need to produce every brown
bird, nor do I need to produce red, yellow, green, blue, or
gray birds as well, or any birds that have different colors at
different points in their life cycle; one counterexample serves
to expose the flaw in the proposition.  That's fundamental
logic, whether or not you "find that very convincing".

> Instead, a user can feed sets of received messages to a datamining
> tool which extracts the salient features which characterize his SMTP
> neighbourhood.  For example, important hostname aliases and domain
> literals will be automatically highlighted by their frequency of
> appearance in received mail.  This requires a representative sample of
> messages arriving at a single destination address, but no detailed
> understanding of the mail server configurations and local network geometry.

If such a hypothetical tool existed, it would be of little help;
it may determine that there is some mix of messages that go
via system "foo", and that some of those are labeled "spam"
and others are labeled "non-spam".  That is of absolutely no
use in determining whether or not a particular message which
has passed through "foo" is labeled in accordance with the
recipient's values.

> > But if the user has to run a local filter in order to determine whether
> > or not to trust a remote filter, what is the point of having the
> > remote filter in the first place?
> 
> Quite simply diversity. Combining two or more filters with different
> strengths is better (allows more accuracy) than choosing to use a single
> filter with known limitations. It also allows specialization, and 
> smaller resource usage. 

Use of multiple filters does not require that any of them be
"remote".  Having a local filter and a remote filter does not
guarantee different "strengths" (for any definition of that
rather nebulous term).  I believe that you would have a hard
time showing that in general having both a local filter and a
remote filter uses fewer total resources than a local filter
alone; resource usage may well be distributed differently,
but is likely to be greater. Aside from that, smaller resource
usage -- even if that were the case -- would not be an
advantage if it led to incorrect results.

> > Filtering that takes place at the receiving endpoint is consistent
> > with the Internet Architecture.
> 
> I agree. It's just not reflective of reality, unless you pick various
> intermediate points and call them endpoints, such as you did with the
> example of mailing lists. 

In a message sent from [email-masked] to [email-masked],
the latter is unquestionably one endpoint of that communication.
With regard to the Internet protocols, it is in no way
"intermediate".  Likewise a message sent from
[email-masked] to [email-masked] has those
mailboxes as endpoints.  That some message content may be
the same due to re-mailing doesn't change those facts.  If
[email-masked] were to vanish, that would break the first
path because one of the endpoints disappeared. Disappearance
of some intermediate system (in a properly-configured system,
e.g. with at least two MX hosts) would not break that
communication path.

> MUAs have had spam filtering capabilities for years, starting with 
> keyword searches in the subject line which had to be entered by hand.

The Subject field is not designed to be a spam indicator; it is
intended to be set by the author as an indication of the message
topic, however there is no guarantee that authors will not abuse
the field -- if you expect spammers (and/or vested corporate
interests, and/or individuals or groups with some axe to grind
(politicians, political parties, governments, preachers, religious
groups, etc.) to always tell the truth, you are in for a great deal
of disappointment. Searching Subject fields alone will therefore
not be reliable for spam detection.  And mere detection is not
the same as filtering.

> Mozilla has filtering capabilities, but it's not very good compared
> with alternatives.

Mozilla has a variety of individual capabilities which can be
combined to perform true filtering at the MUA level.

> Alternatives insert X-headers. Corporate 
> filters add X-headers.

Adding cruft isn't "filtering". It makes the problems (transmission
bandwidth and storage capacity requirements) worse.  True
corporate filters (as opposed to "filters" that do no filtering) can
and do reduce the problems; but neither they nor any other true
filtering method have any need to add the field that you propose.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Wed, 10 Nov 2004 09:42:38 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Mon November 8 2004 06:24, Charles Lindsey wrote:

> My reading of RFC 2822 is that it changes <domain> to <id-right>, after
> which the semantics of an <id-right> apply to it.

Then you're going around in circles, paying insufficient attention
to what you read and failing to think clearly about the issue before
writing. We've covered this ground; RFC 2822 introduced id-right
for pure *syntax* reasons (viz. to avoid FWS)[1] -- it makes no
assertions about any supposed difference in *semantics*. That is in
fact the very heart of the problem with that part of 2822 -- it is
ambiguous precisely because it fails to state whether or not the
semantics of a msg-id are changed.  Writing about "the semantics
of an <id-right>" is pure poppycock; there are no such semantics
defined anywhere in RFC 2822 or its normative references.

1. "the msg-id has
   a similar syntax to angle-addr (identical except that comments and
   folding white space are not allowed)"

To: [email-masked]
Subject: Re: Angle brackets surrounding Content-ID
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 08 Nov 2004 10:27:25 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Martin Trautmann <[email-masked]> writes:

>> Maybe - my newsreader just saves this info before, e.g.

>> From [email-masked] Fri Nov  5 11:39:39 2004
>> Newsgroups: <any>
>> From: Martin Trautmann <[email-masked]>
>> Subject: ...

>> It take the address from From: and the time stampe while actually saving
>> the posting

> Ah! I see. It is unfortunate that it mimics the mail style of "^From�"
> line so efectively that you cannot tell the difference :-( .

That's the whole point.  Unix news readers have had the ability to save
articles into mbox files so that mail readers can read them for decades,
dating all the way back to at least rn.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Cc: ietf-822 <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Mon, 8 Nov 2004 10:51:30 -0500
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

On Nov 8, 2004, at 6:15 AM, Charles Lindsey wrote:

> You might like to look at
> http://www.imc.org/ietf-usefor/drafts/drafts/draft-lindsey-usefor- 
> signed-01.txt.

You might indeed, but I predict better luck with:

	http://www.imc.org/ietf-usefor/drafts/draft-lindsey-usefor-signed 
-01.txt

To: LIST: [email-masked];
Xref: clerew local.mime:4175
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Mon, 8 Nov 2004 11:15:52 GMT
Lines: 52
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>What's really needed is a generic way of computing a hash of a MIME object that
>takes as many of these issues as possible into account. I've had the
>specification of such a thing on my to-do list literally for years but I never
>seem to find the time to finish writing it up.

>Basically what you want to do is define a hash methodology that computes
>separate hashes on leaf nodes in the MIME object and then combines those
>separate hashes along with hashes of canonicalized headers and the MIME
>structure itself in a specific way to arrive at a single result. The
>advantages of this approach are numerous:

>(1) Encodings can be changed without breaking signatures. (This can help
>    with handling whitespace, and it makes it possible for signatures to
>    survive 8->7 conversion.)
>(2) Boundary markers can be changed without breaking signatures. (How
>    to handle preamble and postamble text is an interesting side issue here.)
>(3) A message store can cache hash values for large leaf objects and use
>    the cache to quickly sign very large messages.
>(4) Some sorts of header mangling can be tolerated.

>So, is it time for me to finish the specification for this? Does anybody
>care, and more to the point, will anybody actually implement it?

You might like to look at
http://www.imc.org/ietf-usefor/drafts/drafts/draft-lindsey-usefor-signed-01.txt.
ALthough it was designed to solve a different problem, it does include a
vicious canonicalizing algorithm for header fields (though perhaps not
vicious enough).

Also, for body parts it recommends using Content-MD5 for computing hashes
of bodies, and then including the Content-MD5 field in the overall
signature so that, if something fails, you have a better chance of finding
which of various parts of the message it failed in.

Also, as regards suggestions for an application/rfc-822 type, there
already exists (registered with IANA) an application/news-transmission
which was intended for essentially the same purpose - namely to transmit a
complete newsd article without fear of gratuitous munging by intermediate
sites.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4177
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 8 Nov 2004 11:22:41 GMT
Lines: 39
Sender: [email-masked]

In <[email-masked]> Martin Trautmann <[email-masked]> writes:

>Maybe - my newsreader just saves this info before, e.g.

> From [email-masked] Fri Nov  5 11:39:39 2004
> Newsgroups: <any>
> From: Martin Trautmann <[email-masked]>
> Subject: ...

>It take the address from From: and the time stampe while actually saving
>the posting

Ah! I see. It is unfortunate that it mimics the mail style of "^From�"
line so efectively that you cannot tell the difference :-( .

>> The problem arises in User Agents that try to overload one of the same two
>> buttons "Reply" and "Reply-to-All" to double up as the "Followup" button
>> when operating in News mode (a practice strongly deprecated by the GNKSA,
>> but prevalent in many 2nd rate agents anyway).

>Then it's more a question of poor user interface, where the agent trained
>the user to use 'reply/reply to all', while it did not teach the
>difference between postings and mail, between follow-up and reply

There I agree. If there is doubt as to whether to send a news followup or
an email reply, then it is really the responding user who needs to make
the choice, and a decent User Agent interface should offer both in a clear
manner.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4176
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Understanding response protocols
Date: Mon, 8 Nov 2004 11:18:08 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Martin Trautmann <[email-masked]> writes:

>On 2004-11-01 17:48, Charles Lindsey wrote:
>> I much prefer those lists which set Reply-To (I have been on lists where
>> the members have voted to have it set so), even though I have to remember
>> which list is which in order to press the right Reply button.

>What happens on these lists when the sender defined a Reply-To of his own?
>I guess it's just replaced?

The opinion expressed several times in these recent threads is that a
Reply-To field already set by the sender ought to be left alone by a
mailing list expander. Likewise an MFT header if/when we have one.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4178
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 8 Nov 2004 11:24:11 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Fri November 5 2004 10:34, Charles Lindsey wrote:

>> You said
>> 
>> >RFC 2822 id-right is a problem because it changes the semantics.
>> 
>> So what other change of semantics did you have in mind then?

>As I said, it changes "domain" to "maybe a domain, maybe not,
>with no way to tell".  That has many implications other than
>case-insensitivity of domain names.

My reading of RFC 2822 is that it changes <domain> to <id-right>, after
which the semantics of an <id-right> apply to it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
From: Nick Shelness <[email-masked]>
Date: Mon, 8 Nov 2004 09:39:10 +0000
Sender: [email-masked]

[email-masked] wrote on 11/05/2004 18:28:56

> ... Basically what you want to do is define a hash methodology that 
computes
> separate hashes on leaf nodes in the MIME object and then combines those
> separate hashes along with hashes of canonicalized headers and the MIME
> structure itself in a specific way to arrive at a single result. ...

Ned already knows this, as he and I have discussed defining an alternative 
MIC over the years.

Lotus Notes in its proprietary message format employs this approach. When 
a Notes document is signed, both embedded objects and attached files are 
individually hashed, and these hashes are then placed in control blocks in 
the main message body. The main message body, including these control 
blocks, and other associated items (think headers) is then hashed and this 
hash is signed.

When a signed Notes document is opened, the validity of the main message 
body hash is verified. If it is valid the main message body is rendered. 
This rendering may include the rendering of embedded objects. As each of 
these objects is opened and de-compressed, the validity of the hash in 
their associated control block is also verified. Similarly, when an 
attachment is opened and de-compressed, the validity of the hash in its 
associated control block is verified. If Notes fails to validate either a 
top level, embedded object, or attachment hash then a signature failure is 
asserted. In the case of the top-level message and its embedded objects, 
this failure will be signalled prior to them being displayed. In the case 
of an attachment, this failure will be signalled prior to the attachment 
being invoked or saved.

If one presumes a flat "on the wire" storage format for a delivered MIME 
message, then a hierarchical hash scheme would appear to deliver no 
benefits. If on the other hand on assumes a hierarchical storage format 
with leaf node bodies decoupled from MIME headers as is largely mandated 
my IMAP semantics, then a hierarchical hash scheme delivers significant 
benefits, especially when initially opening and verifying the signature of 
a message with large attachments.

Nick

Date: Mon, 8 Nov 2004 13:47:09 +1000
From: Laird Breyer <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: a header authentication scheme
Reply-To: [email-masked]
Mail-Followup-To: ietf-822 <[email-masked]>
Sender: [email-masked]

Ok, thanks for the clarification. Can I summarize your position in the
following terms? Mail transport is in essence an end-to-end service which
cannot be trusted by anyone due to the potential for unethical
intermediaries to change the content of messages against user's
wishes.

If this summarizes your points, then I'm not sure how productive such
an extreme position can be on this issue. It seems to me that the
underlying assumption of mail is that people do at least trust the
transport infrastructure to faithfully replicate the sender's message
at the destination. Otherwise there can be no communication at all.

On Nov 04 2004, Bruce Lilly wrote:

> > So we always have a model of message transport as follows:
> > 
> > A -> M1 -> M2 -> M3 -> ... -> T1 -> T2 -> ... -> B
> 
> Not necessarily.  There may well be multiple paths to B, N-1
> of which do not pass through "T2".  "T1", "T2", etc. might
> pass some messages w/o change, but may change some
> fraction of messages (i.e. those messages not sympathetic
> to the interests of the owner/operators of those systems).

I didn't expect this picture to specify a single path from A to B
which must always be used by all messages from A to B, rather it
represents the actual path taken by a given message. "T1" and "T2" are
whatever makes sense for that actual path. My assumption which can be
challenged is that all paths to B are covered by a small number of transport
agents which can be reasonably labeled by B as being of type T.

>  
> > A is the sender of the message, B is the destination, M represents
> > computers which are untrusted by B, and T represents computers which
> > are trusted by B and add a Received field.  None of the computers of
> > type T will ever perform either a) or b) above, unless they have been
> > compromised.
> 
> No, repeating "unless they have been compromised" doesn't
> make it true. 

Instead, your examples so far invoke institutionalized corruption by those
computer systems. I don't find that very convincing as a counterexample.

> 
> > Computers of type M may well perform a) or b). It is up 
> > to B to only trust Processed fields associated with computers of type
> > T.
> 
> You are assuming an extraordinarily simplistic model which
> simply doesn't conform to reality; a model where there are
> only "M" and "T" and where each system is either one or the
> other, invariant with message content and over time, with no
> message transport paths through other mechanisms.

No, I'm expressing the essential framework needed for the scheme in the
simplest terms I can think of. I accept the criticism that a message
being sent from A to B without using SMTP at all would be outside the scope
of consideration.

> > But are you really saying that a given user B
> > cannot, by examining a sample of past mail messages he personally received,
> > deduce the form of the Received fields added by the computers I've labeled
> > T above?
> 
> It certainly depends somewhat on B's knowledge and experience.
> I would estimate that for > 99.9% of email recipients, a *valid*
> deduction would be extremely unlikely; probably 99.5% have no
> clue what a Received field is in the first place.  Judging by illegal
> syntax in many generated Received fields, it's clear that a large
> fraction of those who think that they know what a Received
> field is are simply wrong.  On top of that, there are hostname
> aliases, domain literals, etc. which require an additional
> understanding of the DNS.

I'm sorry, this is my fault for badly expressing myself. Asking
humans to perform header analysis individually is not of any great value.

Instead, a user can feed sets of received messages to a datamining
tool which extracts the salient features which characterize his SMTP
neighbourhood.  For example, important hostname aliases and domain
literals will be automatically highlighted by their frequency of
appearance in received mail.  This requires a representative sample of
messages arriving at a single destination address, but no detailed
understanding of the mail server configurations and local network geometry.

> But if the user has to run a local filter in order to determine whether
> or not to trust a remote filter, what is the point of having the
> remote filter in the first place?

Quite simply diversity. Combining two or more filters with different
strengths is better (allows more accuracy) than choosing to use a single
filter with known limitations. It also allows specialization, and 
smaller resource usage. 

> Filtering that takes place at the receiving endpoint is consistent
> with the Internet Architecture.

I agree. It's just not reflective of reality, unless you pick various
intermediate points and call them endpoints, such as you did with the
example of mailing lists. 

> * years ago, things like X-RBL etc. were sometimes inserted by ISPs;
>    some users complained about them (apparently believing that it
>    involved eavesdropping, which was not the case), but they were
>    largely useless because of DHCP and other types of shared IP
>    addresses, shared IP address blocks, etc.

Which doesn't mean it's no longer used. I don't care what individual schemes
ISPs come up with next, I want to see whether their schemes can be traced
back to them through a lightweight method.

> * About 6 months ago, my ISP tried to implement message mangling
>    w/o user input or consent.  That quickly went away after I
>    complained about a legitimate mailing list message which was
>    modified by the ISP (among other things, "*Possible Junk Mail* "
>    was prepended to the message Subject field).  That *was*
>    eavesdropping; it was forgery as well.

Good for you. Judging by the popularity of Gmail, eavesdropping isn't 
much of an issue to many people, nor is forgery. The world would be 
better if more people believed in privacy as you do.

> * A large number of MUAs have spam filtering capability

MUAs have had spam filtering capabilities for years, starting with 
keyword searches in the subject line which had to be entered by hand.
Mozilla has filtering capabilities, but it's not very good compared
with alternatives. Alternatives insert X-headers. Corporate
filters add X-headers. Who did what? Back to square one.

-- 
Laird Breyer.

Date: Sun, 07 Nov 2004 14:20:39 -0800 (PST)
From: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
To: Bruce Lilly <[email-masked]>
Cc: ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

> On Sat November 6 2004 20:37, [email-masked] wrote:

> > >     * it should be easy to combine and separate the hashes
> >
> > Only hash combination is needed, not separation.

> That depends on the goals; if signature validity checking is
> to be limited to something/nothing changed, then you're
> right. However if one wants to be able to say these
> specific N objects are unchanged; those specific M objects
> have been altered, then the receiver will need to have access
> to the individual hashes.

> And if one is going to settle for a binary result, then it's
> not clear that separate hashes are necessary in the first
> place (some suitable canonicalization should suffice).

Separate hashes add effectively no cost to the scheme, and allow precomputation
and caching of hash values for large stored objects.

> > >     * it should be possible to determine if some object and its
> > >        corresponding hash has been removed (or if some content
> > >        has been added)
> >
> > This one is next to impossible to do IMO, and falls far outside
> > the goals I would have to such a mechanism.

> If some part of a multipart/mixed or multipart/related
> message is elided, the meaning of the message as a whole
> is altered, and should be detectable by the receiver.  Now
> hashing hashes or a single hash with canonicalization are
> methods that are not going to present a problem here, but
> other methods (e.g. simple concatenation of hashes)
> might.

Being able to detect modification is the goal. Being able to determine what the
modification was is a nongoal.

> > I'm well aware that in general there is no way to do header canonicalization
> > and hashing perfectly. The question is how far is far enough.

> Or from another perspective: how far is too far.  I suspect
> that much beyond unfolding and squeezing of linear
> whitespace (after unfolding) would be going too far.  One
> *might* uniformly up-case or down-case all alphabetics
> to try to deal with case changes of protocol elements
> such as field name tags during field rewriting, but it risks
> losing sensitivity to significant case changes in header
> text (in the RFC 2277 sense of "text").
 
I pretty much agree with this assessment, modulo issues with
specific fields like CT and CTE.

> > > Would changing boundary markers not also change the MIME-part
> > > header (boundary parameter in Content-Type field)?
> >
> > Content-type is the one field that simply must be canonicalized in a way
> > that avoids this problem.

> That implies not including the boundary parameter (whether
> a simple parameter, or fragmented per RFC 2231) [probably
> also the semicolon preceding the attribute name] in the
> hash computation (before squeezing linear whitespace).

Correct. RFC 2231 is at least part of the reason for this, although
I tend to think worrying about RFC 2231 encoding of boundary parameters
is overkill.

				Ned

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Sun, 7 Nov 2004 16:42:50 -0500
Sender: [email-masked]

On Sun November 7 2004 16:06, Bruce Lilly wrote:
> 
> On Sat November 6 2004 19:16, Nathaniel Borenstein wrote:

> > And the only difference would be that if you were signing it, you gave 
> > it the primary type "application" instead of "message"?
> 
> No, in a case where message/rfc822 inside multipart/signed is
> known to be mangled by some particular transport,
> application/rfc822 would be used instead. 
[...]
> In your example scenario, if you send a signed message/rfc822
> and the signature is unverifiable due to a mismatch (as opposed
> to something like lack of an available public key), and if having
> a valid signature is deemed important, I would report that the
> message signature could not be verified, and ask you to resend
> it.  If you had reason to believe that the mangling was due to
> some improper manipulation of the message/rfc822 content, you
> could package it as application/rfc822 (or application/octet-stream
> w/o having to have a new media type registered for this case).

Just to clarify two points:
1. The primary reason to have application/rfc822 would be to
    obviate a manual note (with application/octet-stream) that
    the content is to be treated as a message; it's shorthand.
2. In addition to using the application type to work around
    message handlers that don't respect multipart/signed,
    transport encoding could be used as additional protection
    with an application type (explicitly forbidden with composite
    types).

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Sun, 7 Nov 2004 16:06:36 -0500
Cc: Nathaniel Borenstein <[email-masked]>, [email-masked]
Sender: [email-masked]

On Sat November 6 2004 19:16, Nathaniel Borenstein wrote:

> > There may be a way around that; simply define a discrete media
> > type that otherwise looks like message/rfc822 -- maybe
> > application/rfc822 a la message/http vs. application/http. That
> > discrete media type could then have protective encoding applied
> > for transport without violating the rule regarding identity
> > encodings of composite media types.  Of course that won't be
> > "transparent" in the same sense as message/rfc822
> 
> And the only difference would be that if you were signing it, you gave 
> it the primary type "application" instead of "message"?

No, in a case where message/rfc822 inside multipart/signed is
known to be mangled by some particular transport,
application/rfc822 would be used instead.  I suspect that
message/rfc822 works fine in the vast majority of cases
provided that the applicable existing canonicalization rules
are applied to valid message/rfc822 content.  Use of
application/rfc822 would be for pathological cases.

> That makes me  
> unhappy because now we're saying that the data type depends on the way 
> the data is being protected/wrapped/used.  (It's like saying I'm a girl 
> if you put in me in a gown, but a boy if you put me in a tux.)  Let's 
> say that I forward you a message that is encapsulated as a 
> message/rfc822 object.  Imagine further that at the outgoing gateway 
> from my email domain, a PGP signature is being applied.  Would you have 
> the software that is doing the signing also change the content-type of 
> the embedded object?

No. I'd have it leave the media type unchanged.

> Would you have decoding software change it back?   

No.

> If so, what you're really doing is building  into the PGP signature 
> algorithm a small "transfer-encoding" that specifically protects 
> headers in a message/rfc822 object and nothing else.  I'm not saying 
> that wouldn't work, but there's got to be a prettier way to do 
> things.....  doesn't there?  -- Nathaniel

In your example scenario, if you send a signed message/rfc822
and the signature is unverifiable due to a mismatch (as opposed
to something like lack of an available public key), and if having
a valid signature is deemed important, I would report that the
message signature could not be verified, and ask you to resend
it.  If you had reason to believe that the mangling was due to
some improper manipulation of the message/rfc822 content, you
could package it as application/rfc822 (or application/octet-stream
w/o having to have a new media type registered for this case).
On receipt, the decoding application (probably an MUA) would
save the result as a file, which could be handled by any application
that can read message/rfc822 (I am presuming that when you
package the content as an application type to work around the
mangling, that you include a brief note indicating that the
content is in fact of type message/rfc822, but relabeled specifically
to work around the signature invalidation issue).  I'm not
advocating building anything into signature algorithms; I'm
suggesting a human workaround agreed upon by the parties
involved in the end-to-end communication.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Sun, 7 Nov 2004 15:52:49 -0500
Cc: [email-masked]
Sender: [email-masked]

On Sat November 6 2004 20:37, [email-masked] wrote:

> >     * it should be easy to combine and separate the hashes
> 
> Only hash combination is needed, not separation.

That depends on the goals; if signature validity checking is
to be limited to something/nothing changed, then you're
right. However if one wants to be able to say these
specific N objects are unchanged; those specific M objects
have been altered, then the receiver will need to have access
to the individual hashes.

And if one is going to settle for a binary result, then it's
not clear that separate hashes are necessary in the first
place (some suitable canonicalization should suffice).

> >     * it should be possible to determine if some object and its
> >        corresponding hash has been removed (or if some content
> >        has been added)
> 
> This one is next to impossible to do IMO, and falls far outside
> the goals I would have to such a mechanism.

If some part of a multipart/mixed or multipart/related
message is elided, the meaning of the message as a whole
is altered, and should be detectable by the receiver.  Now
hashing hashes or a single hash with canonicalization are
methods that are not going to present a problem here, but
other methods (e.g. simple concatenation of hashes)
might.

> I'm well aware that in general there is no way to do header canonicalization
> and hashing perfectly. The question is how far is far enough.

Or from another perspective: how far is too far.  I suspect
that much beyond unfolding and squeezing of linear
whitespace (after unfolding) would be going too far.  One
*might* uniformly up-case or down-case all alphabetics
to try to deal with case changes of protocol elements
such as field name tags during field rewriting, but it risks
losing sensitivity to significant case changes in header
text (in the RFC 2277 sense of "text").
 
> > Would changing boundary markers not also change the MIME-part
> > header (boundary parameter in Content-Type field)?
> 
> Content-type is the one field that simply must be canonicalized in a way
> that avoids this problem.

That implies not including the boundary parameter (whether
a simple parameter, or fragmented per RFC 2231) [probably
also the semicolon preceding the attribute name] in the
hash computation (before squeezing linear whitespace).

Date: Sat, 06 Nov 2004 18:29:14 -0800 (PST)
From: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
To: Bruce Lilly <[email-masked]>
Cc: ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

> > Finally, as far as lines over the SMTP limit of 998 are concerned, I have
> > zero sympathy since it is technically illegal to send such a thing
> > in the first place, irrespective of the MIME constructs involved. Anyone
> > who expects overly long lines to be treated with fidelity is simply being
> > silly and deserves whatever happens.

> While I haven't counted octets, it certainly doesn't look like the
> Received field that was folded was anywhere near 998 octets
> in unfolded form.  I don't think that's an issue.

OK, fair enough. That puts the change in the "gratuitous inside of a signed
object" category for me.

				Ned

Date: Sat, 06 Nov 2004 17:37:12 -0800 (PST)
From: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
To: Bruce Lilly <[email-masked]>
Cc: ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

> On Fri November 5 2004 13:28, [email-masked] wrote:

> > What's really needed is a generic way of computing a hash of a MIME object that
> > takes as many of these issues as possible into account. I've had the
> > specification of such a thing on my to-do list literally for years but I never
> > seem to find the time to finish writing it up.
> >
> > Basically what you want to do is define a hash methodology that computes
> > separate hashes on leaf nodes in the MIME object and then combines those
> > separate hashes along with hashes of canonicalized headers and the MIME
> > structure itself in a specific way to arrive at a single result. The
> > advantages of this approach are numerous:

> 1. The "combines" part is likely to pose some problems because there
>     are some conflicting characteristics which one would like to have:

>     * it should be easy to combine and separate the hashes

Only hash combination is needed, not separation.

>     * the result should be sensitive to reordering of objects

This is quite simple.

>     * it should be possible to determine if some object and its
>        corresponding hash has been removed (or if some content
>        has been added)

This one is next to impossible to do IMO, and falls far outside
the goals I would have to such a mechanism.

>     * for large objects, a hash represents data reduction; as the
>        ratio of object size to hash length increases, there is a
>        reduction in sensitivity (ability to detect changes). The act
>        of combining hashes should not result in additional decrease
>        in sensitivity

Sufficiently strong hashes make this a nonissue, I believe.

>     * for small objects, a hash may be larger than the object itself;
>        as a message is split into a greater number of smaller objects
>        which are individually hashed, the total size of the hashes
>        grows, resulting in undesirable increased overhead

You're assuming that all the separate hashes are present; this is not how the
scheme works. Rather, the hashes are themselves hashed.

> 2. Canonicalization of headers presents several problems:
>     a) it presupposes that the syntax and structure of each field is known,
>         which can become a snag as new fields are defined, when user-
>         defined or experimental fields are used, as new protocol element
>         keywords (charset names, media types and subtypes, etc.) are
>         registered, etc.   A similar problem exists for RFC 2047 encoded-
>         words; because they can appear only in certain contexts (some
>         comments, in phrases, and in unstructured fields), one needs to
>         know detailed field syntax  to determine if something which
>         looks like an encoded-word is in fact an encoded-word.
>    b)  Unfolding, normalization and compression of whitespace are
>         probably reasonable for structured fields, but differences in line
>         folding, tabs vs. spaces, and quantity of whitespace characters
>        may be significant in some instances in unstructured fields
>         (Subject, Comments, Content-Description, etc.).
>    c) Some objects are case-sensitive, others case-insensitive; that
>        should be taken into account during canonicalization. However,
>        in some instances is is simply not possible to determine whether
>        some field text is a case-insensitive object or not.

I'm well aware that in general there is no way to do header canonicalization
and hashing perfectly. The question is how far is far enough.

> > (1) Encodings can be changed without breaking signatures. (This can help
> >     with handling whitespace, and it makes it possible for signatures to
> >     survive 8->7 conversion.)

> Maybe. Change in encoding should result in change of
> Content-Transfer-Encoding fields.  How would one handle that
> change to the MIME-part header associated with the encoded
> part, bearing in mind that once an object has been encoded, there
> is no record of whether it was encoded from an original specified
> as 8bit or as binary (which might or might not have been mislabeled)?

Simple: You exclude the CTE field from the hash. And yes, I know this has some
issues and corner cases, but IMO the benefits far outweigh the costs.

> > (2) Boundary markers can be changed without breaking signatures. (How
> >     to handle preamble and postamble text is an interesting side issue here.)

> Would changing boundary markers not also change the MIME-part
> header (boundary parameter in Content-Type field)?

Content-type is the one field that simply must be canonicalized in a way
that avoids this problem.

> [...]
> > So, is it time for me to finish the specification for this?

> I suspect that 2a above might be a show-stopper.

I disagree.

> > Does anybody
> > care, and more to the point, will anybody actually implement it?

> And if so, will there be multiple interoperable implementations (which
> continue to interoperate as new fields, charsets, media types and
> subtypes, etc. develop)?   Would such a method interoperate with
> current methods?  How would the different method be indicated:
> MIME-Version: 2.0?  Won't there still be problems if the original
> isn't in canonical form when signed (and won't more complex
> rules add to that problem)?  Won't there still be problems with
> non-MIME-aware message handlers and with legacy MIME
> implementations?

This has nothing to do with MIME proper and everything to do with specific
signature schemes. No changes to MIME are needed or expected for this. And yes,
I realize this approach has the same problems as the separate specification of
mutlipart/signed, and for the same reasons. But as I said before, some things
are too late to change.

				Ned

Date: Sat, 06 Nov 2004 17:32:21 -0800 (PST)
From: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
To: Nathaniel Borenstein <[email-masked]>
Cc: ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

> > There may be a way around that; simply define a discrete media
> > type that otherwise looks like message/rfc822 -- maybe
> > application/rfc822 a la message/http vs. application/http. That
> > discrete media type could then have protective encoding applied
> > for transport without violating the rule regarding identity
> > encodings of composite media types.  Of course that won't be
> > "transparent" in the same sense as message/rfc822

> And the only difference would be that if you were signing it, you gave
> it the primary type "application" instead of "message"?  That makes me
> unhappy because now we're saying that the data type depends on the way
> the data is being protected/wrapped/used.  (It's like saying I'm a girl
> if you put in me in a gown, but a boy if you put me in a tux.)  Let's
> say that I forward you a message that is encapsulated as a
> message/rfc822 object.  Imagine further that at the outgoing gateway
> from my email domain, a PGP signature is being applied.  Would you have
> the software that is doing the signing also change the content-type of
> the embedded object?  Would you have decoding software change it back?
> If so, what you're really doing is building  into the PGP signature
> algorithm a small "transfer-encoding" that specifically protects
> headers in a message/rfc822 object and nothing else.  I'm not saying
> that wouldn't work, but there's got to be a prettier way to do
> things.....  doesn't there?  -- Nathaniel

The significant problem with the encapsulation approach is the same as its
significant advantage: It renders the content opaque to MIME readers.

We go around this particular maypole all the time and it is always the same:
You either allow near-ubiquituous access to the content, which invariably leads
to content modification (no matter what the standards do or do not say) or you
encapsulate opaquely, in the process rendering the content hard to access.

This is fundamental and irreconcilable, and is always what trips up proposals
along these lines.

				Ned

Date: Sat, 06 Nov 2004 17:29:24 -0800 (PST)
From: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
To: Bruce Lilly <[email-masked]>
Cc: ietf-822 <[email-masked]>, Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

> On Fri November 5 2004 08:32, Nathaniel Borenstein wrote:
> >
> > Let me be
> > the first to say that at least part of the problem here may be in the
> > MIME standard itself.
> >
> > This situation could in theory be avoided by simply applying a suitable
> > content-transfer-encoding to the embedded message/rfc822 object
> > *before* PGP-signing it, thus protecting against line wrapping.
> > However, this is explicitly *prohibited* by the MIME standard, as per
> > RFC 2046:
> >
> >  >￼￼￼￼￼   No encoding other than "7bit", "8bit", or "binary" is
> > permitted for
> >  >   the body of a "message/rfc822" entity.
> >
> > That may still have been the right design decision -- forbidding
> > protective C-T-E's for multipart and message makes message structure
> > much more transparent -- but as far as I can tell, there is NO
> > legitimate way to encode and sign a message/rfc822 object to protect
> > against line wrapping in the header fields of the embedded
> > message/rfc822 object.

> There may be a way around that; simply define a discrete media
> type that otherwise looks like message/rfc822 -- maybe
> application/rfc822 a la message/http vs. application/http. That
> discrete media type could then have protective encoding applied
> for transport without violating the rule regarding identity
> encodings of composite media types.  Of course that won't be
> "transparent" in the same sense as message/rfc822.

This has been proposed numerous times, but none of the proposals have
made it to RFC status.

				Ned

Cc: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Sat, 6 Nov 2004 19:16:37 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

Some quick comments:

I'm not convinced 2a is a showstopper, but then I'm happy with just 
signing whatever isn't whitespace on any headers we don't make special 
rules for.

>> Does anybody
>> care, and more to the point, will anybody actually implement it?

I suspect this is worth doing, if only to address the problems that I 
anticipate will follow upon the apparent inevitability of some form of 
domain signatures.  I am particularly worried about messages that are 
signed at different times by multiple mechanisms for multiple purposes. 
  It's going to be complicated, and it would be nice if implementors all 
did it at least *roughly* the same way.  :-(

> There may be a way around that; simply define a discrete media
> type that otherwise looks like message/rfc822 -- maybe
> application/rfc822 a la message/http vs. application/http. That
> discrete media type could then have protective encoding applied
> for transport without violating the rule regarding identity
> encodings of composite media types.  Of course that won't be
> "transparent" in the same sense as message/rfc822

And the only difference would be that if you were signing it, you gave 
it the primary type "application" instead of "message"?  That makes me 
unhappy because now we're saying that the data type depends on the way 
the data is being protected/wrapped/used.  (It's like saying I'm a girl 
if you put in me in a gown, but a boy if you put me in a tux.)  Let's 
say that I forward you a message that is encapsulated as a 
message/rfc822 object.  Imagine further that at the outgoing gateway 
from my email domain, a PGP signature is being applied.  Would you have 
the software that is doing the signing also change the content-type of 
the embedded object?  Would you have decoding software change it back?  
If so, what you're really doing is building  into the PGP signature 
algorithm a small "transfer-encoding" that specifically protects 
headers in a message/rfc822 object and nothing else.  I'm not saying 
that wouldn't work, but there's got to be a prettier way to do 
things.....  doesn't there?  -- Nathaniel

Date: Sat, 6 Nov 2004 14:45:14 -0500
From: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Archives
Sender: [email-masked]

> > because the "name" is a URL, it _will_ be different for different 
> > access schemes.  I'm expecting that the archiver will have to know
> > about the access methods that are available.
> 
> I think that's a significant practical problem.   I have outlined
> a means of implementing access to an archive of mailing list
> messages via a half dozen URI schemes using off-the-shelf
> software packages.  Adding an Archived-At field with IMAP, POP,
> and perhaps NNTP URIs would certainly require modification --
> perhaps extensive -- but is probably feasible [1,2]. Expecting an
> IMAP local delivery archiver to possess detailed information
> about ftp, http, etc. access methods and configuration is a bit
> much; again, probably feasible, but not without a considerable
> amount of modification.

when I know how to implement a mail archiver that not only stores the archive
in a file but adds the archived-at fields and forwards the mail to the list
expander in a few lines of script, I have a hard time believing it's so
onerous. 

> 1. A news URI referring to the message identifier would be
>     trivial; however it would also be nearly useless for remote
>     access as the news URI scheme has no provision for
>     specifying host or port.  Likewise for the mid URI scheme,
>     which is indistinguishable (when referring to an entire message)
>     from the specific variety of news scheme URI mentioned here.

defining a new URI scheme that says "use NNTP to find this message-id
at this host and port" would not be rocket science.  and since we already
pretty much know that existing UAs do not support the ability to access
mail archives at URIs, backward compatibility would not be an issue.

though I'm still partial to using FTP and HTTP.

Keith

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Archives
Date: Sat, 6 Nov 2004 07:16:43 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Thu November 4 2004 11:49, Keith Moore wrote:

> because the "name" is a URL, it _will_ be different for different 
> access schemes.  I'm expecting that the archiver will have to know
> about the access methods that are available.

I think that's a significant practical problem.   I have outlined
a means of implementing access to an archive of mailing list
messages via a half dozen URI schemes using off-the-shelf
software packages.  Adding an Archived-At field with IMAP, POP,
and perhaps NNTP URIs would certainly require modification --
perhaps extensive -- but is probably feasible [1,2]. Expecting an
IMAP local delivery archiver to possess detailed information
about ftp, http, etc. access methods and configuration is a bit
much; again, probably feasible, but not without a considerable
amount of modification.

1. A news URI referring to the message identifier would be
    trivial; however it would also be nearly useless for remote
    access as the news URI scheme has no provision for
    specifying host or port.  Likewise for the mid URI scheme,
    which is indistinguishable (when referring to an entire message)
    from the specific variety of news scheme URI mentioned here.

2. One issue is that adding a field at that late stage will change
    the message size, which might have already been recorded
    in a database (for returning message metadata and folder
    statistics via IMAP, POP, and NNTP protocols).

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Sat, 6 Nov 2004 04:12:20 -0500
Cc: [email-masked]
Sender: [email-masked]

On Fri November 5 2004 13:28, [email-masked] wrote:

> What's really needed is a generic way of computing a hash of a MIME object that
> takes as many of these issues as possible into account. I've had the
> specification of such a thing on my to-do list literally for years but I never
> seem to find the time to finish writing it up.
> 
> Basically what you want to do is define a hash methodology that computes
> separate hashes on leaf nodes in the MIME object and then combines those
> separate hashes along with hashes of canonicalized headers and the MIME
> structure itself in a specific way to arrive at a single result. The
> advantages of this approach are numerous:

1. The "combines" part is likely to pose some problems because there
    are some conflicting characteristics which one would like to have:
    * it should be easy to combine and separate the hashes
    * the result should be sensitive to reordering of objects
    * it should be possible to determine if some object and its
       corresponding hash has been removed (or if some content
       has been added)
    * for large objects, a hash represents data reduction; as the
       ratio of object size to hash length increases, there is a
       reduction in sensitivity (ability to detect changes). The act
       of combining hashes should not result in additional decrease
       in sensitivity
    * for small objects, a hash may be larger than the object itself;
       as a message is split into a greater number of smaller objects
       which are individually hashed, the total size of the hashes
       grows, resulting in undesirable increased overhead
2. Canonicalization of headers presents several problems:
    a) it presupposes that the syntax and structure of each field is known,
        which can become a snag as new fields are defined, when user-
        defined or experimental fields are used, as new protocol element
        keywords (charset names, media types and subtypes, etc.) are
        registered, etc.   A similar problem exists for RFC 2047 encoded-
        words; because they can appear only in certain contexts (some
        comments, in phrases, and in unstructured fields), one needs to
        know detailed field syntax  to determine if something which
        looks like an encoded-word is in fact an encoded-word.
   b)  Unfolding, normalization and compression of whitespace are
        probably reasonable for structured fields, but differences in line
        folding, tabs vs. spaces, and quantity of whitespace characters
       may be significant in some instances in unstructured fields
        (Subject, Comments, Content-Description, etc.).
   c) Some objects are case-sensitive, others case-insensitive; that
       should be taken into account during canonicalization. However,
       in some instances is is simply not possible to determine whether
       some field text is a case-insensitive object or not.

> (1) Encodings can be changed without breaking signatures. (This can help
>     with handling whitespace, and it makes it possible for signatures to
>     survive 8->7 conversion.)

Maybe. Change in encoding should result in change of
Content-Transfer-Encoding fields.  How would one handle that
change to the MIME-part header associated with the encoded
part, bearing in mind that once an object has been encoded, there
is no record of whether it was encoded from an original specified
as 8bit or as binary (which might or might not have been mislabeled)?

> (2) Boundary markers can be changed without breaking signatures. (How
>     to handle preamble and postamble text is an interesting side issue here.)

Would changing boundary markers not also change the MIME-part
header (boundary parameter in Content-Type field)?

[...]
> So, is it time for me to finish the specification for this?

I suspect that 2a above might be a show-stopper.

> Does anybody 
> care, and more to the point, will anybody actually implement it?

And if so, will there be multiple interoperable implementations (which
continue to interoperate as new fields, charsets, media types and
subtypes, etc. develop)?   Would such a method interoperate with
current methods?  How would the different method be indicated:
MIME-Version: 2.0?  Won't there still be problems if the original
isn't in canonical form when signed (and won't more complex
rules add to that problem)?  Won't there still be problems with
non-MIME-aware message handlers and with legacy MIME
implementations?

It's an interesting idea, but
a) I'm not convinced that all of the supposed benefits are real
b) I'm not convinced that it's practical w.r.t. continuing evolution
    of header fields and field components
c) It's unclear whether it's necessary (i.e. whether it will address
    the situations where the existing signature mechanisms fail)
and
d) I suspect that additional complexity will compound the
    problem.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Sat, 6 Nov 2004 04:23:51 -0500
Cc: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

On Fri November 5 2004 08:32, Nathaniel Borenstein wrote:
> 
> Let me be 
> the first to say that at least part of the problem here may be in the 
> MIME standard itself.
> 
> This situation could in theory be avoided by simply applying a suitable 
> content-transfer-encoding to the embedded message/rfc822 object 
> *before* PGP-signing it, thus protecting against line wrapping.  
> However, this is explicitly *prohibited* by the MIME standard, as per 
> RFC 2046:
> 
>  >￼￼￼￼￼   No encoding other than "7bit", "8bit", or "binary" is 
> permitted for
>  >   the body of a "message/rfc822" entity.
> 
> That may still have been the right design decision -- forbidding 
> protective C-T-E's for multipart and message makes message structure 
> much more transparent -- but as far as I can tell, there is NO 
> legitimate way to encode and sign a message/rfc822 object to protect 
> against line wrapping in the header fields of the embedded 
> message/rfc822 object.

There may be a way around that; simply define a discrete media
type that otherwise looks like message/rfc822 -- maybe
application/rfc822 a la message/http vs. application/http. That
discrete media type could then have protective encoding applied
for transport without violating the rule regarding identity
encodings of composite media types.  Of course that won't be
"transparent" in the same sense as message/rfc822.

To: [email-masked]
Cc: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822 
From: [email-masked]
Date: Fri, 05 Nov 2004 14:32:31 -0500
Sender: [email-masked]

--==_Exmh_1494463755P
Content-Type: text/plain; charset=us-ascii

On Fri, 05 Nov 2004 10:28:56 PST, [email-masked] said:

> This begs the question of how long the line actually was. IMO a line inside a
> multipart/signed that fits in the 998 SMTP limit should at least have had the
> option of not being rewrapped. If OTOH, the line was longer than the SMTP
> limit, then as I said before, I have no sympathy because this is yet another
> case of sending agent misbehavior.

The unwrapping was on a line of 190-240 or so characters into 3 <80 char lines..

--==_Exmh_1494463755P
Content-Type: application/pgp-signature

--==_Exmh_1494463755P--

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 14:09:01 -0500
Cc: [email-masked]
Sender: [email-masked]

On Fri November 5 2004 13:06, [email-masked] wrote:

> Even modulo the 'From ' breakage, a correctly calculated signature
> on canonical form won't survive software that decided to wrap a
> long single-line Recieved: and make it into 3 or 4 shorter lines...

True, but the invalid media content ("From " line and possibly
non-standard line endings) may have triggered the changes.

You haven't said which list was involved; certainly this mailing
list is able to handle signed messages:
http://www.imc.org/ietf-822/mail-archive/msg05007.html
is an HTML-munged version of a list message that survived expansion
with PGP signature intact.
 
> If such rewrapping is considered OK, then we really need to look
> at fixing the PGP signature algorithm to consider all spans of
> whitespace as equivalent (collapse down to single blank for signature
> purposes, maybe?)

It's probably not that simple; as far as header fields are concerned,
comments are semantically equivalent to whitespace -- that should
be considered as well. Arnt raises the issue of (RFC 2047) encoding
in human-readable filed components (comments, phrases, and
unstructured fields).  As far as body content is concerned, whether
or not "whitespace" is considered significant depends on the
specific media type -- it can certainly be argued that it might be
significant for application/octet-stream, and that the signature
should be sensitive to whitespace variations for that media type.
And Ned and I have raised the issues of legacy MIME implementations,
non-MIME implementations, and non-conforming implementations.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 13:57:26 -0500
Cc: [email-masked]
Sender: [email-masked]

On Fri November 5 2004 13:00, [email-masked] wrote:

> To the extent this is a problem in MIME itself, it is due to the fact that
> multipart/signed and the rules for handling it aren't in the base
> specification. This left the door open for perfectly valid implementations to
> completely ignore the need to handle mutipart/signed with the care it deserves.

Agreed.

> I will also point out that this sort of activity is closely related to gateway
> handling of MIME objects, which I wrote an RFC about: RFC 2480.

Thanks for the pointer.

> Finally, as far as lines over the SMTP limit of 998 are concerned, I have
> zero sympathy since it is technically illegal to send such a thing
> in the first place, irrespective of the MIME constructs involved. Anyone
> who expects overly long lines to be treated with fidelity is simply being
> silly and deserves whatever happens.

While I haven't counted octets, it certainly doesn't look like the
Received field that was folded was anywhere near 998 octets
in unfolded form.  I don't think that's an issue.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 12:21:20 -0500
Cc: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

On Fri November 5 2004 08:32, Nathaniel Borenstein wrote:
> 
> While I repeat that there is a surfeit of candidates at which to point 
> a finger in this case, we may not yet have gone deep enough.  Let me be 
> the first to say that at least part of the problem here may be in the 
> MIME standard itself.
> 
> This situation could in theory be avoided by simply applying a suitable 
> content-transfer-encoding to the embedded message/rfc822 object 
> *before* PGP-signing it, thus protecting against line wrapping.  
> However, this is explicitly *prohibited* by the MIME standard, as per 
> RFC 2046:
[...]
> That may still have been the right design decision -- forbidding 
> protective C-T-E's for multipart and message makes message structure 
> much more transparent -- but as far as I can tell, there is NO 
> legitimate way to encode and sign a message/rfc822 object to protect 
> against line wrapping in the header fields of the embedded 
> message/rfc822 object.  In other words, if you want to forward a full 
> message including long Received headers, and you want to sign the 
> forwarding message with PGP, you may be simply out of luck.

No, RFC 1847 is quite clear that multipart/signed (as used with
PGP per RFC 3156) should not be modified.  We can't tell for
certain whether the message described by Valdis was encapsulated
in multipart/signed (though I would guess that it might have been,
considering that his message describing the situation was a
multipart/signed message).

One problem is that really "ancient" MIME implementations
(predating RFC 1847) will treat multipart/signed as multipart/mixed,
and might modify content.  Non-MIME-aware implementations
of MSAs, MTAs, list expanders, MDAs, etc. likewise might not
handle multipart/signed in accordance with RFC 1847, and
then of course there is the distinct possibility of non-conforming
recent implementations which are intended to be MIME-aware
but which botch multipart/signed.  Returning to multipart/mixed,
the MIME specification doesn't explicitly state that either
multipart/mixed or unrecognized multipart subtypes treated as
mixed shouldn't be modified.  Is that a problem with the MIME
standard(s)?  Maybe.

I fully agree with your initial observation; there are so many
things that might have gone wrong, piled on top of at least
one thing that we can be sure that the sender did wrong, 
coupled with insufficient information (we don't know whether
the "message" was in canonical line-ending form before
signing, we don't know where the "From " line was elided
or where the Received field was unfolded, etc.), that we
can't definitively identify all of the problems. 

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Fri, 5 Nov 2004 12:27:37 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Fri November 5 2004 10:34, Charles Lindsey wrote:

> You said
> 
> >RFC 2822 id-right is a problem because it changes the semantics.
> 
> So what other change of semantics did you have in mind then?

As I said, it changes "domain" to "maybe a domain, maybe not,
with no way to tell".  That has many implications other than
case-insensitivity of domain names.

Date: Fri, 05 Nov 2004 10:28:56 -0800 (PST)
From: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
To: [email-masked]
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

> Even modulo the 'From ' breakage, a correctly calculated signature
> on canonical form won't survive software that decided to wrap a
> long single-line Recieved: and make it into 3 or 4 shorter lines...

This begs the question of how long the line actually was. IMO a line inside a
multipart/signed that fits in the 998 SMTP limit should at least have had the
option of not being rewrapped. If OTOH, the line was longer than the SMTP
limit, then as I said before, I have no sympathy because this is yet another
case of sending agent misbehavior.

> If such rewrapping is considered OK, then we really need to look
> at fixing the PGP signature algorithm to consider all spans of
> whitespace as equivalent (collapse down to single blank for signature
> purposes, maybe?)

What's really needed is a generic way of computing a hash of a MIME object that
takes as many of these issues as possible into account. I've had the
specification of such a thing on my to-do list literally for years but I never
seem to find the time to finish writing it up.

Basically what you want to do is define a hash methodology that computes
separate hashes on leaf nodes in the MIME object and then combines those
separate hashes along with hashes of canonicalized headers and the MIME
structure itself in a specific way to arrive at a single result. The
advantages of this approach are numerous:

(1) Encodings can be changed without breaking signatures. (This can help
    with handling whitespace, and it makes it possible for signatures to
    survive 8->7 conversion.)
(2) Boundary markers can be changed without breaking signatures. (How
    to handle preamble and postamble text is an interesting side issue here.)
(3) A message store can cache hash values for large leaf objects and use
    the cache to quickly sign very large messages.
(4) Some sorts of header mangling can be tolerated.

So, is it time for me to finish the specification for this? Does anybody
care, and more to the point, will anybody actually implement it?

				Ned

Date: Fri, 05 Nov 2004 10:00:12 -0800 (PST)
From: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
To: Nathaniel Borenstein <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, [email-masked], Tony Hansen <[email-masked]>, =?HZ-GB-2312?Q?Valdis_Kl~{=28%~}tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

> While I repeat that there is a surfeit of candidates at which to point
> a finger in this case, we may not yet have gone deep enough.  Let me be
> the first to say that at least part of the problem here may be in the
> MIME standard itself.

To the extent this is a problem in MIME itself, it is due to the fact that
multipart/signed and the rules for handling it aren't in the base
specification. This left the door open for perfectly valid implementations to
completely ignore the need to handle mutipart/signed with the care it deserves.

Like it or not, list processors are going to alter message content in 
a variety of ways. You can debate whether or not this is "right" or
"legal" or "correct" or whatever all you wish and it will change exactly
nothing.

The problem here is that a list processor has to make various choices when
processing content, and one of those choices should be how to balance the need
to alter content with the need to preserve signatures. An implementation that
is multipart/signed-aware can make that choice (although perhaps it won't make
it in the way you'd like), whereas one that treats multipart/signed as
multipart/mixed won't take the signature into consideration. And we allow
legitimate implementations to be multipart/signed-unaware. And worse still, it
is a much too late to change this rather basic bit of MIME. (I personally chalk
much of it up to a scheduling issue: The meetings where much of MIME's
structure was debated tended to conflict with the meetings on PEM, so a certain
corpus of security folk weren't present. Their presence wouldn't have led to
recognition of the value of multipart/signed - we spotted that fairly early on
- but might have led us to understand the need for this to be in the base
specification.)

I will also point out that this sort of activity is closely related to gateway
handling of MIME objects, which I wrote an RFC about: RFC 2480.

And FWIW, when I've implemented this stuff I let the system administrator
select whether or not multipart/signed objects are handled.

Finally, as far as lines over the SMTP limit of 998 are concerned, I have
zero sympathy since it is technically illegal to send such a thing
in the first place, irrespective of the MIME constructs involved. Anyone
who expects overly long lines to be treated with fidelity is simply being
silly and deserves whatever happens.

				Ned

To: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822 
From: [email-masked]
Date: Fri, 05 Nov 2004 13:06:20 -0500
Sender: [email-masked]

--==_Exmh_1427612844P
Content-Type: text/plain; charset=us-ascii

On Fri, 05 Nov 2004 08:06:34 EST, Bruce Lilly said:

> header fields makes no semantic difference.  Third, if one is using PGP
> as recommended (RFC 3156), refolding of a MIME-part field within
> a message/rfc822 part that is not within a signed multipart (RFC 1847)
> will have no effect on signature validity.

There *was* an enclosing signed multipart - which is how I found
out about the issue. The original structure was:

(rfc822 headers)
multipart/signed
  text/plain
  message/rfc822
     (rfc822 headers in question)
     multipart/signed
       text/plain
       application/pgp-signature
  text/plain
  application/pgp-signature

The 'headers in question' had 2 things done to them: (1) the removal
of a non-822 'From ' header and (2) the wrapping of long Received:
lines into multiple shorter lines.

The modifications to the message/rfc822 headers invalidated the
outer signature.  The inner signature survived just fine, as the
signature doesn't cover the inner message's headers.
    
> > 2) Am I at fault for including the mbox-style 'From ' line,
> 
> Yes, absolutely. 

OK.. I'll look at the MH code about fixing that one.

> message was signed and wrapped?  If a signature was
> improperly computed on content not in canonical form (CRLF
> line endings), it is virtually guaranteed to be invalid.

Even modulo the 'From ' breakage, a correctly calculated signature
on canonical form won't survive software that decided to wrap a
long single-line Recieved: and make it into 3 or 4 shorter lines...

If such rewrapping is considered OK, then we really need to look
at fixing the PGP signature algorithm to consider all spans of
whitespace as equivalent (collapse down to single blank for signature
purposes, maybe?)

--==_Exmh_1427612844P
Content-Type: application/pgp-signature

--==_Exmh_1427612844P--

To: LIST: [email-masked];
Xref: clerew local.mime:4131
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Fri, 5 Nov 2004 15:34:21 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Wed November 3 2004 06:25, Charles Lindsey wrote:
>> 
>> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
>> 
>> >On Mon November 1 2004 11:52, Charles Lindsey wrote:
>> 
>> >> You agreed that RFC 2822 had (intentionally or otherwise) changed the
>> >> semantics of <msg-id>s with regard to the case-sensitivity of whatever
>> >> came after the '@'
>> 
>> 
>> No, I am putting your own words back into your mouth.
>[...]
>> Which is exactly what I said you had said.

>No! I did not claim that RFC 2822 changed case-insensitivity
>requirements.

You said

>RFC 2822 id-right is a problem because it changes the semantics.

So what other change of semantics did you have in mind then?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 08:06:34 -0500
Cc: [email-masked]
Sender: [email-masked]

On Thu November 4 2004 14:54, [email-masked] wrote:
> Scenario:
> 
> I post to a mailing list, and include a multipart, one of
> which is a message/rfc822.  My MUA software takes the entire message
> as it lives in the message store, and sends it off.  When my copy
> comes back from the mailing list, the PGP signature is busticated.
> 
> Why?  Because the first few lines outbound looked like:
> 
> +++ Start outbound copy here
> ------- =_aaaaaaaaaa0
> Content-Type: message/rfc822
> Content-ID: <[email-masked]>
> Content-Description: forwarded message
> 
> From [email-masked]  Thu Nov  4 14:12:36 2004
> Return-Path: <[email-masked]>
> Received: from localhost (IDENT:[email-masked] [127.0.0.1])     by turing-police.cc.vt.edu (8.13.
> 1/8.13.1) with ESMTP id iA4JCU9n014549  for <[email-masked]>; Thu, 4 Nov 2004 14:12:35 -0500
> +++ end outbound copy

[...]

> 1) I am presuming that the mailing list is at fault for unwrapping the
> Received: line? (or more correctly, "the piece of software that did the
> unwrapping should be fixed")?

Not necessarily; in the first place, if the message is malformed (see below),
all bets are off. We can see that the message is indeed malformed in
one way, and we can surmise that it may be malformed in at least one
other important way. Secondly, in general unfolding or refolding of
header fields makes no semantic difference.  Third, if one is using PGP
as recommended (RFC 3156), refolding of a MIME-part field within
a message/rfc822 part that is not within a signed multipart (RFC 1847)
will have no effect on signature validity.  It is impossible to determine
from your snippet whether or not there was an enclosing signed
multipart, and you haven't said how "the PGP signature" was
computed or communicated.

> 2) Am I at fault for including the mbox-style 'From ' line,

Yes, absolutely.  What you have generated is in fact not a valid
message/rfc822 MIME-part.  There may well be other problems,
which cannot be determined from your snippet; for example, were
line endings CRLF pairs as required (RFCs 822, 2822, 2046, 2049
(and 1847 if multipart/signed was used)) before the alleged
message was signed and wrapped?  If a signature was
improperly computed on content not in canonical form (CRLF
line endings), it is virtually guaranteed to be invalid.

> or is 
> the mailing list (or whoever did it) at fault for removing it?

Unclear. The message was clearly malformed; it could have been
returned with some error message -- it appears that there was an
attempt to repair it (one cannot tell where from what you have
provided) instead. There are pros and cons to each approach.

Cc: [email-masked], Nathaniel Borenstein <[email-masked]>, =?UTF-8?Q?Valdis_Kl=C4=93tnieks?= <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 09:50:43 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>> As per my previous message, it's worse than this -- the From field is 
>> clearly a case of malformed content, but the long lines are not.  
>> Thus you can get some "glitches" even with perfectly valid messages.  
>> Sigh.
>
> In storage it can be even worse. Undo RFC2047/IDN encoding for 
> addresses to make them easier to search, and you immediately have a 
> signature verification problem. Haven't solved that one yet.

the undone encodings etc used for searching need to be stored 
out-of-band

Cc: [email-masked], =?HZ-GB-2312?Q?Valdis_Kl~{=28%~}tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 09:08:31 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

Sigh.  Too many levels of encoding/decoding, not enough clarity about a 
canonical form to sign.  Sigh.

On Nov 5, 2004, at 9:07 AM, Arnt Gulbrandsen wrote:

> Nathaniel Borenstein writes:
>> As per my previous message, it's worse than this -- the From field is 
>> clearly a case of malformed content, but the long lines are not.  
>> Thus you can get some "glitches" even with perfectly valid messages.  
>> Sigh.
>
> In storage it can be even worse. Undo RFC2047/IDN encoding for 
> addresses to make them easier to search, and you immediately have a 
> signature verification problem. Haven't solved that one yet.
>
> Arnt
>
>

Date: Fri, 5 Nov 2004 15:07:51 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Cc: Nathaniel Borenstein <[email-masked]>, Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

Nathaniel Borenstein writes:
> As per my previous message, it's worse than this -- the From field is 
> clearly a case of malformed content, but the long lines are not.  
> Thus you can get some "glitches" even with perfectly valid messages.  
> Sigh.

In storage it can be even worse. Undo RFC2047/IDN encoding for addresses 
to make them easier to search, and you immediately have a signature 
verification problem. Haven't solved that one yet.

Arnt

Cc: [email-masked], =?HZ-GB-2312?Q?Valdis_Kl~{=28%~}tnieks?= <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 08:57:58 -0500
To: Keith Moore <[email-masked]>
Sender: [email-masked]

On Nov 5, 2004, at 8:37 AM, Keith Moore wrote:

> Intermediaries should try to be as transparent as possible.  But 
> because of 8-to-7 conversion if nothing else, message/rfc822 and 
> multiparts are special.  If you send messages with either of these 
> content-types and the content is malformed, you're going to get some 
> glitches.

As per my previous message, it's worse than this -- the From field is 
clearly a case of malformed content, but the long lines are not.  Thus 
you can get some "glitches" even with perfectly valid messages.  Sigh.

Cc: [email-masked], Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, =?UTF-8?Q?Valdis_Kl=C4=93tnieks?= <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 08:42:10 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>> But IMO, the list processor software is a recipient as far as the 
>> internet mail system is concerned. (You get a final delivery DSN when 
>> the message is delivered to the mailing list processor.)
>
> Let me put it differently.
>
> A sender should either expect the recipient to use the label, or else 
> put in application/octet-stream.

no, I don't think so.  the sender should label content clearly, so as 
to inform the recipient and his MUA. the recipient and his MUA then 
make use of that information in deciding what to do with that 
attachment.

> Saying things like "the label is useful and should be there, but only 
> human recipients should use it, programs should ignore it" is IMO 
> nonsensical.

I don't think anyone's saying that.  clearly the label is there so that 
the recipient's MUA can use it.

Keith

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Understanding response protocols
Date: Fri, 5 Nov 2004 08:39:00 -0500
To: Martin Trautmann <[email-masked]>
Sender: [email-masked]

>> I much prefer those lists which set Reply-To (I have been on lists 
>> where
>> the members have voted to have it set so), even though I have to 
>> remember
>> which list is which in order to press the right Reply button.
>
> What happens on these lists when the sender defined a Reply-To of his 
> own?
> I guess it's just replaced?

depends on the list.  with mailman, for instance, this is a per-list 
settable option.

Cc: [email-masked], =?UTF-8?Q?Valdis_Kl=C4=93tnieks?= <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 08:37:42 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

> The RFC seems clear: signatures apply to the message as transmitted 
> over the wire and changing bytes in a signed part of a message is a 
> no-no.
>
> But I'm unhappy about that RFC. For example: If a mailing list 
> processor is coupled with an MTA that accepts long lines, the list 
> processor can be left with the task of transmitting a 5000-character 
> line to all list subscribers.

As long as the mailing list processor doesn't alter long lines, I don't 
think that's such a problem.  The recipients that had a problem 
receiving or reading the message from the list would also have the same 
problem reading that message if it were sent directly to them.  It's 
not the list's job to "improve" the message.

>> 2) Am I at fault for including the mbox-style 'From ' line, or is the 
>> mailing list (or whoever did it) at fault for removing it?
>
> Those are two questions. The first is clear: Anything a sending 
> program labels as message/rfc822 should have RFC[2]822 syntax, not 
> berkeley mbox syntax. The second is IMO arguable and less important.

Intermediaries should try to be as transparent as possible.  But 
because of 8-to-7 conversion if nothing else, message/rfc822 and 
multiparts are special.  If you send messages with either of these 
content-types and the content is malformed, you're going to get some 
glitches.

Cc: [email-masked], Tony Hansen <[email-masked]>, =?HZ-GB-2312?Q?Valdis_Kl~{=28%~}tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 08:32:23 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

While I repeat that there is a surfeit of candidates at which to point 
a finger in this case, we may not yet have gone deep enough.  Let me be 
the first to say that at least part of the problem here may be in the 
MIME standard itself.

This situation could in theory be avoided by simply applying a suitable 
content-transfer-encoding to the embedded message/rfc822 object 
*before* PGP-signing it, thus protecting against line wrapping.  
However, this is explicitly *prohibited* by the MIME standard, as per 
RFC 2046:

 >￼￼￼￼￼   No encoding other than "7bit", "8bit", or "binary" is 
permitted for
 >   the body of a "message/rfc822" entity.

That may still have been the right design decision -- forbidding 
protective C-T-E's for multipart and message makes message structure 
much more transparent -- but as far as I can tell, there is NO 
legitimate way to encode and sign a message/rfc822 object to protect 
against line wrapping in the header fields of the embedded 
message/rfc822 object.  In other words, if you want to forward a full 
message including long Received headers, and you want to sign the 
forwarding message with PGP, you may be simply out of luck.

The best solution that comes to mind is to revise the PGP signature 
computation to be tolerant of inserted white space, at least in the 
headers of embedded message objects.  -- Nathaniel

On Nov 5, 2004, at 8:14 AM, Arnt Gulbrandsen wrote:

>
> Arnt Gulbrandsen writes:
>> But IMO, the list processor software is a recipient as far as the 
>> internet mail system is concerned. (You get a final delivery DSN when 
>> the message is delivered to the mailing list processor.)
>
> Let me put it differently.
>
> A sender should either expect the recipient to use the label, or else 
> put in application/octet-stream.
>
> Saying things like "the label is useful and should be there, but only 
> human recipients should use it, programs should ignore it" is IMO 
> nonsensical. I hope I'm not erecting strawmen.
>
> Arnt
>
>
>

Date: Fri, 5 Nov 2004 14:26:34 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Cc: Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
>>> no.  you shouldn't be expected to hide useful information for the 
>>> recipient just so that the mail system will be transparent.
>>
>> Agreed. But IMO, the list processor software is a recipient as far as 
>> the internet mail system is concerned. (You get a final delivery DSN 
>> when the message is delivered to the mailing list processor.)
>
> okay, I'll grant that.  but if the list processor is going to alter 
> the message, it needs to consider the effect this has on digital 
> signatures.

For a valid message I certainly agree. But I think the case is far 
weaker for a syntactically invalid message.

If it works, fine. If not, that's a pity. Maybe add another test case if 
the problem seems worth a workaround.

Arnt

Cc: [email-masked], Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, =?UTF-8?Q?Valdis_Kl=C4=93tnieks?= <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 08:11:38 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>> no.  you shouldn't be expected to hide useful information for the 
>> recipient just so that the mail system will be transparent.
>
> Agreed. But IMO, the list processor software is a recipient as far as 
> the internet mail system is concerned. (You get a final delivery DSN 
> when the message is delivered to the mailing list processor.)

okay, I'll grant that.  but if the list processor is going to alter the 
message, it needs to consider the effect this has on digital 
signatures.

>> there are plenty of reasons for looking at a message to decide 
>> whether to bounce or filter it.  there are far fewer valid reasons 
>> for altering a message.
>
> Converting to internal format and then back can transform invalid 
> input into valid output. Is that bad?

in general, yes.  the goal of the mail transport (even of lists, IMHO) 
is not to produce "valid" output, but to convey the message that the 
sender sent.  for instance, MTAs that alter dates because they are not 
invalid format can cause the message to appear that it was sent at a 
different date and time than it really was.

Date: Fri, 5 Nov 2004 14:14:14 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Cc: Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

Arnt Gulbrandsen writes:
> But IMO, the list processor software is a recipient as far as the 
> internet mail system is concerned. (You get a final delivery DSN when 
> the message is delivered to the mailing list processor.)

Let me put it differently.

A sender should either expect the recipient to use the label, or else 
put in application/octet-stream.

Saying things like "the label is useful and should be there, but only 
human recipients should use it, programs should ignore it" is IMO 
nonsensical. I hope I'm not erecting strawmen.

Arnt

Date: Fri, 5 Nov 2004 14:05:31 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Cc: Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
>>> yeah, maybe I misunderstood, but my take was that the problem was 
>>> caused by the some intermediary which did not treat the 
>>> message/rfc822 body part as an opaque object.
>>
>> If the body part were meant to be treated as an opaque object, it 
>> should have been labelled application/octet-stream.
>
> no.  you shouldn't be expected to hide useful information for the 
> recipient just so that the mail system will be transparent.

Agreed. But IMO, the list processor software is a recipient as far as 
the internet mail system is concerned. (You get a final delivery DSN 
when the message is delivered to the mailing list processor.)

> there are plenty of reasons for looking at a message to decide whether 
> to bounce or filter it.  there are far fewer valid reasons for 
> altering a message.

Converting to internal format and then back can transform invalid input 
into valid output. Is that bad?

(Wasn't there a similar thread years ago about lone CRs/LFs transformed 
into CRLF?)

Arnt

Cc: [email-masked], Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, =?UTF-8?Q?Valdis_Kl=C4=93tnieks?= <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Fri, 5 Nov 2004 07:25:55 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>> yeah, maybe I misunderstood, but my take was that the problem was 
>> caused by the some intermediary which did not treat the 
>> message/rfc822 body part as an opaque object.
>
> If the body part were meant to be treated as an opaque object, it 
> should have been labelled application/octet-stream.

no.  you shouldn't be expected to hide useful information for the 
recipient just so that the mail system will be transparent.

> IMO, there are many good reasons for a mailing list processor to look 
> closely at the mime parts it receives. For example, it should try to 
> not retransmit bounces, virii, etc. Looking very closely at 
> message/rfc822 parts is a defensible choice.

there are plenty of reasons for looking at a message to decide whether 
to bounce or filter it.  there are far fewer valid reasons for altering 
a message.

Date: Fri, 5 Nov 2004 12:47:09 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Cc: Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>
Sender: [email-masked]

Valdis Klētnieks writes:
> 1) I am presuming that the mailing list is at fault for unwrapping the 
> Received: line? (or more correctly, "the piece of software that did 
> the unwrapping should be fixed")?

This issue is very troublesome, to me at least.

The RFC seems clear: signatures apply to the message as transmitted over 
the wire and changing bytes in a signed part of a message is a no-no.

But I'm unhappy about that RFC. For example: If a mailing list processor 
is coupled with an MTA that accepts long lines, the list processor can 
be left with the task of transmitting a 5000-character line to all list 
subscribers. I've run into quite a few other similar problems.

> 2) Am I at fault for including the mbox-style 'From ' line, or is the 
> mailing list (or whoever did it) at fault for removing it?

Those are two questions. The first is clear: Anything a sending program 
labels as message/rfc822 should have RFC[2]822 syntax, not berkeley 
mbox syntax. The second is IMO arguable and less important.

Arnt

Date: Fri, 5 Nov 2004 12:19:20 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Cc: Tony Hansen <[email-masked]>, Nathaniel Borenstein <[email-masked]>, Valdis =?utf-8?q?Kl=C4=93tnieks?= <[email-masked]>, Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
> yeah, maybe I misunderstood, but my take was that the problem was 
> caused by the some intermediary which did not treat the 
> message/rfc822 body part as an opaque object.

If the body part were meant to be treated as an opaque object, it should 
have been labelled application/octet-stream.

IMO, there are many good reasons for a mailing list processor to look 
closely at the mime parts it receives. For example, it should try to 
not retransmit bounces, virii, etc. Looking very closely at 
message/rfc822 parts is a defensible choice.

Arnt

Date: Fri, 5 Nov 2004 11:52:06 +0100
From: Martin Trautmann <[email-masked]>
Cc: LIST:  [email-masked];
Subject: Re: Angle brackets surrounding Content-ID
Sender: [email-masked]

On 2004-11-01 17:09, Charles Lindsey wrote:
> >Why can't it be mbox files? There's lots of postings that I archive 
> >locally. It looks to me like a news article in mbox format. What's the 
> >difference?
> 
> An mbox file contains '^From�' lines that contain the envelope From
> address, plus a time stamp. Since news articles don't have any envelope,
> they cannot be stored in pure mbox files.

Maybe - my newsreader just saves this info before, e.g.

 From [email-masked] Fri Nov  5 11:39:39 2004
 Newsgroups: <any>
 From: Martin Trautmann <[email-masked]>
 Subject: ...

It take the address from From: and the time stampe while actually saving
the posting

> The problem arises in User Agents that try to overload one of the same two
> buttons "Reply" and "Reply-to-All" to double up as the "Followup" button
> when operating in News mode (a practice strongly deprecated by the GNKSA,
> but prevalent in many 2nd rate agents anyway).

Then it's more a question of poor user interface, where the agent trained
the user to use 'reply/reply to all', while it did not teach the
difference between postings and mail, between follow-up and reply

Date: Fri, 5 Nov 2004 11:45:47 +0100
From: Martin Trautmann <[email-masked]>
Cc: LIST:  [email-masked];
Subject: Re: Understanding response protocols
Sender: [email-masked]

On 2004-11-01 17:48, Charles Lindsey wrote:
> I much prefer those lists which set Reply-To (I have been on lists where
> the members have voted to have it set so), even though I have to remember
> which list is which in order to press the right Reply button.

What happens on these lists when the sender defined a Reply-To of his own?
I guess it's just replaced?

Date: Fri, 5 Nov 2004 10:27:04 +0000
From: Tony Finch <[email-masked]>
To: Tony Hansen <[email-masked]>
cc: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Sender: [email-masked]

On Thu, 4 Nov 2004, Tony Hansen wrote:
>
> Both From_ and Return-Path: are considered to be relics of the storage format,
> and not part of the message. (The ill-fated Content-Length: is a similar type
> of header.)

I don't think that Return-Path: can be considered a relic of the storage
format. It's standardized in all the relevant RFCs (821,822,2821,2822) so
isn't specific to any particular storage format. It's slightly unusual
that it's only added at final delivery, but it's important to preserve it
after that because it holds information that wouldn't otherwise be
available.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
MALIN HEBRIDES: NORTHEAST 4 OR 5 INCREASING 6. RAIN LATER. GOOD BECOMING
MODERATE.

Cc: Tony Hansen <[email-masked]>, [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Thu, 4 Nov 2004 23:42:55 -0500
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

> This may seem like splitting hairs, because I agree that the sender 
> caused the initial problem, but are you saying that because the 
> message/rfc822 object was incorrectly formatted (because of the 
> inclusion of the From line), the downstream software was justified in 
> throwing that offending line away?  I'm skeptical because it doesn't 
> actually make anything better, and it seems to me that "correcting" 
> the contents is far more likely to cause harm than good.  -- Nathaniel

yeah, maybe I misunderstood, but my take was that the problem was 
caused by the some intermediary which did not treat the message/rfc822 
body part as an opaque object.  that seems like a bug regardless of 
whether the From_ line is there.  though I suppose if it went through 
an 8bit to 7bit conversion the intermediary would be expected to 
convert any internal message/rfc822 body parts to  7bit also, and such 
a conversion might be confused by a From_ line.

Date: Thu, 04 Nov 2004 23:40:08 -0500
From: Tony Hansen <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
CC: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Sender: [email-masked]

Nathaniel Borenstein wrote:

> This may seem like splitting hairs, because I agree that the sender 
> caused the initial problem, but are you saying that because the 
> message/rfc822 object was incorrectly formatted (because of the 
> inclusion of the From line), the downstream software was justified in 
> throwing that offending line away?  I'm skeptical because it doesn't 
> actually make anything better, and it seems to me that "correcting" the 
> contents is far more likely to cause harm than good.  -- Nathaniel

When you step outside the bounds of defined behavior, what's the right 
thing to do?

Postel's Maxim is a two edged sword. Yes, the mailing list server should 
accept the message. But if it cares about sending out correct email, 
it's facing a dilemna. What should it do?

Obviously, the programmer in this situation must make some choices. S/he 
could do nothing, or s/he could fix up the message somehow. And if s/he 
chooses to fix up the message, what's the right fix? Certainly, all of 
those choices are justifiable.

	Tony Hansen
	[email-masked]

Cc: [email-masked], [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Subject: Re: 2 MIME questions re: message/rfc822
Date: Thu, 4 Nov 2004 22:31:01 -0500
To: Tony Hansen <[email-masked]>
Sender: [email-masked]

This may seem like splitting hairs, because I agree that the sender 
caused the initial problem, but are you saying that because the 
message/rfc822 object was incorrectly formatted (because of the 
inclusion of the From line), the downstream software was justified in 
throwing that offending line away?  I'm skeptical because it doesn't 
actually make anything better, and it seems to me that "correcting" the 
contents is far more likely to cause harm than good.  -- Nathaniel

On Nov 4, 2004, at 9:53 PM, Tony Hansen wrote:

>
> [email-masked] wrote:
>
>> Scenario:
>> I post to a mailing list, and include a multipart, one of
>> which is a message/rfc822.  My MUA software takes the entire message
>> as it lives in the message store, and sends it off.  When my copy
>> comes back from the mailing list, the PGP signature is busticated.
>> Why?  Because the first few lines outbound looked like:
>> +++ Start outbound copy here
>> ------- =_aaaaaaaaaa0
>> Content-Type: message/rfc822
>> Content-ID: <[email-masked]>
>> Content-Description: forwarded message
>> From [email-masked]  Thu Nov  4 14:12:36 2004
>> Return-Path: <[email-masked]>
>> Received: from localhost (IDENT:[email-masked] 
>> [127.0.0.1])     by turing-police.cc.vt.edu (8.13.
>> 1/8.13.1) with ESMTP id iA4JCU9n014549  for 
>> <[email-masked]>; Thu, 4 Nov 2004 14:12:35 -0500
>> +++ end outbound copy
>> (Yes, one long line - that was done by a local procmail rule).
>> +++ Start as-received here
>> ------- =_aaaaaaaaaa0
>> Content-Type: message/rfc822
>> Content-ID: <[email-masked]>
>> Content-Description: forwarded message
>> Return-Path: <[email-masked]>
>> Received: from localhost (IDENT:[email-masked] 
>> [127.0.0.1]) by
>>         turing-police.cc.vt.edu (8.13.1/8.13.1) with ESMTP id
>>         iA4JCU9n014549 for <[email-masked]>;
>>         Thu, 4 Nov 2004 14:12:35 -0500
>> +++ End as-received
>> 1) I am presuming that the mailing list is at fault for unwrapping the
>> Received: line? (or more correctly, "the piece of software that did 
>> the
>> unwrapping should be fixed")?
>
> I'll let others comment on this question.
>
>> 2) Am I at fault for including the mbox-style 'From ' line, or is
>> the mailing list (or whoever did it) at fault for removing it?
>
> Yes, you are wrong to include the From_ pseudo-header. It is NOT a 
> line that follows the ABNF for a header field. Since it's a broken 
> header starting the message, the receiving software is free to do 
> whatever it wants with the non-message. For example, the From_ line 
> could be treated as the first line of the message body.
>
> Both From_ and Return-Path: are considered to be relics of the storage 
> format, and not part of the message. (The ill-fated Content-Length: is 
> a similar type of header.)
>
> 	Tony Hansen
> 	[email-masked]
>
>
>

Date: Thu, 04 Nov 2004 21:53:51 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: 2 MIME questions re: message/rfc822
Sender: [email-masked]

[email-masked] wrote:

> Scenario:
> 
> I post to a mailing list, and include a multipart, one of
> which is a message/rfc822.  My MUA software takes the entire message
> as it lives in the message store, and sends it off.  When my copy
> comes back from the mailing list, the PGP signature is busticated.
> 
> Why?  Because the first few lines outbound looked like:
> 
> +++ Start outbound copy here
> ------- =_aaaaaaaaaa0
> Content-Type: message/rfc822
> Content-ID: <[email-masked]>
> Content-Description: forwarded message
> 
> From [email-masked]  Thu Nov  4 14:12:36 2004
> Return-Path: <[email-masked]>
> Received: from localhost (IDENT:[email-masked] [127.0.0.1])     by turing-police.cc.vt.edu (8.13.
> 1/8.13.1) with ESMTP id iA4JCU9n014549  for <[email-masked]>; Thu, 4 Nov 2004 14:12:35 -0500
> +++ end outbound copy
> 
> (Yes, one long line - that was done by a local procmail rule).
> 
> +++ Start as-received here
> ------- =_aaaaaaaaaa0
> Content-Type: message/rfc822
> Content-ID: <[email-masked]>
> Content-Description: forwarded message
> 
> Return-Path: <[email-masked]>
> Received: from localhost (IDENT:[email-masked] [127.0.0.1]) by
>         turing-police.cc.vt.edu (8.13.1/8.13.1) with ESMTP id
>         iA4JCU9n014549 for <[email-masked]>;
>         Thu, 4 Nov 2004 14:12:35 -0500
> +++ End as-received
> 
> 1) I am presuming that the mailing list is at fault for unwrapping the
> Received: line? (or more correctly, "the piece of software that did the
> unwrapping should be fixed")?

I'll let others comment on this question.

> 2) Am I at fault for including the mbox-style 'From ' line, or is
> the mailing list (or whoever did it) at fault for removing it?

Yes, you are wrong to include the From_ pseudo-header. It is NOT a line 
that follows the ABNF for a header field. Since it's a broken header 
starting the message, the receiving software is free to do whatever it 
wants with the non-message. For example, the From_ line could be treated 
as the first line of the message body.

Both From_ and Return-Path: are considered to be relics of the storage 
format, and not part of the message. (The ill-fated Content-Length: is a 
similar type of header.)

	Tony Hansen
	[email-masked]

Return-Path: <[email-masked]>
Received: from localhost (IDENT:[email-masked] [127.0.0.1])     by turing-police.cc.vt.edu (8.13.
1/8.13.1) with ESMTP id iA4JCU9n014549  for <[email-masked]>; Thu, 4 Nov 2004 14:12:35 -0500
+++ end outbound copy

(Yes, one long line - that was done by a local procmail rule).

+++ Start as-received here
------- =_aaaaaaaaaa0
Content-Type: message/rfc822
Content-ID: <[email-masked]>
Content-Description: forwarded message

Return-Path: <[email-masked]>
Received: from localhost (IDENT:[email-masked] [127.0.0.1]) by
        turing-police.cc.vt.edu (8.13.1/8.13.1) with ESMTP id
        iA4JCU9n014549 for <[email-masked]>;
        Thu, 4 Nov 2004 14:12:35 -0500
+++ End as-received

1) I am presuming that the mailing list is at fault for unwrapping the
Received: line? (or more correctly, "the piece of software that did the
unwrapping should be fixed")?

2) Am I at fault for including the mbox-style 'From ' line, or is
the mailing list (or whoever did it) at fault for removing it?

--==_Exmh_-2116978169P
Content-Type: application/pgp-signature

--==_Exmh_-2116978169P--

To: [email-masked]
Subject: 2 MIME questions re: message/rfc822
From: [email-masked]
Date: Thu, 04 Nov 2004 14:54:02 -0500
Sender: [email-masked]

--==_Exmh_-2116978169P
Content-Type: text/plain; charset=us-ascii

Scenario:

I post to a mailing list, and include a multipart, one of
which is a message/rfc822.  My MUA software takes the entire message
as it lives in the message store, and sends it off.  When my copy
comes back from the mailing list, the PGP signature is busticated.

Why?  Because the first few lines outbound looked like:

+++ Start outbound copy here
------- =_aaaaaaaaaa0
Content-Type: message/rfc822
Content-ID: <[email-masked]>
Content-Description: forwarded message

Date: Thu, 4 Nov 2004 11:49:11 -0500
From: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Archives
Sender: [email-masked]

> > > > being able to store lots of messages in a single file can save a
> > > > lot of disk space, particularly if you compress those mbox files.
> > > 
> > > I suppose that's so in general. However for the problem
> > > domain (a server), disk space with modern hardware
> > > isn't generally an issue.  If it is deemed to be a greater
> > > issue than compression/decompression overhead for a
> > > particular instance, compression can be implemented in
> > > the file system, independent of storage format.
> > 
> > that's much more difficult to do.  the point is to make this easy.
> 
> More difficult for whom? 

for the server. 

> On an OS that supports compressed
> filesystems, FS compression is transparent to applications
> (other than performance issues) [and there are OSes that
> support FS compression].  On the other hand, if compression
> is implemented at the application level, every application that
> accesses the files has to be able to use the same type of
> compression and decompression, and has to be explicitly
> coded to do so. 

yes that's true but it doesn't seem to be a huge problem in practice.

> Where multiple access methods are used,
> that means that it has to be handled by ftp servers, http
> servers, etc. as well as support tools (does "ucbmail" support
> editing of compressed mbox files?).

we don't need to concern ourselves with solutions that are 
confined to the server.  what we need to do is define the interface
between the client and the server, such that there is a reasonable
balance of burden between the client and the server.   so if I
were to propose a spec that allowed mail archives to be compressed,
I'd assume that the client was going to do the decompression.
otherwise it's purely a server-side issue and not the client's
concern.  at this point I'm nowhere nearly ready to propose a spec,
I'm just thinking about it.  it's not at all clear to me where
the right tradeoff is yet.

> > > I think one of the biggest practical issues with Archived-At
> > > support is arranging for the URI(s) to be put into the
> > > message before the message is placed in the archive; the
> > > URI (related to archive file name and/or UID) might not
> > > be determinate until the message is actually placed in the
> > > archive.
> > 
> > that's fairly easy to solve.  send the messages to the archiver 
> > before distributing them to list subscribers.
> > 
> > 
> >               +----------+       +----------+       +----------+
> > incoming      | incoming |       |          |       |   list   |
> > message ----> |   mail   | ----> | archiver | ----> | expander |
> >               |  filter  |       |          |       |          |
> >               +----------+       +----------+       +----------+
> >                     |                  |                  |
> >                     V                  V                  V
> >                   rejects          archived          subscribers
> >                                    messages
> 
> That doesn't address the issue of how the URIs that are to be put
> in Archived-At fields are generated before the archive file path
> and/or UID are known, nor how the field gets inserted in the
> message.  Sure, list expansion can happen after archival; that's
> the easy part.

the archiver determines the archive file path and adds the archived-at
field to the message before sending it to the list expander.

> > either that or base the name of the archive on some property of
> > the message that can be derived independently, say the message-id
> > or a hash of some kind.
> 
> "The name of the archive" may well be different for different
> access schemes 

because the "name" is a URL, it _will_ be different for different 
access schemes.  I'm expecting that the archiver will have to know
about the access methods that are available.

Keith

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: a header authentication scheme
Date: Thu, 4 Nov 2004 10:21:13 -0500
Sender: [email-masked]

On Mon November 1 2004 23:36, Laird Breyer wrote:
> 
> On Nov 01 2004, Bruce Lilly wrote:
> > 
> > You have missed the points; at any point a party can
> > a) s/result-tag="spam"/result-tag="ham"/
> > b) insert a Received field, then insert a "processed" field referring
> >     to that Received field, with any desired "processed" field content
> > N.B. neither tactic would be effective if MIME security multiparts
> > were used.
> 
> Perhaps you have a concrete example where you think a) or b) occurs,

"Occurs" implies that this happens on a regular basis, and as we're
discussing a hypothetical field which has not been completely
defined, much less deployed, that term is inappropriate.

> the only examples I can think of are compromised systems, for which
> the problem is moot.

No. I have not ascribed any motives, but it should be easy to
read between the lines and figure out why such things might
happen for reasons other than "compromised systems", i.e. a
field might be intentionally forged by a party involved in
message transfer.  Case a was an example;
s/result-tag="ham"/result-tag="spam" should be the obvious
alternative example of what could happen.  Now parties
responsible for message transfer ought not to alter content,
but it does happen (and in fact the premise of the present
discussion is that that is to be encouraged by inserting these
"processed" fields hither and yon).  Ethical conduct would
preclude such tampering, but ethical (moral, if you prefer)
conduct seems to be the exception rather than the rule these
days (despite claims to the contrary by the unethical parties,
which is hardly surprising).  It is not difficult to envision a
commercial interest inserting a field declaring competitors'
messages to be spam. Likewise for political, religious, and/or
governmental organizations with a vested interest in curtailing
the free exchange of ideas.  In many parts of the world,
Internet Service Providers who routinely transport messages
are either commercial entities (often with a diverse set of
business interests, and therefore many competitors) or are
governmental entities.

> a) requires modifying some existing Processed field. No well behaved
> filter needs to modify existing fields in a message.

Since we're discussing spam and intentional disinformation,
we're obviously not talking about "well behaved" anything.
If everybody and everything under human control were
"well behaved", there wouldn't be a spam problem in the first
place.  You seem to be assuming that there is only one "bad
guy", and that everybody else is always a "good guy". Sorry,
reality isn't that simple, nor is it that black-and-white.

> If a modification 
> occurs, then the process doing this modification is doing something
> prohibited (tampering with the message),

Agreed. So what? Individuals, companies, political parties,
religious organizations, and governments do prohibited
(and/or allowed but unethical) things all the time.  Don't
be nieve.

> and the computer system which 
> is running this process has therefore been hacked

Incorrect conclusion -- it may be intentional on the part of
the system owner/operator.

> or belongs to an 
> untrustworthy party.

Do you trust everybody involved in the transmission of every
message?  Why would you?

> b) requires inserting a new Received field.

So what? De rigeur for SMTP handling and not prohibited
otherwise.

> The sequence of Received 
> fields added close to the message destination is predictable by the
> recipient,

No. Certainly not so for non-SMTP transfer. And you have
provided examples of non-Internet systems where header field
order is indeterminate.

> and can only be changed by an attacker if one of the 
> computers in the recipient's organization is compromised.

No. First, the very last system to have inserted a Received field
might not be "in the recipient's organization"; it might belong
to a commercial or government-run service provider. Second,
as explained above, it is not necessary that such a system be
compromised; the change may be deliberate.
 
> So we always have a model of message transport as follows:
> 
> A -> M1 -> M2 -> M3 -> ... -> T1 -> T2 -> ... -> B

Not necessarily.  There may well be multiple paths to B, N-1
of which do not pass through "T2".  "T1", "T2", etc. might
pass some messages w/o change, but may change some
fraction of messages (i.e. those messages not sympathetic
to the interests of the owner/operators of those systems).
 
> A is the sender of the message, B is the destination, M represents
> computers which are untrusted by B, and T represents computers which
> are trusted by B and add a Received field.  None of the computers of
> type T will ever perform either a) or b) above, unless they have been
> compromised.

No, repeating "unless they have been compromised" doesn't
make it true. 

> Computers of type M may well perform a) or b). It is up 
> to B to only trust Processed fields associated with computers of type
> T.

You are assuming an extraordinarily simplistic model which
simply doesn't conform to reality; a model where there are
only "M" and "T" and where each system is either one or the
other, invariant with message content and over time, with no
message transport paths through other mechanisms.

> If the recipient's MTA doesn't add a Received field of its own (ie
> there are no T type computers at all), then the recipient B must already know
> not to trust any Received fields at all in a message.

Again you are ignoring multiple transport paths and assuming
a black-and-white trust model which is invariant over time and
with message content.

> But are you really saying that a given user B
> cannot, by examining a sample of past mail messages he personally received,
> deduce the form of the Received fields added by the computers I've labeled
> T above?

It certainly depends somewhat on B's knowledge and experience.
I would estimate that for > 99.9% of email recipients, a *valid*
deduction would be extremely unlikely; probably 99.5% have no
clue what a Received field is in the first place.  Judging by illegal
syntax in many generated Received fields, it's clear that a large
fraction of those who think that they know what a Received
field is are simply wrong.  On top of that, there are hostname
aliases, domain literals, etc. which require an additional
understanding of the DNS.

> I must admit I find that hard to believe. 

Conduct a survey of a random sample of people who receive
email and convince yourself.
 
> > For a concrete example, suppose that some recipient
> > is interested in a home mortgage.  A spam filter that is configured
> > to treat all messages containing the word "mortgage" as spam is
> > worse than useless to that recipient; it is counter-productive. And
> > on the basis of the (lack of) information in your proposed
> > "processed" field, no recipient is able to determine whether or not
> > configuration of any alleged process is appropriate *for him*.
> 
> Obviously such a user will learn to not accept the supplied spam label
> as authoritative. He will either ignore the Processed field's
> recommendation completely, or configure his other filters if he has
> any to lower the weight of that particular field so as to let through
> mortgage mails but not other spam.

What "*other* spam"?  The premise is that for that recipient, that
message isn't spam.  "Other" doesn't apply.  How is the user
supposed to be able to differentiate those "processed" fields
from any others -- does he magically acquire an understanding
of Received fields and "processed" field by osmosis in his sleep,
is he able to compute hashes in his head, has he memorized the
entire worldwide DNS database and does he telepathically receive
updates?  Ignoring all that, suppose that the hypothetical
filter is 100% accurate for that recipient except for messages
containing the specific keyword in question.  In order to
determine the validity of any ostensible value judgment made by
that remote filter -- if the message is received in spite of the supposed
"filtering" -- , the user will have to apply a local filter to first
determine whether that keyword is present in the message.
If a local filter is therefore going to have to be used, there is little
point in having a remote filter, a black-and-white time- and
content-invariant trust model, "processed" fields, etc.
  
> For example, statistical content filters already perform this exact 
> adjustment automatically, ie giving more weight to tokens which correlate
> well with the user's personal idea of spam say, and reducing the weight of
> tokens which don't.

But if the user has to run a local filter in order to determine whether
or not to trust a remote filter, what is the point of having the
remote filter in the first place?

> This is done by asking users to mark incorrectly 
> filtered messages, i.e. without the need to understand the internals of the 
> particular spam filter which marks all messages containg "mortgage" as spam.

If messages are in fact being filtered, i.e. if messages judged to be
"spam" are not sent to the user, then there is no way for the user to
handle false positives.  If there is in fact no filtering, then the user's
problem is simply compounded; he not only has to deal with the
transmission bandwidth and bulk of the spam, but also of the added
"processed" fields.
 
> > An assertion made by some
> > intermediate system which is supposed to be providing transport
> > services simply fails to fit into the Internet Architecture on many
> > levels:
> > a) is is not an end-to-end process
> > b) eavesdropping on private communications is immoral and
> >     offensive (and may be illegal in some places)
> > c) it is not under control of either party in the end-to-end
> >     communication
> > etc.
> 
> Would you say that current mail practices such as providing spam filtering
> fit into the Internet Architecture described by RFC 1958 ? 

Filtering that takes place at the receiving endpoint is consistent
with the Internet Architecture.
 
> For example, a) is broken by spam filters set at the level of SMTP
> servers rather than filters set up at the individual MUA level.

Not if the server and its associated filter are under the recipient's
control.  And you are ignoring processing at the level of MDAs,
message stores, etc.

> Also, 
> b) is obviously broken by spam filters and virus checkers.

Only if performed without the knowledge and consent of the
recipient.

> c) is 
> broken by spam filtering services offered at the server level,

Sometimes (but not in cases where the server is under control of one
of the end parties).

> and 
> also by spam filtering offered by mailing list software.

In the specific case of mailing lists, the list mailbox *is* the
receiving end of the communication.
 
> What can be done to make spam filtering Internet Architecture compliant?

Leave it up to recipients.

> > What "filter cooperation"?  In precisely what way do filters
> > at separate sites need to "cooperate", and how is that practical
> > without detailed configuration information?
> 
> As a means of reducing the cruft which is currently added by all these
> filters.  A standardized syntax with known semantics (whatever it
> turns out to be) stops people from reinventing their own X- headers again
> and again for a standard service. 

You seem to be at least several months behind the times:
* years ago, things like X-RBL etc. were sometimes inserted by ISPs;
   some users complained about them (apparently believing that it
   involved eavesdropping, which was not the case), but they were
   largely useless because of DHCP and other types of shared IP
   addresses, shared IP address blocks, etc.
* About 6 months ago, my ISP tried to implement message mangling
   w/o user input or consent.  That quickly went away after I
   complained about a legitimate mailing list message which was
   modified by the ISP (among other things, "*Possible Junk Mail* "
   was prepended to the message Subject field).  That *was*
   eavesdropping; it was forgery as well.
* At about the same time, fields like
     X-Junkmail-Status: score=0/70, host=mr01.mrf.mail.rcn.net
   started being added; such fields have not been useful, and
   (fortunately) are no longer being added.
* A large number of MUAs have spam filtering capability
* There are a several MUA-independent filtering packages that can
   be installed and used by end users, even those with low levels of
   computer literacy

I'm just not seeing "the cruft which is currently added by all
these filters".  If you are, I suggest that you open a dialog
with the parties responsible for adding the cruft, with the
objective of getting them to stop doing so.  The problem of
that cruft which *at one time* did exist here, appears to have
gone away, and I don't want to encourage its return.  And I
certainly don't want to encourage eavesdropping on private
correspondence, much less alteration of such correspondence
in transit.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: MUA Mail Options for a Mailing List [was Re: non-member messages to lists]
Date: Thu, 4 Nov 2004 10:52:26 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Mon November 1 2004 12:17, Charles Lindsey wrote:

> Thank you for proving my point.

If you had a point, I certainly didn't prove it.

> All that says is that RFC 3028 provides no 
> machinery for sieving based on the group name

What exactly is "sieving" supposed to mean? "Sieve"
is the name of a filtering language; it is not used
in RFC 3028 as a verb, which you would know if you
had bothered to read it.

> Likewise, it cannot sieve out messages

"sieve out"?!?  No such thing.

> From: "John Smith". 

That's a syntax error.

> So what? It may be regrettable 

It is not regrettable; it is a desirable feature that
non-protocol elements be ignored when making
decisions regarding protocol elements.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Archives
Date: Wed, 3 Nov 2004 21:00:43 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Tue November 2 2004 13:04, Keith Moore wrote:

> IIRC, SEEN should not change in a shared folder.

Perhaps (I can't find any definitive requirement); but SEEN
isn't the only flag (the "Recent" flag can be changed by the
server itself, even though clients are forbidden from changing
it).
 
> > > being able to store lots of messages in a single file can save a
> > > lot of disk space, particularly if you compress those mbox files.
> > 
> > I suppose that's so in general. However for the problem
> > domain (a server), disk space with modern hardware
> > isn't generally an issue.  If it is deemed to be a greater
> > issue than compression/decompression overhead for a
> > particular instance, compression can be implemented in
> > the file system, independent of storage format.
> 
> that's much more difficult to do.  the point is to make this easy.

More difficult for whom?  On an OS that supports compressed
filesystems, FS compression is transparent to applications
(other than performance issues) [and there are OSes that
support FS compression].  On the other hand, if compression
is implemented at the application level, every application that
accesses the files has to be able to use the same type of
compression and decompression, and has to be explicitly
coded to do so.  Where multiple access methods are used,
that means that it has to be handled by ftp servers, http
servers, etc. as well as support tools (does "ucbmail" support
editing of compressed mbox files?).
  
> > I think one of the biggest practical issues with Archived-At
> > support is arranging for the URI(s) to be put into the
> > message before the message is placed in the archive; the
> > URI (related to archive file name and/or UID) might not
> > be determinate until the message is actually placed in the
> > archive.
> 
> that's fairly easy to solve.  send the messages to the archiver 
> before distributing them to list subscribers.
> 
> 
>               +----------+       +----------+       +----------+
> incoming      | incoming |       |          |       |   list   |
> message ----> |   mail   | ----> | archiver | ----> | expander |
>               |  filter  |       |          |       |          |
>               +----------+       +----------+       +----------+
>                     |                  |                  |
>                     V                  V                  V
>                   rejects          archived          subscribers
>                                    messages

That doesn't address the issue of how the URIs that are to be put
in Archived-At fields are generated before the archive file path
and/or UID are known, nor how the field gets inserted in the
message.  Sure, list expansion can happen after archival; that's
the easy part.

> either that or base the name of the archive on some property of
> the message that can be derived independently, say the message-id
> or a hash of some kind.

"The name of the archive" may well be different for different
access schemes (and one would like to use the same underlying
files to avoid duplicate (triplicate, etc.) storage of messages).
It seems that either the final location, file name, and UID (for
POP and IMAP access) would either need to be determinate
before the files are placed in the archive, or the Archived-At
field would have to be generated and inserted in the messages
in situ after they're placed in the archive (leaving a time
window during which the messages appear (e.g. to users
browsing a folder) but do not yet have an Archived-At field).

Different schemes have different restrictions on the characters
allowable in UIDs and pathnames; also in lengths of identifiers,
etc., which would have to be taken into consideration in any
method that used message-ids or hashes. For example, POP
UIDs are limited in length to a maximum of 70 octets from a
specific set of characters (RFC 1939); many of those characters
are reserved or otherwise excluded from use in http or ftp URIs.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Wed, 3 Nov 2004 20:05:32 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Wed November 3 2004 06:25, Charles Lindsey wrote:
> 
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> >On Mon November 1 2004 11:52, Charles Lindsey wrote:
> 
> >> You agreed that RFC 2822 had (intentionally or otherwise) changed the
> >> semantics of <msg-id>s with regard to the case-sensitivity of whatever
> >> came after the '@'
> 
> >There you go again, trying to put words in other peoples' mouths.
> >I never said any such thing; I said that RFC 2822 is imprecise and
> >I have specifically pointed out that there is absolutely nothing in
> >RFC 2822 that supports your assertion that id-right is always case-
> >sensitive.
> 
> No, I am putting your own words back into your mouth.
[...]
> Which is exactly what I said you had said.

No! I did not claim that RFC 2822 changed case-insensitivity
requirements.

To: LIST: [email-masked];
Xref: clerew local.mime:4121
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Wed, 3 Nov 2004 11:25:25 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Mon November 1 2004 11:52, Charles Lindsey wrote:

>> You agreed that RFC 2822 had (intentionally or otherwise) changed the
>> semantics of <msg-id>s with regard to the case-sensitivity of whatever
>> came after the '@'

>There you go again, trying to put words in other peoples' mouths.
>I never said any such thing; I said that RFC 2822 is imprecise and
>I have specifically pointed out that there is absolutely nothing in
>RFC 2822 that supports your assertion that id-right is always case-
>sensitive.

No, I am putting your own words back into your mouth.

>From http://www.imc.org/ietf-822/mail-archive/msg04722.html

>You have missed the critically important fact that in all cases
>discussed, the relevant RFCs refer to RFC 822 (a full Standard),
>not RFC 2822 (a Proposed Standard). And RFC 822 defines the
>identifiers in question as containing a local-part and a domain.
>
>RFC 2822 id-right is a problem because it changes the semantics.

Which is exactly what I said you had said.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4122
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Archives
Date: Wed, 3 Nov 2004 11:43:21 GMT
Lines: 44
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> On Sun October 31 2004 18:03, Keith Moore wrote:

>> > though for the  
>> > sake of efficiency I'm wondering about options to use mbox format 
>> > (yeech) and compression,
>> 
>> It's not clear what efficiency would result; mbox format is a pain
>> to edit (e.g. to remove spam or other inappropriate messages that
>> might get into the archive)

>there are lots of tools that edit mbox files, e.g. ucbmail.

>> and has the disadvantage of putting
>> all of the eggs in one rather vulnerable basket. It would also be
>> difficult to provide ftp*etc.) access to individual messages from
>> a flat mbox file.

>yup.  the clients would have to download the entire mbox file.

Actually, you can construct an http url that would access an individual
message within an mbox file, and you would only pay for the bandwidth of
that particular message.

But I do not suggest that as a practicable solution to this problem,
because there are too many other snags with it, as you have outlined.

One of my concerns with the archive of this list (and of other lists
handled by imc.org) is that the accessible archive is in html. Yes, there
is indeed a text version at http://www.imc.org/ietf-822/entire-arch.txt,
but that is a monolithic mbox file of everything since 1997. I shudder to
think how big it must be, and there appears to be no index into it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 2 Nov 2004 13:04:29 -0500
From: Keith Moore <[email-masked]>
To: Bruce Lilly <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Archives
Sender: [email-masked]

> > I'm assuming that archives are read-only, so archives stored in 
> > maildirs shouldn't result in files being renamed.  
> 
> According to
> http://www.washington.edu/imap/documentation/formats.txt.html
> and confirmed by the URI that you provided, file names change
> whenever status (e.g. seen) changes, and that would seem to
> apply even in an environment where folders are read-only from
> an IMAP protocol perspective.

IIRC, SEEN should not change in a shared folder.

> > there are lots of tools that edit mbox files, e.g. ucbmail.
> 
> With corresponding locking issues, etc.

yup.  but assuming archives aren't changed very often, these
seem managable.

> > being able to store lots of messages in a single file can save a
> > lot of disk space, particularly if you compress those mbox files.
> 
> I suppose that's so in general. However for the problem
> domain (a server), disk space with modern hardware
> isn't generally an issue.  If it is deemed to be a greater
> issue than compression/decompression overhead for a
> particular instance, compression can be implemented in
> the file system, independent of storage format.

that's much more difficult to do.  the point is to make this easy.
 
> > in my experience most IMAP clients don't do a good job with folders
> > that large.  I'm thinking that archives should be able to support
> > hierarchal directory trees.
> 
> Some clients do seem quite a bit slower than others (that
> applies to other protocols, e.g. POP3, as well). It depends
> on what a particular client requests, and for a listing of
> an entire folder (as opposed to a fetch of a particular
> message), the number of messages in a folder could be
> an issue with some clients, on the first access of that folder
> by that client, for some client implementations.

yup.  some client implementations want to download every message
in the folder; others want to download headers from every 
message in the folder.  these behaviors are quite common.

> I think one of the biggest practical issues with Archived-At
> support is arranging for the URI(s) to be put into the
> message before the message is placed in the archive; the
> URI (related to archive file name and/or UID) might not
> be determinate until the message is actually placed in the
> archive.

that's fairly easy to solve.  send the messages to the archiver 
before distributing them to list subscribers.

              +----------+       +----------+       +----------+
incoming      | incoming |       |          |       |   list   |
message ----> |   mail   | ----> | archiver | ----> | expander |
              |  filter  |       |          |       |          |
              +----------+       +----------+       +----------+
                    |                  |                  |
                    V                  V                  V
                  rejects          archived          subscribers
                                   messages

either that or base the name of the archive on some property of
the message that can be derived independently, say the message-id
or a hash of some kind.

Keith

From: Bruce Lilly <[email-masked]>
Reply-To: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Archives
Date: Tue, 2 Nov 2004 12:39:31 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Tue November 2 2004 10:52, Keith Moore wrote:
 
> http://cr.yp.to/proto/maildir.html is as close to a specification
> as I know of.  
[...]
> the reason I've been thinking of maildirs is that it's simple to use,
> it's robust, there are already IMAP and POP protocol servers that
> support it, and it's not too  difficult to make maildirs accessible by
> WebDav and FTP either. they are a tad awkward for WebDav and FTP access
> (as there are reasons you want to support a hierarchy of folders
> and maildirs use an extra layer of directories) but I don't think this
> is onerous.
> 
> I'm assuming that archives are read-only, so archives stored in 
> maildirs shouldn't result in files being renamed.  

According to
http://www.washington.edu/imap/documentation/formats.txt.html
and confirmed by the URI that you provided, file names change
whenever status (e.g. seen) changes, and that would seem to
apply even in an environment where folders are read-only from
an IMAP protocol perspective.

> there are lots of tools that edit mbox files, e.g. ucbmail.

With corresponding locking issues, etc.

> being able to store lots of messages in a single file can save a
> lot of disk space, particularly if you compress those mbox files.

I suppose that's so in general. However for the problem
domain (a server), disk space with modern hardware
isn't generally an issue.  If it is deemed to be a greater
issue than compression/decompression overhead for a
particular instance, compression can be implemented in
the file system, independent of storage format.

> in my experience most IMAP clients don't do a good job with folders
> that large.  I'm thinking that archives should be able to support
> hierarchal directory trees.

Some clients do seem quite a bit slower than others (that
applies to other protocols, e.g. POP3, as well). It depends
on what a particular client requests, and for a listing of
an entire folder (as opposed to a fetch of a particular
message), the number of messages in a folder could be
an issue with some clients, on the first access of that folder
by that client, for some client implementations.

I think one of the biggest practical issues with Archived-At
support is arranging for the URI(s) to be put into the
message before the message is placed in the archive; the
URI (related to archive file name and/or UID) might not
be determinate until the message is actually placed in the
archive.

To: LIST: [email-masked];
Xref: clerew local.mime:4111
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: a header authentication scheme
Date: Tue, 2 Nov 2004 16:13:12 GMT
Lines: 69
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Sun October 31 2004 01:54, Laird Breyer wrote:

>> > Given a message with a valid Received field; it's trivial; simply
>> > construct a Processed field accordingly and insert it where
>> > desired, eliding any preexisting "processed" field that corresponds
>> > to the Received field.  W/o any Received field; forge one and
>> > insert it and a corresponding "processed" field where desired.
>> 
>> I'm afraid none of this is yet convincing as a successful attack. 
>> 
>> You must be able to forge the Processed field before the message
>> contains the pertinent Received field, otherwise you're not actually
>> forging anything of value. In fact, quoting existing Received fields
>> is legal, and marks your location correctly. 

>You have missed the points; at any point a party can
>a) s/result-tag="spam"/result-tag="ham"/
>b) insert a Received field, then insert a "processed" field referring
>    to that Received field, with any desired "processed" field content
>N.B. neither tactic would be effective if MIME security multiparts
>were used.

Please can you paint a more precise scenario?

Exacly where is the "any point" that you have in mind? If it is inside
your firewall, then your system has already been subverted and nothing is
going to protect you.

If it is inside your ISP's systems, then your ISP has already been
subverted and nothing is going to protect you.

If it was done somewhere before the message reached your ISP, and you
trust your ISP not to have gone rogue, then you can trust your ISP to have
added his Received header (and maybe his Processed header) at the front of
the message, in which case who cares what the spammer may have put there
earlier on.

If it was done by intercepting the transmission between your ISP and your
firewall, then indeed there is something to look into further. If that
link took the form of you interrogating your mailbox (POP3 or IMAP) on the
ISP's machine, then I don't know enough about the protocols involved be be
able to say, but I suspect it would be rather difficult for the Bad Guy to
make that intercept.

If that link took the form of an SMTP connection, then I think it is
generally possible for your system to detect the IP from which the packets
came, and check that they did indeed come from your ISP.

The whole point of Laird's scheme is that, the further back you go down
the chain of intermediate hosts, the less you can assume about the
reliability of what the headers are telling you. The particular case that
he is interested in is where he is using some filtering service provided
by his ISP (e.g. Spamassassin) and wants to be as certain as can be that
some other bogus filtering service has not been substituted instead.

AFAICS, his scheme does achieve that rather limited objective.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 2 Nov 2004 10:52:54 -0500
From: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Archives
Sender: [email-masked]

> On Sun October 31 2004 18:03, Keith Moore wrote:
> 
> > I'm thinking that 
> > archives should be based on maildir at the server end,
> 
> It's not clear exactly what you mean by "maildir";

http://cr.yp.to/proto/maildir.html is as close to a specification
as I know of.  

but I'm not really thinking that maildirs should be standardized
as the archive format, so much as I'm thinking that it should
be possible to use maildirs as an archive format.  

the reason I've been thinking of maildirs is that it's simple to use,
it's robust, there are already IMAP and POP protocol servers that
support it, and it's not too  difficult to make maildirs accessible by
WebDav and FTP either. they are a tad awkward for WebDav and FTP access
(as there are reasons you want to support a hierarchy of folders
and maildirs use an extra layer of directories) but I don't think this
is onerous.

I'm assuming that archives are read-only, so archives stored in 
maildirs shouldn't result in files being renamed.  

> > though for the  
> > sake of efficiency I'm wondering about options to use mbox format 
> > (yeech) and compression,
> 
> It's not clear what efficiency would result; mbox format is a pain
> to edit (e.g. to remove spam or other inappropriate messages that
> might get into the archive)

there are lots of tools that edit mbox files, e.g. ucbmail.

> and has the disadvantage of putting
> all of the eggs in one rather vulnerable basket. It would also be
> difficult to provide ftp*etc.) access to individual messages from
> a flat mbox file.

yup.  the clients would have to download the entire mbox file.

the question is - what's the right balance of client work vs.
archive server work?  do we want to make things easy for archive
providers, easy for client writers, or somewhere in between?
being able to store lots of messages in a single file can save a
lot of disk space, particularly if you compress those mbox files.

there's a separate question of how you reference a particular
message within an mbox file.  maybe something like 
http://host/directory/path/file.mbox#message-id (yeech)

> > and also how to avoid having several years'  
> > worth of archives in a single directory or file while still making
> > the whole archive appear seamless to the client.
> 
> Subfolders would be one way to organize messages into groups
> (e.g. by year and month), but I'm not convinced that it's
> necessary (I've used Cyrus imapd with more than 30,000
> messages in a folder with no problem).

in my experience most IMAP clients don't do a good job with folders
that large.  I'm thinking that archives should be able to support
hierarchal directory trees.

> Using Cyrus imapd would natively provide IMAP, POP, and
> NNTP access; since each message is stored as a file, ftp access
> would be trivial to provide (simply provide a link between
> ftp namespace and the IMAP folder).  Likewise for serving
> native message format via http.  Using off-the-shelf products
> and minimal effort, that would provide access to messages
> via five distinct protocols (six URI schemes (news and nntp
> schemes both use NNTP protocol)) from a common message
> store.

yup.  and I want people to be able to use tools like that, without
actually being too specific to one particular tool.

Keith

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Archives
Date: Mon, 1 Nov 2004 22:57:09 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

On Sun October 31 2004 18:03, Keith Moore wrote:

> I'm thinking that 
> archives should be based on maildir at the server end,

It's not clear exactly what you mean by "maildir"; the UW IMAP
documents mention a "Maildir" format, but that has a number
of problems including frequent renaming of files, which won't
work well with URIs.  Cyrus imapd folder structure seems
amenable; UIDs are numeric and the message file name
consists of the UID followed by a period. There are a few
additional database files (cyrus.cache, cyrus.header,
cyrus.index) and there may be subfolders.  Messages are
stored in pure RFC [2]822 format (including CRLF line endings).

> though for the  
> sake of efficiency I'm wondering about options to use mbox format 
> (yeech) and compression,

It's not clear what efficiency would result; mbox format is a pain
to edit (e.g. to remove spam or other inappropriate messages that
might get into the archive) and has the disadvantage of putting
all of the eggs in one rather vulnerable basket. It would also be
difficult to provide ftp*etc.) access to individual messages from
a flat mbox file.

> and also how to avoid having several years'  
> worth of archives in a single directory or file while still making the 
> whole archive appear seamless to the client.

Subfolders would be one way to organize messages into groups
(e.g. by year and month), but I'm not convinced that it's
necessary (I've used Cyrus imapd with more than 30,000
messages in a folder with no problem).

> Offhand it looks like FTP  
> is probably the most straightforward access method to implement, but 
> that it should be possible to define reasonable access via other 
> protocols also.

Using Cyrus imapd would natively provide IMAP, POP, and
NNTP access; since each message is stored as a file, ftp access
would be trivial to provide (simply provide a link between
ftp namespace and the IMAP folder).  Likewise for serving
native message format via http.  Using off-the-shelf products
and minimal effort, that would provide access to messages
via five distinct protocols (six URI schemes (news and nntp
schemes both use NNTP protocol)) from a common message
store.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 1 Nov 2004 20:54:34 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Mon November 1 2004 12:09, Charles Lindsey wrote:

> An mbox file contains '^From�' lines that contain the envelope From
> address, plus a time stamp.
[...]
> the mbox format is noted for lack of any rigid 
> definition

You have once again contradicted yourself. Do yourself and
the rest of us a favor and try *thinking* about the issues for
a while instead of spouting contradictory bits of nonsense.

> The problem arises in User Agents that try to overload one of the same two
> buttons "Reply" and "Reply-to-All" to double up as the "Followup" button

No, Charles, as has been explained to you several times, posting
to newsgroups specified in an original as well as mailing to mailboxes
is a perfectly reasonable set of actions corresponding to a
Reply-To-All function.  There were no "buttons", "Followup"
or otherwise, involved.

> but prevalent in many 2nd rate agents anyway).

Where you no doubt expect everybody to agree with your
definition of "2nd rate" as "anything that contradicts
Charles' peculiar world view".  Sorry, Charles, it doesn't
work that way.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 1 Nov 2004 20:34:15 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Mon November 1 2004 11:52, Charles Lindsey wrote:

> You agreed that RFC 2822 had (intentionally or otherwise) changed the
> semantics of <msg-id>s with regard to the case-sensitivity of whatever
> came after the '@'

There you go again, trying to put words in other peoples' mouths.
I never said any such thing; I said that RFC 2822 is imprecise and
I have specifically pointed out that there is absolutely nothing in
RFC 2822 that supports your assertion that id-right is always case-
sensitive.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 1 Nov 2004 20:30:56 -0500
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

On Mon November 1 2004 11:44, Charles Lindsey wrote:

> Then Pray Tell what it puts in the "^From�" line (that usually contains
> the envelope From address plus a time stamp).

If you want to know that badly, consult the source (hint: in trn
look for "mbox.saver").

> Any why is rn saving articles in the first place

Because the user issued the command to save a copy of the message.

> But there is provision for implementation-defined flags. I never claimed
> that IMAP could solve this problem 'out of the box' - merely that it
> contained 'hooks', such as those flags, which an implementor could (and to
> do a proper job should) use to keep track of which articles were 'news'
> and which were 'email' as they were moved between IMAP boxes.

No.  You clearly don't understand the client/server nature of IMAP.
 
> There IS a way (flags, as I have just demonstrated)

You have demonstrated nothing.

> , and there is a need,

Saying so (repeatedly) doesn't make it so.

> I am not trying to impose anything. I am just pointing out that
> inplementors of IMAP have to solve this problem

The "problem" is a figment of your imagination.

> >What are "broken" are your ability to comprehend what has been
> >explained to you several times by several people, and your ability
> >to realize that your ability to ram your odd ideas down the IESG's
> >throats is nil.
> 
> Resort to rudeness is always a clear indication that you are losing the
> argument.

No rudeness, just the facts.
 
> >> Hence the presence of a Return-Path (or of an equivalent "^From�" line
> >> arising from an mbox file) is a sure sign that it is an Email.
[...]
> Indeed, the presence/absence of a Return-Path is not conclusive evidence
> of whether a message is a news article

You have just contradicted yourself, ("presence of a Return-Path is"
"a sure sign"/"not conclusive evidence") and that is a sure sign that
you are making things up as you go along.

Cc: ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: General considerations for new message field specifications 
Date: Tue, 2 Nov 2004 09:16:56 -0500
To: Peter Koch <[email-masked]>
Sender: [email-masked]

>> Can anybody think of additional considerations (other than
>
> the transparency issue: what happens, if one/some/all of the MUAs, MTAs
> involved in generating, transporting, delivering or digesting the 
> message
> do *not* understand this header field?

agreed.  in general, only the sending and recipient MUA should have to 
understand the new field; MTAs shouldn't have to care about it.  if 
you're defining new functionality for MTAs it generally doesn't belong 
in the message header.  but in reality there are some exceptional 
cases.  for instance MTAs do rewrite addresses (even though they 
shouldn't) so this has implications for any new field that has 
addresses.  and with the to-noreply and cc-noreply proposal it became 
obvious that the submission server would also have to understand them 
(at least for sendmail -t scenarios).

so basically any new proposal for header fields should analyze its 
impact on sending MUA, submission servers, intermediate MTAs, final 
delivery agents, and recipient MUA (including how this affects replies 
if applicable).  when applicable, it should also analyze the impact on 
mailing lists, forwarding, mail filters, and other intermediaries, and 
auto responders.

To: ietf-822 <[email-masked]>
Subject: Re: General considerations for new message field specifications 
Date: Tue, 02 Nov 2004 14:06:39 +0100
From: Peter Koch <[email-masked]>
Sender: [email-masked]

> Can anybody think of additional considerations (other than

the transparency issue: what happens, if one/some/all of the MUAs, MTAs
involved in generating, transporting, delivering or digesting the message
do *not* understand this header field?

-Peter

Date: Tue, 2 Nov 2004 14:36:43 +1000
From: Laird Breyer <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: a header authentication scheme
Reply-To: [email-masked]
Mail-Followup-To: ietf-822 <[email-masked]>
Sender: [email-masked]

On Nov 01 2004, Bruce Lilly wrote:
> 
> You have missed the points; at any point a party can
> a) s/result-tag="spam"/result-tag="ham"/
> b) insert a Received field, then insert a "processed" field referring
>     to that Received field, with any desired "processed" field content
> N.B. neither tactic would be effective if MIME security multiparts
> were used.

Perhaps you have a concrete example where you think a) or b) occurs,
the only examples I can think of are compromised systems, for which
the problem is moot. I'll detail my thinking below in case you can see
which assumptions you disagree with, sorry if it looks like I'm
repeating myself (I am).

a) requires modifying some existing Processed field. No well behaved
filter needs to modify existing fields in a message. If a modification
occurs, then the process doing this modification is doing something
prohibited (tampering with the message), and the computer system which
is running this process has therefore been hacked or belongs to an
untrustworthy party.

b) requires inserting a new Received field. The sequence of Received
fields added close to the message destination is predictable by the
recipient, and can only be changed by an attacker if one of the
computers in the recipient's organization is compromised.

So we always have a model of message transport as follows:

A -> M1 -> M2 -> M3 -> ... -> T1 -> T2 -> ... -> B

A is the sender of the message, B is the destination, M represents
computers which are untrusted by B, and T represents computers which
are trusted by B and add a Received field.  None of the computers of
type T will ever perform either a) or b) above, unless they have been
compromised. Computers of type M may well perform a) or b). It is up
to B to only trust Processed fields associated with computers of type
T.

If the recipient's MTA doesn't add a Received field of its own (ie
there are no T type computers at all), then the recipient B must already know
not to trust any Received fields at all in a message.

> 
> Evidently you have little experience in examining Received fields;
> detecting forgeries is not always trivial (in part because legitimate
> fields are often screwed up, in part because sometimes fields
> are reordered, in part due to host aliases, etc.).
> 

That's a fair statement. But are you really saying that a given user B
cannot, by examining a sample of past mail messages he personally received,
deduce the form of the Received fields added by the computers I've labeled
T above?  I must admit I find that hard to believe.

> For a concrete example, suppose that some recipient
> is interested in a home mortgage.  A spam filter that is configured
> to treat all messages containing the word "mortgage" as spam is
> worse than useless to that recipient; it is counter-productive. And
> on the basis of the (lack of) information in your proposed
> "processed" field, no recipient is able to determine whether or not
> configuration of any alleged process is appropriate *for him*.

Obviously such a user will learn to not accept the supplied spam label
as authoritative. He will either ignore the Processed field's
recommendation completely, or configure his other filters if he has
any to lower the weight of that particular field so as to let through
mortgage mails but not other spam.

For example, statistical content filters already perform this exact 
adjustment automatically, ie giving more weight to tokens which correlate
well with the user's personal idea of spam say, and reducing the weight of
tokens which don't. This is done by asking users to mark incorrectly
filtered messages, i.e. without the need to understand the internals of the 
particular spam filter which marks all messages containg "mortgage" as spam.

> An assertion made by some
> intermediate system which is supposed to be providing transport
> services simply fails to fit into the Internet Architecture on many
> levels:
> a) is is not an end-to-end process
> b) eavesdropping on private communications is immoral and
>     offensive (and may be illegal in some places)
> c) it is not under control of either party in the end-to-end
>     communication
> etc.

Would you say that current mail practices such as providing spam filtering
fit into the Internet Architecture described by RFC 1958 ? 

For example, a) is broken by spam filters set at the level of SMTP
servers rather than filters set up at the individual MUA level.  Also,
b) is obviously broken by spam filters and virus checkers. c) is
broken by spam filtering services offered at the server level, and
also by spam filtering offered by mailing list software.

What can be done to make spam filtering Internet Architecture compliant?

> What "filter cooperation"?  In precisely what way do filters
> at separate sites need to "cooperate", and how is that practical
> without detailed configuration information?

As a means of reducing the cruft which is currently added by all these
filters.  A standardized syntax with known semantics (whatever it
turns out to be) stops people from reinventing their own X- headers again
and again for a standard service. 
It makes it easier for other software to pick and choose which
headers to ignore and which headers to read. It gives software writers
a way of adding their own bits in controlled ways.

Which bits they will otherwise add anyway regardless of the damage it
might do to the mail system.

-- 
Laird Breyer.

Cc: ietf-822 list <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Understanding response protocols
Date: Mon, 1 Nov 2004 23:22:26 -0500
To: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

>> It's hardly surprising that your customers who have already become 
>> accustomed to the way your product behaves are annoyed when you 
>> change it.  LISTSERV and ListProc users who were used to Reply-To 
>> being  set by those servers also tended to think that having lists 
>> set Reply-To was "the right thing".
>
>> None of this changes the fact that having lists set Reply-To is less
>> functional and does harm to the message.
>
> That is a commonly expressed view, but I think it is wrong.

You are simply incorrect.   (I've already explained why, so I won't 
repeat it)

> I much prefer those lists which set Reply-To (I have been on lists 
> where
> the members have voted to have it set so), even though I have to 
> remember
> which list is which in order to press the right Reply button.

You are entitled to your preferences.  Note that having replies go only 
to the list is a separate issue from whether it's appropriate for lists 
to set Reply-To.

> It will be even better, of course, when we have a MFT field available 
> as
> well, so that Reply-To is still available for direct contact with the
> author.

You've just aptly demonstrated one of the biggest potential problems 
with MFT - that lists will misuse it just as they've misused Reply-To.

>>  All it means is that users
>> often choose the behavior they have adapted to in their isolated
>> communities, rather than the behavior that works best overall.
>
> But at the moment we have the worst of both worlds. Some communities do
> one thing, some do the other, and others have flame wars about whether 
> it
> is ruder to do one thing or to do the other.

That's always the case when you have multiple communities with 
different conventions intersecting.  The question is, do you want to 
resolve the conflict in favor of the more functional outcome, or the 
less functional outcome?  Or do you want to perpetuate the conflict?

Keith

To: LIST: [email-masked];
Xref: clerew local.mime:4100
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: MUA Mail Options for a Mailing List [was Re: non-member messages to lists]
Date: Mon, 1 Nov 2004 17:17:06 GMT
Lines: 39
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>It would help if you would actually read the documents that
>you are referred to.  RFC 3028 (a Standards Track RFC) is
>quite clear:

>   That is, routes and group syntax are not permitted.  If multiple
>   addresses are required, use a string list.  Named groups are not used
>   here.
>[...]
>   The address primitive never acts on the phrase part of an email
>   address, nor on comments within that address.  It also never acts on
>   group names, although it does act on the addresses within the group
>   construct.

Thank you for proving my point. All that says is that RFC 3028 provides no
machinery for sieving based on the group name (as opposed to procmail
which could do it quite well). Likewise, it cannot sieve out messages
From: "John Smith". So what? It may be regrettable, but it does not outlaw
the use of group names in emails, and is totally neutral as to whether a
convention of using the group name "LIST" would be a Good Thing or not.

>It would help even more if you would familiarize yourself with
>relevant Standards Track RFCs and Standards so as to avoid
>proposing cockamamie schemes.

I am still awaiting your better suggestion.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4102
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: http://www.ietf.org/internet-drafts/draft-duerst-archived-at-02.txt
Date: Mon, 1 Nov 2004 17:41:53 GMT
Lines: 48
Sender: [email-masked]

In <6.0.0.20.2.20041028174107.1002c0b8@localhost> Martin Duerst <[email-masked]> writes:

>What I wanted was to make sure that we get a provisional registration
>as soon as possible, and then can move to the permanent registration
>as the draft moves forward. That was the way I understood RFC 3864.
>Are you saying that I shouldn't care about the provisional registration?
>Or should I do the provisional totally separately? Rereading
>RFC 3864, I now seem to understand that registering separately
>is how it should be done.

I think that a provisional registration is desirable if you are going to
start using it in the wild (the whole idea of the provisional registry was
to prevent the like of X-Archived-At appearing in the wild and getting
hard-wired into too many agents).

>I think it could be potentially useful for netnews, although netnews
>has URI schemes that make it possible to refer to a news item, and
>the way netnews is distributed makes it somewhat easier to find them
>than email messages. Also, adding an 'Archived-At' header to netnews
>can be trickier than for email unless it would be done at the source.

It could be quite handy for some special-purpose uses within Netnews, such
as for FAQs (as posted to news.anwers) and for archives of announce
groups.

> >If you are really not sure about netnews, then I suggest you leave it out 
>and separately request an entry for netnews in the provisional header field 
>registry:  it can be moved later if netnews community think that's reasonable.

But I think that is the best advice for now. There are quite a few email
headers that make reasonable sense in News, and vice versa, and agents
that reckon they can make use of them are welcome to do so without
necessity for explicit mention in the IANA registry. This registry is
quite new, so better not to clutter it up until we see how practice
develops.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4101
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: New Internet Draft: draft-duerst-archived-at-00.txt
Date: Mon, 1 Nov 2004 17:32:54 GMT
Lines: 36
Sender: [email-masked]

In <2IMY8L/[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>Martin Duerst writes:
>> As said in other mails, the way we use these headers, it's mostly 
>> humans rather than email programs using them. Is there something in 
>> the email specs (e.g. RFC 2822 or so) that would forbid such headers?

>No, although...

>I believe that as a rule of thumb, it's bad to go against the grain of a 
>design. In theory it can be sensible and right, but generally it turns 
>out to be a bad idea eventually. And in mail, the header fields that 
>are meant for direct human consumption are generally unstructured (ie. 
>they use the 'unstructured' production in RFC2822). The ones that 
>aren't unstructured are generally meant to be parsed.

It's not as simple as that. Some fields may be _primarily_ for human
consumption, and it may well be that no general-purpose MUA would ever
process them, but individual users might well write scripts to do cool
things with them, in which case they will want to relay on some
predictable format.

So how about a script to search my emails for some keyword, and to return
the URL from that Archive-At header of the matching messages, ready to
paste into some new document?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4103
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Understanding response protocols
Date: Mon, 1 Nov 2004 17:48:14 GMT
Lines: 39
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>It's hardly surprising that your customers who have already become
>accustomed to the way your product behaves are annoyed when you change
>it.  LISTSERV and ListProc users who were used to Reply-To being 
>set by those servers also tended to think that having lists set
>Reply-To was "the right thing".   

>None of this changes the fact that having lists set Reply-To is less
>functional and does harm to the message.

That is a commonly expressed view, but I think it is wrong.

I much prefer those lists which set Reply-To (I have been on lists where
the members have voted to have it set so), even though I have to remember
which list is which in order to press the right Reply button.

It will be even better, of course, when we have a MFT field available as
well, so that Reply-To is still available for direct contact with the
author.

>  All it means is that users
>often choose the behavior they have adapted to in their isolated
>communities, rather than the behavior that works best overall.

But at the moment we have the worst of both worlds. Some communities do
one thing, some do the other, and others have flame wars about whether it
is ruder to do one thing or to do the other.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 1 Nov 2004 12:41:22 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked];
Subject: Re: Angle brackets surrounding Content-ID
Sender: [email-masked]

> An mbox file contains '^From�' lines that contain the envelope From
> address, plus a time stamp. 

there's no standard specification for mbox files.  and from the earliest
days that mbox files were used it was not unusual for the "From_" line
to contain a meaningless username like uucp.

Keith

To: LIST: [email-masked];
Xref: clerew local.mime:4097
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 1 Nov 2004 16:44:20 GMT
Lines: 115
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Fri October 22 2004 07:45, Charles Lindsey wrote:

>> A file in mbox format cannot contain a news article.

>The newsreader rn and its derivatives and a number of other
>newsreaders save news articles in that format, and can read
>messages (which might be "news" messages or "mail" messages)
>in that format.

Then Pray Tell what it puts in the "^From�" line (that usually contains
the envelope From address plus a time stamp).

Any why is rn saving articles in the first place (or is that for some
archiving facility)? Last time I used rn, it worked online, acquiring
articles on demand from a server, either directly or via NNTP.

>> There are plenty of hooks within IMAP to enable an implementor to
>> distinguish Email from Netnews.

>No.

>> For example, it is customary to store news 
>> articles in folders within a "#news" hierarchy.

>> Messages also carry flags 
>> with them which can be used to distinguish various classes of message.

>There is no "news" flag or anything remotely resembling
>one.

But there is provision for implementation-defined flags. I never claimed
that IMAP could solve this problem 'out of the box' - merely that it
contained 'hooks', such as those flags, which an implementor could (and to
do a proper job should) use to keep track of which articles were 'news'
and which were 'email' as they were moved between IMAP boxes.

>> Indeed so. The message you are now reading has passed through a
>> news-to-mail gateway and is therefore, by definition, an email message.

>Yes, and the message that it is a response to was also an email
>message, whether or not you viewed it in a newsreader.

Correct.

>> If it is in an
>> IMAP box, then it depends how it was put there, and that is a matter for
>> the IMAP implementor.

>No. IMAP is a Standards Track protocol that handles messages;
>"mail", "news", "hand-crafted" is immaterial. Human users using
>a variety of standards-compliant UAs can copy/move messages
>between folders at will, as many times as they wish, and there is
>no way (or need) to track such movement.

There IS a way (flags, as I have just demonstrated), and there is a need,
because just inspecting the message content is unreliable.

>  This too has been
>explained to you many times, and you have been informed in
>no uncertain terms that any attempt by you to try to impose
>any such requirements on IMAP will not pass IESG muster.

I am not trying to impose anything. I am just pointing out that
inplementors of IMAP have to solve this problem somehow, and that it can
be solved rather easily without making any change to the IMAP standards
(though an extension to define a suitable flag could be considered).

>> >It's handling "messages".  When a message is being transported, it
>> >might be transported via a news-specific protocol, or via a mail-specific
>> >protocol, or via a protocol which is non-specific.  Once it has been
>> >transported, it is simply a message.
>> 
>> No. Email and Netnews are distinct media.

>> If a dual-use agent cannot keep 
>> track of which medium it is handling, then it is broken, as is any system
>> which uses a non-specific protocol without suitable tagging.

>What are "broken" are your ability to comprehend what has been
>explained to you several times by several people, and your ability
>to realize that your ability to ram your odd ideas down the IESG's
>throats is nil.

Resort to rudeness is always a clear indication that you are losing the
argument.

>> >Return-Path fields are only inserted by SMTP when making final delivery;
>> 
>> Hence the presence of a Return-Path (or of an equivalent "^From�" line
>> arising from an mbox file) is a sure sign that it is an Email.

>No; as usual you haven't been paying attention. A message having
>been delivered by SMTP to a mail-to-news gateway may very well
>still have a Return-Path field when it is transported by a "news"
>protocol.

Indeed, the presence/absence of a Return-Path is not conclusive evidence
of whether a message is a news article, but it is considerably more
reliable than the presence/absence of a Newsgroups header.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4099
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 1 Nov 2004 17:09:46 GMT
Lines: 67
Sender: [email-masked]

In <[email-masked]> Martin Trautmann <[email-masked]> writes:

>On 2004-Oct-22, at 13:45, Charles Lindsey wrote:

>> A file in mbox format cannot contain a news article. If the file is in
>> some "mbox-like" format, then it is up to the implementor of the 
>> format to
>> make provision.

>Why can't it be mbox files? There's lots of postings that I archive 
>locally. It looks to me like a news article in mbox format. What's the 
>difference?

An mbox file contains '^From�' lines that contain the envelope From
address, plus a time stamp. Since news articles don't have any envelope,
they cannot be stored in pure mbox files.

I have no doubt that files in a similar style to mbox can be used to
hold news articles (the mbox format is noted for lack of any rigid
definition anywhere), but in that case I would expect the news articles
for have a '^From�' line formatted in some different way that could be
used to distinguish them from any email messages that might have got mixed
in.

>On the other hand Newsgroups would qualify as a good news indicator - 
>while the message may have been Cc-ed and thus is in fact an email.

Exactly.

>But I lost track what for we do need to know the difference: If it's 
>here for me, I may add one or the other processing. But I can build a 
>posting from an E-Mail or do the opposit as well. It would not be good 
>behavior because of many reasons anyway (e.g. someone might be confused 
>because of a reply while he can't get the originated message).

The problem arises in User Agents that try to overload one of the same two
buttons "Reply" and "Reply-to-All" to double up as the "Followup" button
when operating in News mode (a practice strongly deprecated by the GNKSA,
but prevalent in many 2nd rate agents anyway).

If there is an explicit "Followup" button, then the agent knows it is
supposed to be producing a News followup (and if you try and use it on an
email with no Newsgroups header, then it knows to complain).

But if there is just some form of "Reply" button, and it is not keeping
track of whether it is in a News folder or an Email folder, then it has to
"guess" which it is supposed to be doing. And my original point was that
just looking for the presence of a Newsgroups header was a most unreliable
discriminant, because many emails that have been gatewayed from news, or
which have been Posted-and-Mailed, can contain a perfectly legitimate
Newsgroups header (which is indeed quite informative for the recipient),
but they are still emails, and the Reply button should generate an email
reply. Of course, if there is a "Followup" button as well, then you have a
choice.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4098
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Angle brackets surrounding Content-ID
Date: Mon, 1 Nov 2004 16:52:03 GMT
Lines: 45
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Fri October 22 2004 07:54, Charles Lindsey wrote:

>> But once the domain name has been copied into the id-right

>There is no "copying" involved; RFC 2822 recommends that
>id-right *be* a domain name, full stop.

>> (whether in 
>> upper or lower case) it is no longer a domain name for any protocol
>> purpose.

>Says who? Certainly not RFC 2822.

>>  its only purpose is to be compared for identity
>> with other <id-right>s for the removal of duplicates, for threading, etc.

>No, Charles; the original message in this discussion mentioned
>content identifiers, which explicitly use RFC 822 syntax. There
>was no mention of "removal of duplicates" or "threading" or
>anything vaguely similar.

This thread has developed a long way since its original Subject was set.

You agreed that RFC 2822 had (intentionally or otherwise) changed the
semantics of <msg-id>s with regard to the case-sensitivity of whatever
came after the '@'. I said that I considered the change to be an
improvement, and you have been harranguing me ever since for being so
presumptious.

Whilst is is arguable whether RFC 2822 applies retrospectively to
Content-ID, there is no doubt that it applies to Message-ID, and that is
the more interesting case.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: the gap regarding Archived-At
Date: Mon, 1 Nov 2004 10:56:58 -0500
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

An update on MUA/browser capabilities:

Kmail does seem to be able to handle IMAP URIs including those
that specify a particular message; via the File -> Open -> Network
dialog, a URI can be typed or pasted, and Kmail will open the
specified message.   I suspect that it wouldn't be terribly
difficult to modify Kmail to recognize IMAP URIs and open any
specified message in a new window, but as of now that capability
doesn't exist.

I can't find any similar capability in Mozilla, Evolution, or Sylpheed.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Use of Reply-To
Date: Mon, 1 Nov 2004 10:47:41 -0500
Cc: Graham Klyne <[email-masked]>
Sender: [email-masked]

On Mon November 1 2004 05:14, Graham Klyne wrote:

> BTW, Bruce, your use of reply-to header fields is confusing me -- you 
> nearly didn't get a directly addressed copy of this message.  Did you 
> really mean this?:
> [[
> From: Bruce Lilly <[email-masked]>
> Reply-To: Martin D�rst <[email-masked]>,
>     ietf-822 <[email-masked]>
> ]]

Yes, that's intentional (i am subscribed to the list; therefore
don't need a separate copy, and discussion should probably
go to the list; Martin had requested copies of the discussion
of Archived-At as he is not subscribed to ietf-822) -- the use
of Reply-To for lists is documented in RFC 822 and its
predecessors, and has been recently discussed on ietf-822.
[The Reply-To field had better have RFC 2047 encoding
rather than raw 8-bit characters, though]

Date: Mon, 1 Nov 2004 16:45:11 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: a header authentication scheme
Sender: [email-masked]

Claus Assmann writes:
> On Fri, Oct 22, 2004, Arnt Gulbrandsen wrote:
>>  Step 1. I'd send five messages to [email-masked] and wait for 
>>  the bounces. For each of them, I'd compute the delay from my 
>>  sending time (using my clock) to the receiving MTA's receive time. 
>>  Next, I'd average the five values.
>
> Where do you get the time of the server? Many MTAs don't accept mail 
> for unknown recipients, so it's your MTA that generates the bounce.

I don't suppose it matters, since even in the best case, there's very 
little entropy in the time-date value, so id has to be better.

But since you ask: Usually I can get time and timezone from the SMTP 
server's banner. I can also often guess based on its geographical 
location (which I can guess from IP address). This won't get me the 
delay, but if all I want is to spam n million recipients, assuming 
[1,15> is pretty good. Few MTAs are faster, few are slower.

Arnt

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: a header authentication scheme
Date: Mon, 1 Nov 2004 08:19:32 -0500
Sender: [email-masked]

On Sun October 31 2004 01:54, Laird Breyer wrote:

> > Given a message with a valid Received field; it's trivial; simply
> > construct a Processed field accordingly and insert it where
> > desired, eliding any preexisting "processed" field that corresponds
> > to the Received field.  W/o any Received field; forge one and
> > insert it and a corresponding "processed" field where desired.
> 
> I'm afraid none of this is yet convincing as a successful attack. 
> 
> You must be able to forge the Processed field before the message
> contains the pertinent Received field, otherwise you're not actually
> forging anything of value. In fact, quoting existing Received fields
> is legal, and marks your location correctly. 

You have missed the points; at any point a party can
a) s/result-tag="spam"/result-tag="ham"/
b) insert a Received field, then insert a "processed" field referring
    to that Received field, with any desired "processed" field content
N.B. neither tactic would be effective if MIME security multiparts
were used.

> If you forge an appropriate Received field as well, you still have no
> control over the Received fields added by the subsequent receiving
> systems

Again, you have missed a crucial point; there may be no
"subsequent receiving systems", and even if there are
there might be no added Received fields.

> Nor can you assume that 
> the ultimate reader will trust your forged Received field, whence your
> associated Processed field.  So your forgery will be trivial to detect

Evidently you have little experience in examining Received fields;
detecting forgeries is not always trivial (in part because legitimate
fields are often screwed up, in part because sometimes fields
are reordered, in part due to host aliases, etc.).

> That's a good point. I believe the Processed field degrades gracefully
> in this respect, since if there is no existing Received field to be
> quoted, a Processed field cannot quote it. A later reader of this
> Processed field will be immediately alerted to the fact by the absence
> of an "auth-received" key value.

You are again ignoring the fact that it is trivial to insert a Received
field, then "quote it".

> > I cannot interpret (in any meaningful sense) the "result-tag='spam'"
> > part without knowing specific configuration information and run-time
> > environment information for the instance of "SpamAssassin" version
> > "2.63" from which the purported "result-tag" was allegedly obtained.
> > I know nothing about its spam threshold, the myriad of fudge factors
> > that are used to weigh various characteristics, whether any address
> > patterns are used as whitelists or blacklists and if so what those
> > patterns are, etc.
> 
> This makes no sense as an objection. The Processed field is a record
> of what some other process computed about the message, it has no
> other meaning.

There may well be no "process"; the field can be trivially forged.
It is an assertion that there was a process, which was allegedly
run with unspecified configuration and unspecified run-time
environment and allegedly making some value judgment about
the content of some message (presumably the one containing
the field).    For a concrete example, suppose that some recipient
is interested in a home mortgage.  A spam filter that is configured
to treat all messages containing the word "mortgage" as spam is
worse than useless to that recipient; it is counter-productive. And
on the basis of the (lack of) information in your proposed
"processed" field, no recipient is able to determine whether or not
configuration of any alleged process is appropriate *for him*.
 
> All you need for interpretion is to know what this other process
> intended the fields to mean, and that's to be detailed in a common
> standard.

No, one needs to know configuration and run-time environment.
 
> As long as processes follow the standard, the meaning will be clear,
> viz.  "the process writing this field considers the message to be
> spam".

There's no guarantee that there was ever any such "process".
Moreover, the assertion that some entity considers a message
to be spam is useless unless one knows precisely what went
into making that judgment (well not entirely useless; one could
object to a third party eavesdropping on personal communications).
 
> If I may borrow your language: the Processed field is specifically
> provided as a means for filter processes to suggest facets of the 
> message as observed during transit. 

Please read carefully RFC 1958 on the Internet Architecture,
paying particular attention to "end-to-end".  A Reply-to
field is a conduit for information between endpoints (author(s)
and recipient(s)) in communication.  An assertion made by some
intermediate system which is supposed to be providing transport
services simply fails to fit into the Internet Architecture on many
levels:
a) is is not an end-to-end process
b) eavesdropping on private communications is immoral and
    offensive (and may be illegal in some places)
c) it is not under control of either party in the end-to-end
    communication
etc.

> There are a multitude of X- headers used by filters, each with its own
> syntax and interpretations, which are currently unparseable by all the
> other filters.

And they're all useless and objectionable, for the same reason
that spam itself is objectionable; unwanted cruft that has cost
in terms of communication bandwidth and storage space.

> For the subset of spam filters, a commonly adopted 
> format will allow better filter cooperation

What "filter cooperation"?  In precisely what way do filters
at separate sites need to "cooperate", and how is that practical
without detailed configuration information?

> and reduce the need for 
> future filters to reinvent variations on the same format.

There simply is no "need" to add cruft to messages; that is
a problem (inappropriate consumption of communications
bandwidth and storage space resources, viz. the same problem
caused by spam; beware ostensible "solutions" that are as
bad as the symptoms).

> I still don't see it. To paraphrase your request for a complete
> record of the filter configuration: the Received line "from" component
> is useless unless I have complete information about the state of the
> DNS system in the world at the moment of the transaction, as well
> as a complete record of which IP addresses are used in the world
> at that point in time. So in this sense, the "from" component is
> simply not well defined due to lack of information. 

You are simply wrong; no matter what "the state of the DNS system
in the world at the moment", and completely independent of
"which IP addresses are used in the world", a record of the argument
used in a particular SMTP transaction is what it is. That fact is
invariant w.r.t. DNS and IP address allocations.  It is useful for
the intended purpose of the field, viz. "for humans tracing
mail routes, primarily of diagnosis of faults"; it can be subsequently
used to identify a particular transport transaction performed by
an MTA in the performance of its message transport duties.

Date: Mon, 01 Nov 2004 10:14:31 +0000
To: Bruce Lilly <[email-masked]>, ietf-822 <[email-masked]>
From: Graham Klyne <[email-masked]>
Subject: Re: the gap regarding Archived-At
Sender: [email-masked]

At 14:27 30/10/04 -0400, Bruce Lilly wrote:
> > Yes. One issue is that if there are a lot of comments fields,
> > the user will have difficulties to choose the right one.
>
>The same could be said of archived-at and x-archived-at.

I gave some further thought to the issue of Achived-At vs a general comment 
field and, based on my experience, come to the view:

- [X-]Archived-At as currently used does not specifically need to be a 
structured field in the RFC822 sense, but

- it is useful to use a separate header field name because this can be used 
by an MUA as a basis for determining whether or not to display the field 
value.  My MUA, which certainly has no knowledge of this specific header 
field, still allows me to do this much.  Using a generic comment field 
would obviate this benefit.

In saying this, I don't deny Keith's point about MUAs also being able to 
use the field -- I'm just responding further the earlier question about 
utility of the field as currently proposed.

#g
--

BTW, Bruce, your use of reply-to header fields is confusing me -- you 
nearly didn't get a directly addressed copy of this message.  Did you 
really mean this?:
[[
From: Bruce Lilly <[email-masked]>
Reply-To: Martin D�rst <[email-masked]>,
    ietf-822 <[email-masked]>
]]

------------
Graham Klyne
For email:
http://www.ninebynine.org/#Contact