Subject: Re: Revisiting RFC 2822 grammar (obs-utext and unstructured)
From: Bruce Lilly <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Date: Fri, 30 Jan 2004 20:05:48 -0500
List-Id: <ietf-822.imc.org>

Pete Resnick wrote:
> 
> On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:
> 
>> In <[email-masked]> [email-masked] writes:

>>> obs-utext       =       *LF *CR *(obs-char *LF *CR)
>>> [N.B. was obs-text]
>>
>>
>> No, that does not work because it allows CRLF not followed by WSP in
>> the middle of an 'unstructured'.
> 
> 
> Yup, because you can have "obs-utext obs-utext" which could be (abc
> CR)(LF def).
> 
>> I think the only way out of that is to rewrite the rule for
>> 'unstructured':
>>
>> unstructured    =    *(utext [FWS]) obs-ltext
>> obs-utext    =    (1*LF *CR / 1*CR) obs-char / NUL
>> obs-ltext    =    *LF *CR
> 
> 
> Blech. I'll take a look and see what I can figure out without resorting
> to that.

I think that the following does the right thing (obs-utext can be empty
or it can start or end with any ASCII octet and can have any sequence
except CRLF, which is handled in unstructured by FWS, and unstructured
can be completely empty or it can begin or end with utext or FWS, but
any instance of utext is separated from any other instance by FWS):

obs-utext       =       *(*obs-char (*LF / (*CR 1*obs-char))) *CR

unstructured    =       *(utext FWS) *utext

Date: Mon, 26 Jan 2004 16:30:02 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: James M Galvin <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: cryptographically verifiable fields
Sender: [email-masked]

James M Galvin wrote:

>So you're in need of a "PKI", essentially.  Well, if we had DNSSEC you
>would have one, ideally suited for this application since given an email
>address you've got all the information you need for a lookup and the
>machinery to do it.
>  
>
I've looked at the DNSSEC RFCs and reviewed the base DNS RFCs, and it's
not entirely clear how DNSSEC would help with cryptographic verification
in an email context.

I had suggested an SMTP-based mechanism which would
a) localize most of the implementation details in SMTP MTA implementations
b) localize the administrative overhead to SMTP MTA administrators

It seems that any DNS-based approach is going to add load to DNS
administrators.
Particularly for large organizations, that may be impractical.  This
issue should
not be underestimated; there are typically many more email addresses
than hosts
in a given domain.

Since signing a portion of an email message would have to be performed by an
MUA or MTA, it appears that an approach involving DNS for distribution of
public keys still requires handling the private keys in the MUA or MTA, and
therefore involves synchronization and consistency between two separately
administered protocols (DNS and SMTP).

As email addresses are not domain names per se, there is also the
question of
how well DNS would be able to handle some aspects of distribution of public
keys for email addresses:
a) RFC 1035 section 2.3.4 limits DNS names to 255 octets, whereas an email
    address might be 64 + 3 + 255 = 322 octets (RFC 2821) [addresses
might be
    longer, but RFC 2821-conforming SMTP implementations are required to
support
    at least that much].  One of the benefits of looking up the public
key for a given
    email address via an SMTP server registered as handling email for
the corresponding
    domain is that that server presumably can handle those email addresses
    appropriately (including length, case-sensitivity (see below),
multiple addresses
    per account, etc.).
b) it's not entirely clear what name would be used for a particular
email address:
  i) RFC 1035 section 2.3.4 requires case-insensitivity, which is fine
for a domain
      name, but not necessarily so for the local-part portion of an
email address
  ii) it's not entirely clear whether RFC 1035 section 8 mailbox
encoding should
       be used.  It appears to be unnecessary, and 1035 doesn't
specifically deal with
       email addresses as names (vs. RR data) -- this would have to be
clarified.
  iii) then there's section 8.2 experimental mailbox binding (said
experiment
         presumably having been abandoned)
c) it's unclear how well DNS implementations would be able to scale up to
    handling KEY RRs for large numbers of email addresses within a domain.
    In particular, zone transfers may be unwieldy.
d) what happens if some DNS server starts handing out bogus keys for non-
    existent email addresses? [remember the Verisign (and others)
wildcard DNS
    shenanigans?]

Some of these issues may of course have counterparts in the case of the
suggested
SMTP-only method; others are not applicable.  Issues that would have to
be fleshed
out include:
a) some means of key distribution among multiple SMTP servers operating
as MX
    specified SMTP servers for a particular domain
b) protection of the private keys is an obvious concern
c) some large installations utilize separate groups of servers for
originating
    messages (where signing would take place using private keys) and for
receiving
    messages (registered via MX, and presumably where requests for
public keys
    would be handled).  So in those cases, some synchronization would be
required,
    though that would be within the province of email administration
rather than
    crossing email/DNS administrative boundaries.
d) some abuses may still be possible, but handling public keys via SMTP
servers
    registered as MX handlers reduces the scope for abuse (since wildcard MX
    records are explicitly allowed).  The suggested SMTP-only mechanism
    specified retrieval of public keys from MX handlers specifically to
mitigate
    the potential for abuse.
e) MUAs and/or MTAs that wish to make use of signatures could arrange to
    cache public keys locally rather than retrieve them for each
instance.  While
    use of DNS could automatically cache results, the necessary
information is
    essentially invariant (one may need to verify the signature of a message
    received in the distant past), and such information would probably
best be
    cached by email-specific software rather than consuming generic DNS
cache
    memory with application-specific (email) data.  Caching is of course
not a
    necessity, but could reduce network load.

Date: Fri, 23 Jan 2004 14:08:50 -0800
From: Lyndon Nerenberg <[email-masked]>
To: [email-masked]
Cc: Paul Smith <[email-masked]>
Subject: Re: X-* header fields
Sender: [email-masked]

--On 2004-1-23 2:33 PM +0000 Paul Smith <[email-masked]> wrote:

> What I'd quite like is a way of registering a header field (including
> basic semantics, syntax and an ABNF definition (for many existing X-
> fields this could be trivial with some examples to work from)). This
> would go to a group of 'learned volunteers' who make sure it's not
> abusive (ie not 'grabbing' a useful header field name for an
> inappropriate purpose, or not commercially advantagous to someone to
> the detriment of others etc) and that's it, registered. The 'learned
> volunteers' may also give feedback to the registrant saying something
> like 'we think this is a really good idea, but needs developing in
> such and such a way', and then they reserve the header field name, to
> give the full registration time to be improved.

"Header hijacking" is also one of my major concerns, and I'm in favour 
of what Paul is suggesting.

If X-* dies I suppose the experimental use folks could use something 
like a base64-encoded hash of the proposed header name (or any other 
method that would generate random strings with a suitably low 
probability of namespace collisions).

--lyndon

Date: Fri, 23 Jan 2004 13:57:49 -0800
From: Lyndon Nerenberg <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

--On 2004-1-22 5:08 PM -0800 [email-masked] wrote:

>> My own, primary concern about them is having an "experimental" header
>> become a defacto standard and then having to find a way to get folks
>> to use a new, non-X header name for it.  I believe that hasn't been
>> successful yet, or at least not much.
>
> This is my major concern with them as well.

But how often has this even been tried? If there have been *serious* 
efforts to attempt this in the past it would be useful if someone could 
document in detail the process taken to make the change, and provide an 
analysis of why it failed.

--lyndon

Date: Sat, 24 Jan 2004 13:35:24 -0800
To: (various IETF lists)
From: Paul Hoffman / IMC <[email-masked]>
Subject: New mail-ng mailing list open for sign-ups
Sender: [email-masked]

Greetings again. There seems to be more discussion these days about 
replacing SMTP and/or RFC 2822 and/or POP/IMAP for a variety of 
reasons. The discussion seems to pop up on a few different lists and 
in a few different hallways, and it might be good to have a single 
list where folks can congregate. Thus, I have set up a mail-ng 
mailing list.

The first task probably is to determine what the next generation of 
mail should do, and how that is different than what 
SMTP/2822/POP-or-IMAP or instant messaging does. It is safe to say 
that we can ignore actual protocol proposals for many months (if not 
years) until we figure out what is needed. Once we do that, there are 
plenty of protocol people who can attack the decided-on requirements.

There is no expectation that there will be significant agreement on 
the list. It is likely that over time the discussion will split into 
camps of like-minded design goals. The list might then spawn 
different lists for the folks of the different camps (mail-ng-shoe, 
mail-ng-sandal, ...). The list is explicitly not yet meant to be an 
IETF working group yet (if at all), and is probably more akin to the 
IRTF. But at the beginning, it will most likely be talking, not 
research.

See <http://www.imc.org/mail-ng/> for information on how to 
subscribe. The list is taking subscriptions now, and will probably go 
live for discussions within a week. Having some discussion on a 
mailing list now should make the dinners and bar gatherings at Seoul 
more interesting.

--Paul Hoffman, Director
--Internet Mail Consortium

Date: Sat, 24 Jan 2004 15:33:25 -0500
Subject: Re: making mail traceable
Cc: [email-masked]
To: James M Galvin <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

On Wednesday, January 21, 2004, at 01:12  PM, James M Galvin wrote:

> Just to be clear, when you use "anonymous" you truly mean anonymous in
> the sense of the "remailers" that we used to hear so much about.
> .....
> Would you permit "site traceability" as compared to "author 
> traceability" or is that too in conflict with anonymity?

Well, I'm cynical enough to believe that both perfect anonymity AND 
perfect tracability are impossible.  So we're always talking about 
degrees of difficulty.  In practical terms, I think the ideal would be 
a level of anonymity in which a message could easily be traced to the 
first "reputable" ISP that permitted the message onto the net, but the 
degree of tracability from there would be highly ISP-dependent.  An ISP 
(or anonymous relay) that wanted to be known as unfriendly to spammers 
but friendly to anonymous email would have to implement some special 
measures to keep spammers off *their* system, but I'm not convinced 
that it would be impossible, by any means -- and much easier for a 
single service than a standards body.  So, yes, imagine a traditional 
anonymous remailer augmented with additional, possibly proprietary 
mechanisms to deter spam.

My point is that there are certain types of message that should be easy 
to trace to their senders, and certain types that should be hard to so 
trace, and we should do our best to accommodate both.  The relevant 
variables include not only the originator but also the volume of 
messages from the same source, and the preferences of the recipient.

By the way, I've just become aware of a clever new idea for an economic 
solution to the spam problem, and it's not just another recycling of 
the idea of charging postage for email.  It's called "Attention Bonds" 
and is worth a read for the long-term thoughts it provokes:

	http://papers.ssrn.com/sol3/papers.cfm?abstract_id=488444

Cheers.  -- Nathaniel

Date: Fri, 23 Jan 2004 10:51:19 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Bruce Lilly <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

(one more response since this is back on the original topic of my
message)

> [email-masked] wrote:

> > The bottom line is that the success of x- fields in other places has anywhere
> > from disasterous at worst to mixed at best. And to a large extent the lack of
> > harm in many spaces has come from lack of use. So if you're looking for support
> > for keeping the X- distinction from the successful use of X- in other contexts,
> > I'm afraid it is not forthcoming.

> For the record, I posed the question because there was a great deal of
> discussion regarding X- header fields, but I can recall no recent discussion
> about other use of x- for experimental and private use; as I mentioned, there
> are many instances of such use in text messaging protocols (I was a little
> surprised to see how many, and my list was not intended to be exhaustive -- I
> simply listed the ones that I had implemented in one form or another). That
> struck me as curious, hence my question about what was so unique to X- header
> fields that caused so much discussion, whereas there was virtually no
> discussion about other use of x- for similar purposes.  And if indeed use of x-
> in other places has been disastrous, why isn't there a call for its removal
> from those places?

What makes you think there hasn't been such a call? The fact of the matter is
that X- media types have long been hanging by a thread. There most certainly
was a call to take them out at the time RFC 2048 came out, citing interop
problems. They remained only because the text was redone to discourage their
use and this was deemed to be sufficient. I was seriously tempted to remove
them in the recently approved RFC 2048bis but the mandate behind revision was
to make the minimal set of changes needed to open up the IETF tree to other
standards bodies, not to clean up other aspects of the specification. Even so,
there was some mumbling that retaining them wasn't a good idea. I certainly
would not bet the farm on their continued inclusion in any RFC 2048ter,
especially if the recent DNS tree proposal gains any traction.

There have been other, similar queries as to the wisdom of X- fields in other
contexts, but in most cases the lack of use of such fields is cited as a reason
not to change the status quo. The same cannot be said of X- headers, of course.

				Ned

Date: Fri, 23 Jan 2004 09:33:52 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> "Al Costanzo" <[email-masked]> writes:

> >A clean, simple, fast way to register the non X- header field would do the
> >trick..

> Which is exactly what the Klyne draft provides.

> It is currently stuck in last call in the IESG. They need to get it out
> ASAP.

Actually, if you bother to check the datatracker you'll see that the
document is in IESG evaluation. More specifically, a response was
sent to the two ADs with outstanding discuss votes and we're now waiting
for the two ADs to either (a) Clear their discusses or (b) Respond in some
other way.

				Ned

Date: Fri, 23 Jan 2004 07:59:45 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Bruce Lilly <[email-masked]>
Cc: [email-masked], Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> >But it is far from clear that the cases where X- field use is causing interop
> >problems actually constitute "misuse". For example, there have been cases where
> >two parties opted to use the same X- mail header field for their own private,
> >but sematically different, purposes. (X-Sender is the example that comes to
> >mind.) The specification of X- in RFC 822 doesn't give you much leverage to
> >argue that either of them were "misusing" the field.

> Would the situation have been any better if some other name (i.e. not
> beginning with X-) had been used w/o registration?

Quite possibly. The problem with X- is that it is easy and encourages
unilateral action. Had there been (a) No X- and (b) A simple registration
scheme it is entirely possible that the first user of such a field would have
registered it and subsequent users would have picked up the registration.

And there is precedent for this. When media types were first defined
registration, although possible, was too onerous. This led to a lot of ad-hoc
use. Subsequent revisions to the process have given us a significant rise in
the registration rate. But to the extent X- types were or are still used, they
now constitue a problem rather than a solution.

> If there is a conflict in use of an X-
> field, the responsible parties can a) work out the issue and come to common
> usage or agree to change one or more of the names, or b) the conflict can
> continue.

This assumes that the responsible parties are willing able to get together and work
such things out after the fact. This almost never happens in my
experience.

> With other field names and with current procedures, there is
> another option, viz. one of the parties can issue an RFC registering the name.

There have been attempts to do this in the past. The degree of pushback
experienced killed almost all of them.

> There's no
> guarantee that the IETF will be aware of the other use, and it may well
> be the
> case that a more useful (unregistered) field is ignored.  Moreover,
> there's no
> guarantee that use of the same name will not continue for the unregistered
> purpose (and no official means of enforcement).  And as has been noted by
> others in this discussion, registration without vetting is likely to make matters
> worse, e.g. by polluting the non X- namespace with items of questionable (or
> worse) value.

The incredible number of unregistered non-X- fields in wide use discredits this
assertion rather completely, I think. And again, I would much rather know how
something was designed to be used, warts and all, than have to guess.

> > I see it the other way around. Given that interop problems are known
> > to exist the burden should instead be on the proponents of X- to
> > make a case for the continued inclusion of the feature being worth it.

> Let's keep things in perspective. Have the interoperability problems
> with X- header fields been minor annoyances, or have they caused massive
> disruptions?

They have on occasion caused massive disruptions. X-PMrpc is an example of a
field that caused serious operational problems at one point; there are others
as well.

> Would things have been any better without X- given the current
> realities (e.g. no registration procedure other than publication of an RFC, no
> means of enforcement) and given similar circumstances (i.e. use of the same
> field name for different purposes)?

This is soon not to be the current reality. A registration proposal is in front
of the IESG now. And it changes things substantially. Without it I see little
point in worrying about X-. But once it is in place I think X- becomes actively
harmful and I don't want to see it reintroduced.

I really didn't intend to be drawn into this whole debate; my only purpose in
posting my initial message was to answer your request for citations of X-
experience in other areas (which we have seen does not exactly constitute a
ringing endorsement of the concept). So I'm cutting this off now and it is
unlikely I will post additional messages on this topic.

				Ned

To: [email-masked]
Xref: clerew local.mime:3015
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: X-* header fields
Date: Fri, 23 Jan 2004 14:06:29 GMT
Lines: 20
Sender: [email-masked]

In <[email-masked]> "Al Costanzo" <[email-masked]> writes:

>A clean, simple, fast way to register the non X- header field would do the
>trick..

Which is exactly what the Klyne draft provides.

It is currently stuck in last call in the IESG. They need to get it out
ASAP.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 23 Jan 2004 09:53:40 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

[email-masked] wrote:

>But it is far from clear that the cases where X- field use is causing interop
>problems actually constitute "misuse". For example, there have been cases where
>two parties opted to use the same X- mail header field for their own private,
>but sematically different, purposes. (X-Sender is the example that comes to
>mind.) The specification of X- in RFC 822 doesn't give you much leverage to
>argue that either of them were "misusing" the field.
>  
>
Would the situation have been any better if some other name (i.e. not
beginning
with X-) had been used w/o registration?  If there is a conflict in use
of an X-
field, the responsible parties can a) work out the issue and come to common
usage or agree to change one or more of the names, or b) the conflict can
continue. With other field names and with current procedures, there is
another
option, viz. one of the parties can issue an RFC registering the name. 
There's no
guarantee that the IETF will be aware of the other use, and it may well
be the
case that a more useful (unregistered) field is ignored.  Moreover,
there's no
guarantee that use of the same name will not continue for the unregistered
purpose (and no official means of enforcement).  And as has been noted by
others in this discussion, registration without vetting is likely to
make matters
worse, e.g. by polluting the non X- namespace with items of questionable (or
worse) value.

>I see it the other way around. Given that interop problems are known
>to exist the burden should instead be on the proponents of X- to
>make a case for the continued inclusion of the feature being worth it.
>  
>
Let's keep things in perspective. Have the interoperability problems
with X- header fields been minor annoyances, or have they caused massive
disruptions?  Would things have been any better without X- given the current
realities (e.g. no registration procedure other than publication of an
RFC, no
means of enforcement) and given similar circumstances (i.e. use of the same
field name for different purposes)?

>Less is usually more as far as standards go.
>  
>
True, but changing horses in mid-stream is usually a bad idea as well,
in standards
as well as in other endeavors.

>>At one time it was widely
>>believed that providing a special, reserved space of protocol fields
>>would discourage pollution of the normal space.   Personally I'm glad
>>that we encouraged people to use X- because I think that if those fields
>>didn't use X- there would be a lot more bogus and poorly-defined
>>fields out there and it would be more difficult to distinguish those
>>fields from fields that should be implemented than it is now.  Which is
>>not to say that there aren't bogus non-X- fields out there, just that
>>X- helps identify _some_ of the bogus fields.
>>    
>>
>
>First, since there are huge numbers of bogus header fields (I have 587 on my
>personal list of such things, about evenly split in their use or non-use of X-)
>both with and without X-, I don't buy the argument that X- has helped in any
>material way.
>  
>
Well you really have two lists of about equal size, according to the RFC
822 rules;
one is a list of bogus (unregistered) fields and the other is a list of
experimental or
private-use (unregisterable) fields.  And only one of those lists has
names that could
(under RFC 822 rules) potentially conflict with future registered names.

>Second, the existance of X- has led to the use of well-defined fields in the X-
>space that now cannot be standardized because of the X- rule. The best example
>of this I know of is X-Accept-Language but I'm sure there are others.
>  
>
Such well-defined fields, if also widely used, would constitute
successful experiments.
Accept-Language has been standardized (RFC 3282).  If it's useful and
widely used
(I'm not convinced of either) those who were sufficiently motivated to
participate in
the experiment are also sufficiently motivated to use the official
field. If so, supporting
that change is not materially different (for parties who elected to
support the X-
experiment) from registered header fields whose names have changed
(e.g. Obsoletes -> Supersedes).  For others (who have ignored the
experiment) it's
simply another new header field.  And in the specific case of
X-Accept-Language, lest
you look in this message's header and find it there; the transition
gives some time
for broken implementations (such as Mozilla's) to be repaired in the
process.

>I've also seen cases of fields where the contents are reasonably well defined
>but the field names vary, to the detriment of interoperabilty. For example, it
>is often useful for different gateways to the same type of non-Internet mail
>system to cooperate in their specification of a fields used to carry
>information useful to the non-Internet system they gateway to. (X-VMSMail-To:
>is one example of this, there are many others.) Although there tend to be
>scoping issues associated with the use of such information, the fact remains
>that it would be very useful for vendors of different gateways to the same
>system to be able to agree on a single field name. But the use of X- fields for
>this sort of stuff (which does seem like an obvious choice) has precluded
>specification of such field names in these cases.
>  
>
I don't get your point. If X-Accept-Language has been sufficiently
successful to result
in a standardized version with a registered name, how do the RFC 822
provisions
preclude a similar thing from happening for gateways?  Moreover, there is a
counterexample of sorts; the MIXER RFCs do standardize such things for
gateways
between two specific mail environments.

>>Most of the X- fields that have been deployed shouldn't have been
>>deployed in shipping product with or without the X-.
>>    
>>
>
>Agreed, but this doesn't make the cases where something was improperly
>deployed with X- go away.
>
>  
>
True, but is does prevent them from clashing with registered field
names, past,
present, and future.

>>Furthermore,
>>there has never been a prohibition on defining new fields without X-.
>>    
>>
>
>Which further dilutes the need for X- IMO.
>
>  
>
There are some important distinctions to be made here. Of course there's
no prohibition against
registering new field names -- indeed that is explicitly part of the
extension process
provided for by RFC 822.  While there is no official means of
enforcement preventing use
of unregistered field names, RFC 822 explicitly cautions that the
implementor who does
so is crawling out onto a limb that may be officially cut off behind
him. And it provides
a guarantee that that will not happen with an X- field.

Perhaps there should have been a prohibition against use of unregistered
non-X- field
names.  But such a prohibition would be meaningless in practice without an
enforcement mechanism.

>Sure this is a problem but this isn't the problem getting rid of X- is intended
>to solve.
>  
>
Exactly which problem is it intended to solve, and how is it proposed
that it will
do so?  Especially given that X- header fields have been around for more
than two
decades, will not magically disappear from "old" stored messages, and
lacking an
effective enforcement mechanism will never disappear from new messages.

>This is why we're trying the provisional registry approach.
>  
>
[...]

>The best tack we've been able to come with is what's in the registry
>specification. Of course it remains to be seen if it will actually help.
>But I agree that the X- issue is small beer in this larger context.
>  
>
Wouldn't it be preferable to retain X- while that approach is tried;
that way if the
provisional registry is a disaster or fails to achieve it's purported
goals, we'll still
have an established fallback.  If X- is somehow abolished and the
provisional registry
fails, then what?

Date: Fri, 23 Jan 2004 14:33:25 +0000
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: X-* header fields
Sender: [email-masked]

> >A clean, simple, fast way to register the non X- header field would do the
> >trick..
> >
>Is that possible? Ignoring "fast" for the moment, my definition of
>"clean" is that the registration requires specification of semantics and 
>syntax, including ABNF.
>
>That may conflict with somebody else's definition of "simple".

Well, I'd expect a specification of semantics, syntax and an ABNF 
definition. That doesn't stop it being relatively simple (with guidance notes).

What would make it slow is an extensive 'approval' system as there is now.

What I'd quite like is a way of registering a header field (including basic 
semantics, syntax and an ABNF definition (for many existing X- fields this 
could be trivial with some examples to work from)). This would go to a 
group of 'learned volunteers' who make sure it's not abusive (ie not 
'grabbing' a useful header field name for an inappropriate purpose, or not 
commercially advantagous to someone to the detriment of others etc) and 
that's it, registered. The 'learned volunteers' may also give feedback to 
the registrant saying something like 'we think this is a really good idea, 
but needs developing in such and such a way', and then they reserve the 
header field name, to give the full registration time to be improved.

An automated register might then allow people to register 'interest' in a 
registered header field if they are MTA/MUA developers (eg 'we implement 
this', 'we plan to implement it', 'we hate it and think the author ought to 
be shot', etc) so that other potential users of the field would know 
whether it's worth bothering about.

JM2P

(I'm sure we've been through all this a year or two ago..)

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Fri, 23 Jan 2004 08:32:38 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

[email-masked] wrote:

> The bottom line is that the success of x- fields in other places has
> anywhere
> from disasterous at worst to mixed at best. And to a large extent the
> lack of
> harm in many spaces has come from lack of use. So if you're looking for
> support
> for keeping the X- distinction from the successful use of X- in other
> contexts,
> I'm afraid it is not forthcoming.

For the record, I posed the question because there was a great deal of discussion regarding
X- header fields, but I can recall no recent discussion about other use of x- for experimental
and private use; as I mentioned, there are many instances of such use in text messaging
protocols (I was a little surprised to see how many, and my list was not intended to be
exhaustive -- I simply listed the ones that I had implemented in one form or another). That
struck me as curious, hence my question about what was so unique to X- header fields that
caused so much discussion, whereas there was virtually no discussion about other use of x-
for similar purposes.  And if indeed use of x- in other places has been disastrous, why isn't
there a call for its removal from those places?

Date: Fri, 23 Jan 2004 08:18:50 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Al Costanzo <[email-masked]>, [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

Al Costanzo wrote:

>A clean, simple, fast way to register the non X- header field would do the
>trick..
>
>Al
>  
>
Is that possible? Ignoring "fast" for the moment, my definition of
"clean" is that the
registration requires specification of semantics and syntax, including ABNF.

That may conflict with somebody else's definition of "simple".

Date: Fri, 23 Jan 2004 10:13:43 +0000
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: X-* header fields
Sender: [email-masked]

At 00:43 23/01/2004, Dave Crocker wrote:
>My own, primary concern about them is having an "experimental" header
>become a defacto standard and then having to find a way to get folks to
>use a new, non-X header name for it.  I believe that hasn't been
>successful yet, or at least not much.

I think it is arguable that you need some method of having 'experimental' 
headers 'in the wild'.

Getting rid of X- headers and saying 'you must come up with an agreed 
standard before you can add a new header of any sort' is going to inhibit 
innovation.

At least if you use X- headers, and they become defacto standards, then you 
can tell people off ;-)

One example I can think of of standards changing and people keeping up is 
with SMTP authentication. In the initial drafts of SMTP authentication the 
server would say 'AUTH=LOGIN'. In the final standard, the server says 'AUTH 
LOGIN'. Lots of people used the initial draft because authentication was 
needed so much, but then changed it to AUTH LOGIN (or supported both, in 
many cases) when the final standard came out.

I don't see a big problem with that. Yes, it would have been nice to get it 
right the first time, but if people hadn't been willing to risk getting it 
wrong and then having to put it right, we might never have ended up with an 
SMTP authentication standard!

If you look at other situations, eg the extra trace information that people 
are discussing in the SMTP groups, it's quite possible that someone will 
come up with a workable idea for this and then people may implement it, and 
later problems may be found, and the information will have to change.

If that's done, then either we'd have to use a header like 'Originator', 
and then risk total inoperability by changing the meaning of that header, 
or make 'OriginatorV2', 'OriginatorV3' etc when the changes happen, and not 
be able to use the nice 'Originator' ever again, because that's already 
been used.
Or, we could use 'X-Originator', then 'X-Originator2' etc, but then, when 
everyone's happy with the idea, we could use 'Originator', and deprecate 
the 'X-Originator' ones. Over time people would move to the Originator 
header (they may still READ the X-Originator ones, but wouldn't write them)

(This is just an example - yes, it would be nice to get it right first 
time, but that doesn't always happen cf SMTP authentication).

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Fri, 23 Jan 2004 10:22:07 +0000
To: <[email-masked]>
From: Paul Smith <[email-masked]>
Subject: Re: X-* header fields
Sender: [email-masked]

At 01:21 23/01/2004, you wrote:

>A clean, simple, fast way to register the non X- header field would do the
>trick..

Almost certainly, yes.

The primary reason "good idea" header fields don't get made into standards 
is probably that people can't register them. Trying to get a standard 
approved seems to be a long, arduous task, with lots of heads meeting lots 
of walls.

There are quite a few non-standard headers (X- ones as well as non-X- ones) 
out there which seem to be good ideas to some people - but maybe not 
everyone agrees (eg Delivered-To, Priority, X-RCPTTO etc). I'd actually bet 
that trying to get those through to be standards would be a nightmare, even 
though it'd be nice for many people if they were standards (without the X- 
in the case of X-RCPTTO), so, people use them, and try to guess what they 
should mean, and how they should be used. I've seen many different ways 
that all those headers are used.

This doesn't argue against X- headers. It just argues against having useful 
non-standard headers. One way to solve the problem is to stop people using 
them. The other (better, IMHO) way is to make it easier for people to make 
them into standard headers. If people are trying to use non-standard 
headers they're not creating themselves, then it means they're useful 
(otherwise people wouldn't be trying to use them)...

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Fri, 23 Jan 2004 09:57:02 +0000
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: X-* header fields
Sender: [email-masked]

At 17:46 22/01/2004, Keith Moore wrote:
>Misuse of any protocol feature can cause interoperability problems.
>That is at least as true for X- fields (which IMHO should never have
>been put in shipping products except perhaps as write-only fields) as
>for non X- fields (which should never be in shipping products without
>public vetting).

Hmm.

I'm one of the people who's moderately happy with X- fields when used 
properly (as opposed to using a non-standard non-X- field)

But, I could very easily be convinced that X- fields should be write-only 
in shipping products. But, I'd prefer to see that than people using 
non-standard non-X- fields, or not being able to add any field without 
public vetting.

Having a computer readable X- field is 'dangerous' IMHO - for the same 
reason that using a non-standard non-X- field would be - interoperability. 
Having write-only X- fields doesn't have anywhere near the same amount of risk.

So, I'd be happy to see a 'X- fields SHOULD be write-only' type clause in 
the standard.

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Cc: Keith Moore <[email-masked]>, Dave Crocker <[email-masked]>, [email-masked], [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: X-* header fields
Date: Thu, 22 Jan 2004 22:51:35 -0500
To: [email-masked]
Sender: [email-masked]

>> > My own, primary concern about them is having an "experimental" 
>> header
>> > become a defacto standard and then having to find a way to get 
>> folks to
>> > use a new, non-X header name for it.  I believe that hasn't been
>> > successful yet, or at least not much.
>
>> why does it work better in the case of Experimental vs Standard MIBs?
>> (or do you think that doesn't work well either?)
>
> First, I don't know that it works better. I suspect it does not, and 
> that once
> an experimental MIB is in use it is very difficult to get rid of it. 
> Indeed, if
> such transitions worked well, why isn't the accepted practice for MIBs
> developed in the IETF to assign an experimental OID during development 
> and
> switch to a standard one after IESG approval? It sure isn't done this 
> way -
> what happpens in practice is that there is NO OID assigned in the 
> document
> until after it is approved.

I think that's a change since I was on IESG - I remember having the 
discussion about having to change OIDs from the experimental branch to 
the standard branch more than once.  Maybe not having any OID assigned 
until standardization (presumably each vendor assigns its own for 
prototyping) is a newer strategy.

>  The difference between a MIB registered in the experimental space
> and a MIB registered in the standards space is simply the OID prefix.

That's assuming, of course, that the MIB didn't get changed somehow 
when moving to standards-track.  If experimental MIBs were often good 
enough to standardize as-is, I wouldn't really see the point in 
changing prefixes.   I don't really see why it matters which OID prefix 
is used for any particular MIB, whether it's standards track or not.  
But some people seemed to care.

I'm a lot more concerned about email message headers that are 
human-readable, because I suspect a lot more people guess semantics 
from looking at message header field names than guess semantics by 
looking at MIB OID prefixes.

> Additionally, nobody runs around, management station in hand, and 
> tries to
> access random agents on the network.

You mean there aren't any management stations doing autodiscovery?  
It's been awhile since I looked at any SNMP work but it seems like such 
features were all the rage a few years back.

> Finally, anyone who thinks SNMP isn't bothered by interop problems of 
> various
> sorts hasn't spent much time using it.

I would never claim that :)  I was just wondering whether experience 
with SMTP moving from experimental to standards-track OID prefixes 
would give some additional clue about the likely effect of moving from 
X- to non-X- in email.

Date: Thu, 22 Jan 2004 18:53:02 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Keith Moore <[email-masked]>
Cc: Dave Crocker <[email-masked]>, Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> > My own, primary concern about them is having an "experimental" header
> > become a defacto standard and then having to find a way to get folks to
> > use a new, non-X header name for it.  I believe that hasn't been
> > successful yet, or at least not much.

> why does it work better in the case of Experimental vs Standard MIBs?
> (or do you think that doesn't work well either?)

First, I don't know that it works better. I suspect it does not, and that once
an experimental MIB is in use it is very difficult to get rid of it. Indeed, if
such transitions worked well, why isn't the accepted practice for MIBs
developed in the IETF to assign an experimental OID during development and
switch to a standard one after IESG approval? It sure isn't done this way -
what happpens in practice is that there is NO OID assigned in the document
until after it is approved.

But even if this approach worked well for MIBs, the cases aren't even remotely
comparable. The difference between a MIB registered in the experimental space
and a MIB registered in the standards space is simply the OID prefix. That's
almost certainly something you can configure on a management station and in
some cases on the agent side as well. I don't believe I've ever seen a mail
system component with comparable flexibility in regards to header fields.

Additionally, nobody runs around, management station in hand, and tries to
access random agents on the network. (Well, perhaps people actually do do this,
but I'll bet that when they do they are crackers and are more interested in
exploiting one of the many ASN.1 parser buffer overrun bugs than in performing
any sort of useful management function.) Agents and managements stations tend
to be operated by people in the same administrative domain and a given agent
is likely to only be probed by a small number of management stations. (I'm
confident the most common number is 0, then 1, with numbers greater than 1 a
very distant third.) This means that OID mismatches present a trivial
problem compared to header field mismatches.

Finally, anyone who thinks SNMP isn't bothered by interop problems of various
sorts hasn't spent much time using it. As it happens I have (a) Developed
standards-track MIBs, (b) Written both agent code and agent plugin code, (c)
Written management station plugins, and (d) Use a fairly sophisticated SNMP
monitoring setup to keep tabs on my home network. And I've run into plenty of
problems along the way, including but not limited to: Lack of specificity in
widely used standard MIBs causing interop problems, experimental MIBs
constructed in extremely bizarre ways that are very difficult to build probes
for (if anyone knows how to construct a probe that's able to see Mac OS X
systems connected to an original series Apple Airport please let me know),
plugin architectures that are inherently incompatible with the design of some
MIBs, and fun and games with row creation. And then there's Agent X. You don't
want to get me started on that...

				Ned

P.S. An interesting test will be to see if once CAPWAP gets going whether
it can produce standard MIBs to replace the experimental MIBs in use
in existing wireless products. I am not optimistic.

Cc: Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: X-* header fields
Date: Thu, 22 Jan 2004 21:48:19 -0500
To: Dave Crocker <[email-masked]>
Sender: [email-masked]

> My own, primary concern about them is having an "experimental" header
> become a defacto standard and then having to find a way to get folks to
> use a new, non-X header name for it.  I believe that hasn't been
> successful yet, or at least not much.

why does it work better in the case of Experimental vs Standard MIBs?
(or do you think that doesn't work well either?)

From: "Al Costanzo" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: Re: X-* header fields
Date: Thu, 22 Jan 2004 20:21:23 -0500
Sender: [email-masked]

A clean, simple, fast way to register the non X- header field would do the
trick..

Al

>
>
> > >> Misuse of any protocol feature can cause interoperability problems.
>
> > ni8mc> But it is far from clear that the cases where X- field use is
causing interop
> > ni8mc> problems actually constitute "misuse".
>
>
> > In fact I would say that this is an example of using the "protocol"
exactly the way it
> > was intended, with an undesired, but entirely predictable, effect.
>
> > Alas, sometimes excellent intentions and even good designing/thinking
> > can produce very bad results.  We simply have to acknowledge the outcome
> > and find a way to fix it.
>
> > When we added X-headers, I thought it was a very clever idea, indeed.
> > Instead it has proved both problematic and aesthetically ugly.  (Since
> > email headers are user-visible, aesthetics matter, at least somewhat.)
>
> > My own, primary concern about them is having an "experimental" header
> > become a defacto standard and then having to find a way to get folks to
> > use a new, non-X header name for it.  I believe that hasn't been
> > successful yet, or at least not much.
>
> This is my major concern with them as well.
>
> Ned
>
>

Date: Thu, 22 Jan 2004 17:08:31 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Dave Crocker <[email-masked]>
Cc: [email-masked], Keith Moore <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> >> Misuse of any protocol feature can cause interoperability problems.

> ni8mc> But it is far from clear that the cases where X- field use is causing interop
> ni8mc> problems actually constitute "misuse".

> In fact I would say that this is an example of using the "protocol" exactly the way it
> was intended, with an undesired, but entirely predictable, effect.

> Alas, sometimes excellent intentions and even good designing/thinking
> can produce very bad results.  We simply have to acknowledge the outcome
> and find a way to fix it.

> When we added X-headers, I thought it was a very clever idea, indeed.
> Instead it has proved both problematic and aesthetically ugly.  (Since
> email headers are user-visible, aesthetics matter, at least somewhat.)

> My own, primary concern about them is having an "experimental" header
> become a defacto standard and then having to find a way to get folks to
> use a new, non-X header name for it.  I believe that hasn't been
> successful yet, or at least not much.

This is my major concern with them as well.

				Ned

Date: Thu, 22 Jan 2004 16:43:39 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: [email-masked]
CC: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

>> Misuse of any protocol feature can cause interoperability problems.

ni8mc> But it is far from clear that the cases where X- field use is causing interop
ni8mc> problems actually constitute "misuse".

In fact I would say that this is an example of using the "protocol" exactly the way it
was intended, with an undesired, but entirely predictable, effect.

Alas, sometimes excellent intentions and even good designing/thinking
can produce very bad results.  We simply have to acknowledge the outcome
and find a way to fix it.

When we added X-headers, I thought it was a very clever idea, indeed.
Instead it has proved both problematic and aesthetically ugly.  (Since
email headers are user-visible, aesthetics matter, at least somewhat.)

My own, primary concern about them is having an "experimental" header
become a defacto standard and then having to find a way to get folks to
use a new, non-X header name for it.  I believe that hasn't been
successful yet, or at least not much.

d/

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

Date: Thu, 22 Jan 2004 10:03:24 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Keith Moore <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> > > I don't quite understand why the X- headers or other similar X-
> > > stuff must be successful.
> >
> > Think for a moment about what you're saying. You're saying that it
> > doesn't matter if a particular feature of a specification has caused
> > interoperability problems in other contexts and is known to have
> > caused severe interoperability problems in the present context.

> Misuse of any protocol feature can cause interoperability problems.
> That is at least as true for X- fields (which IMHO should never have
> been put in shipping products except perhaps as write-only fields) as
> for non X- fields (which should never be in shipping products without
> public vetting).

But it is far from clear that the cases where X- field use is causing interop
problems actually constitute "misuse". For example, there have been cases where
two parties opted to use the same X- mail header field for their own private,
but sematically different, purposes. (X-Sender is the example that comes to
mind.) The specification of X- in RFC 822 doesn't give you much leverage to
argue that either of them were "misusing" the field.

Another, similar, case is the X-uuencode CTE. (And before you say that the
right answer is to use base64 and not uuencode, the choice in many cases isn't
between base64 or uuencode, but between embedding uuencode material in text and
calling it out in the MIME structure. In such cases I'll take the latter every
time.)

> The burden for those who want X- fields to be treated just as any other
> field is to convincingly argue that there would be greater overall
> interoperability by doing that.  Making it easier for people to follow
> the rules doesn't help if those rules don't encourage interoperability.

I see it the other way around. Given that interop problems are known
to exist the burden should instead be on the proponents of X- to
make a case for the continued inclusion of the feature being worth it.

Less is usually more as far as standards go.

> > I'm sorry, but the entire point of this exercise is to produce
> > interoperable standards. The question was whether or not X- fields
> > have succeeded in not causing interoperability issues in other
> > contexts. And the answer is that in the cases where they have seen
> > active use they have NOT succeeded in this regard.

> A better question is whether treating X- fields differently (in that
> they cannot be registered) has improved the overall level of
> interoperability over what it would have been to not have them treated
> differently.  I don't think we can answer this with a great deal of
> confidence, since we don't have a control.

True enough. However, the question that started this thread was how well the
use of X- fields have worked in other contexts. And although it is far from
clear the results elsewhere are relevant, the fact is that such usage has not
turned out well.

> At one time it was widely
> believed that providing a special, reserved space of protocol fields
> would discourage pollution of the normal space.   Personally I'm glad
> that we encouraged people to use X- because I think that if those fields
> didn't use X- there would be a lot more bogus and poorly-defined
> fields out there and it would be more difficult to distinguish those
> fields from fields that should be implemented than it is now.  Which is
> not to say that there aren't bogus non-X- fields out there, just that
> X- helps identify _some_ of the bogus fields.

First, since there are huge numbers of bogus header fields (I have 587 on my
personal list of such things, about evenly split in their use or non-use of X-)
both with and without X-, I don't buy the argument that X- has helped in any
material way.

Second, the existance of X- has led to the use of well-defined fields in the X-
space that now cannot be standardized because of the X- rule. The best example
of this I know of is X-Accept-Language but I'm sure there are others.

I've also seen cases of fields where the contents are reasonably well defined
but the field names vary, to the detriment of interoperabilty. For example, it
is often useful for different gateways to the same type of non-Internet mail
system to cooperate in their specification of a fields used to carry
information useful to the non-Internet system they gateway to. (X-VMSMail-To:
is one example of this, there are many others.) Although there tend to be
scoping issues associated with the use of such information, the fact remains
that it would be very useful for vendors of different gateways to the same
system to be able to agree on a single field name. But the use of X- fields for
this sort of stuff (which does seem like an obvious choice) has precluded
specification of such field names in these cases.

> Most of the X- fields that have been deployed shouldn't have been
> deployed in shipping product with or without the X-.

Agreed, but this doesn't make the cases where something was improperly
deployed with X- go away.

> Furthermore,
> there has never been a prohibition on defining new fields without X-.

Which further dilutes the need for X- IMO.

> So I have a hard time understanding how writing new RFCs that change the
> rules for X- helps interoperability, when the real issue is the failure
> to define the fields in a way that allows them to interoperate.   More
> generally, when the problem is due to the failure to read existing RFCs,
> I fail to see how writing new RFCs is going to help.

Sure this is a problem but this isn't the problem getting rid of X- is intended
to solve.

> (Granted, it's easier to get people to follow rules that say "anything
> goes", because they don't have to know the rules to follow them.  But
> while "anything goes" might result in more widespread compliance, it
> doesn't necessarily result in greater interoperability.)

This is why we're trying the provisional registry approach.

> > > Aren't these simply meant for 'private', experimental
> > > and test use? If I want to test out a new idea, if I want _my_
> > > mailing system A and _my_ mailing system B and _my friends_ mailing
> > > system C to do something special, if I want to flag certain messages
> > > for special handling in _my_ environment, then I'll use X- headers.
> > > All this may be highly successful for me, but not for the world at
> > > large.

> > The problem is that often as not such experiments do not stay
> > confined, and instead deploy more widely.

> But at least they're labelled as experimental.  (or more precisely,
> they're labelled as "user-defined" fields that cannot be registered.)

But there's often no truth in that label. This is why having a provisional
registry is a better answer.

> Even though I believe X- fields should not be used in shipping products,
> labelling an unvetted field with X- seems to be at least slightly more
> responsible than not labelling it in that way.

I'm sorry, but this simply isn't a realistic approach for vendors to take.

> > You're
> > expecting people to play by the rules when the rules require them to
> > behave in seriously inconvenient ways. We have ample evidence this is
> > not how things play out. The trick is to write rules that people will
> > actually follow.

> Indeed.  It seems that there's no getting around these two things:

> 1. You don't tend to get widespread interoperability without careful
> design that takes into account a wide range of input (because conditions
> vary widely and no single person or group appreciates the breadth of
> conditions).

> 2. Getting that wide range of input (typically by public review and
> vetting) is "seriously inconvenient" - at least, in comparison to
> how much effort it takes to type

>     printf ("field-name: ...\r\n");

> Frankly I don't think we're going to improve interoperability without
> somehow encouraging more people to do things that are inherently
> "seriously inconvenient".  Maybe we can make them more convenient, but
> it will never be anywhere nearly as easy as typing printf.

This is precisely the problem the registry proposal is trying to address. It
also tries to address the related problem that it is far better to know what
the intent of a given field usage is, even when that intent is flawed in some
way, than it is to be forced to guess. (Again I will point out that simply
ignoring these things is just not an option for vendors -- if they adopt this
attitude their customers will simply go elsewhere for their messaging
products.)

> I really think that the X- debate is a red herring.  The reason we
> seem to always be at an impasse on the X- debate is that we're
> dancing around the real problem, which really has not much at all
> to do with X- .  The problem, I suspect, is something like this:

It certainly is red herring in regards to the topic you're now pursuing.

> In the marketplace, vendors distinguish between each other on features,
> and some features require protocol extensions.   Standardizing those
> protocol extensions before deployment defeats the vendor's purpose for
> deploying those extensions, which was to gain some (however limited in
> time) edge over a competitor.  So it's not going to happen very often.
> But when such features are deployed they may not interoperate well with
> other vendors' products because of inadequate design and/or lack of a
> well-written published specification, and they may actually degrade
> interoperability of the email service in general.

This is probably true in some cases. However, I can also think of many cases
where a vendor would have had no problem with specification of a field prior to
or coincident with deployment, but was thwarted by the daunting process of
publishing an RFC. (There have even been cases where drafts were published but
then languished because the necessary energy for pushing the document through
the process just wasn't there.) Heck, I've done this myself on several
occasions, and if I opted for this approach, what does this say about someone
who knows nothing about IETF processes?

> So the question for those who want to promote interoperability is:

> How do we encourage experimentation with, and deployment of, new
> features in email without degrading interoperability?

The best tack we've been able to come with is what's in the registry
specification. Of course it remains to be seen if it will actually help.
But I agree that the X- issue is small beer in this larger context.

				Ned

Date: Thu, 22 Jan 2004 12:46:16 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

> > I don't quite understand why the X- headers or other similar X-
> > stuff must be successful.
> 
> Think for a moment about what you're saying. You're saying that it
> doesn't matter if a particular feature of a specification has caused
> interoperability problems in other contexts and is known to have
> caused severe interoperability problems in the present context.

Misuse of any protocol feature can cause interoperability problems. 
That is at least as true for X- fields (which IMHO should never have
been put in shipping products except perhaps as write-only fields) as
for non X- fields (which should never be in shipping products without
public vetting).

The burden for those who want X- fields to be treated just as any other
field is to convincingly argue that there would be greater overall
interoperability by doing that.  Making it easier for people to follow
the rules doesn't help if those rules don't encourage interoperability.

> I'm sorry, but the entire point of this exercise is to produce
> interoperable standards. The question was whether or not X- fields
> have succeeded in not causing interoperability issues in other
> contexts. And the answer is that in the cases where they have seen
> active use they have NOT succeeded in this regard.

A better question is whether treating X- fields differently (in that
they cannot be registered) has improved the overall level of
interoperability over what it would have been to not have them treated
differently.  I don't think we can answer this with a great deal of
confidence, since we don't have a control.  At one time it was widely
believed that providing a special, reserved space of protocol fields
would discourage pollution of the normal space.   Personally I'm glad
that we encouraged people to use X- because I think that if those fields
didn't use X- there would be a lot more bogus and poorly-defined 
fields out there and it would be more difficult to distinguish those
fields from fields that should be implemented than it is now.  Which is
not to say that there aren't bogus non-X- fields out there, just that
X- helps identify _some_ of the bogus fields.

Most of the X- fields that have been deployed shouldn't have been
deployed in shipping product with or without the X-.  Furthermore,
there has never been a prohibition on defining new fields without X-.
So I have a hard time understanding how writing new RFCs that change the
rules for X- helps interoperability, when the real issue is the failure
to define the fields in a way that allows them to interoperate.   More
generally, when the problem is due to the failure to read existing RFCs,
I fail to see how writing new RFCs is going to help.  

(Granted, it's easier to get people to follow rules that say "anything
goes", because they don't have to know the rules to follow them.  But
while "anything goes" might result in more widespread compliance, it
doesn't necessarily result in greater interoperability.)

> > Aren't these simply meant for 'private', experimental
> > and test use? If I want to test out a new idea, if I want _my_
> > mailing system A and _my_ mailing system B and _my friends_ mailing
> > system C to do something special, if I want to flag certain messages
> > for special handling in _my_ environment, then I'll use X- headers.
> > All this may be highly successful for me, but not for the world at
> > large.
> 
> The problem is that often as not such experiments do not stay
> confined, and instead deploy more widely.

But at least they're labelled as experimental.  (or more precisely,
they're labelled as "user-defined" fields that cannot be registered.)

Even though I believe X- fields should not be used in shipping products,
labelling an unvetted field with X- seems to be at least slightly more
responsible than not labelling it in that way.

> > If some header-based feature or interoperation should work on a
> > large scale, not just in the environment I control, then I will need
> > to define and describe a non-X- header for it.
> 
> Which doesn't happen in practice.

And it still won't happen if the treatment of X- fields is changed.

> > The X- prefix only gives me
> > assurance that my experimental or internal stuff is not clashing
> > with any currently defined, widely used header, now or in the
> > future.  The X- header will not even guarantee that it will work
> > through externally controlled gateways, even though most of the time
> > it will.
> 
> No such guarantee is present in practice, since any number of
> X- fields are now in wide use.

It is certainly true that some X- fields are in wide use.   Or more
generally, some widely deployed implementations of mail tools are paying
attention to unregistered fields, some of which begin with X-.

> > Anybody who uses X- headers or tags and expects large scale
> > interoperation is simply naive. I can easily make a rule that in my
> > huge private garden cars are to drive backwards. If I want them to
> > do the same on public roads, well, good luck.
> 
> On the contrary, the person who is being naive here is you. 

No, he's being realistic, and more folks need to recognize this.  

More specifically, anyone who expects fields that aren't well defined
and publically vetted to interoperate on a wide scale is being naive.
It might happen by accident, but not in general.
 
Use of X- fields is just a special case of the above. 

> You're
> expecting people to play by the rules when the rules require them to
> behave in seriously inconvenient ways. We have ample evidence this is
> not how things play out. The trick is to write rules that people will
> actually follow.

Indeed.  It seems that there's no getting around these two things:

1. You don't tend to get widespread interoperability without careful 
design that takes into account a wide range of input (because conditions
vary widely and no single person or group appreciates the breadth of
conditions).  

2. Getting that wide range of input (typically by public review and
vetting) is "seriously inconvenient" - at least, in comparison to
how much effort it takes to type 

    printf ("field-name: ...\r\n");

Frankly I don't think we're going to improve interoperability without
somehow encouraging more people to do things that are inherently
"seriously inconvenient".  Maybe we can make them more convenient, but 
it will never be anywhere nearly as easy as typing printf.

> > I like the comparison between X- and 192.168.0.x. I can create a
> > huge internal network with 192.168.0.x IP addresses, I can even
> > tunnel the network over the public internet. This can all work
> > perfectly well. But I can not expect it to work outside of my
> > private network, neither is it accessible from the internet as is,
> > nor is it guaranteed that it can work with another private network
> > using the same address range.
> 
> First, its not a valid comparison since such addresses are unroutable
> and hence are unusable on a wide scale. 

A better analogy would be to the new kinds of addressing being proposed
for IPv6, which _are_ globally unique (at least with high probability)
_aren't_ publically routable (since they aren't aggregatable)
but _can_ still be distinguished from publically routable addresses.
(I call them GUPIs - globally-unique provider-independent addresses,
but this name hasn't caught on.)  One of the arguments that have been
made in favor of deprecating site-locals in favor of GUPIs is that,
while both kinds of addresses would leak, the GUPIs would at least be
recognizable as "addresses that don't belong here" (say in routing
advertisements) and thus you can reliably distinguish them from
legitimate traffic and (at least for registered GUPI prefixes)
you might also be able to tell where the traffic is coming from.

But this is where the analogy falls apart, because for the case of 
non-local GUPIs we might want to filter them, for the case of unknown
extension headers we might want to implement that feature rather than
ignore it.

--

I really think that the X- debate is a red herring.  The reason we
seem to always be at an impasse on the X- debate is that we're 
dancing around the real problem, which really has not much at all
to do with X- .  The problem, I suspect, is something like this:

In the marketplace, vendors distinguish between each other on features,
and some features require protocol extensions.   Standardizing those
protocol extensions before deployment defeats the vendor's purpose for
deploying those extensions, which was to gain some (however limited in
time) edge over a competitor.  So it's not going to happen very often.
But when such features are deployed they may not interoperate well with
other vendors' products because of inadequate design and/or lack of a
well-written published specification, and they may actually degrade
interoperability of the email service in general.  

So the question for those who want to promote interoperability is:

How do we encourage experimentation with, and deployment of, new
features in email without degrading interoperability?

Date: Wed, 21 Jan 2004 18:41:13 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: [email-masked]
Cc: [email-masked]
Sender: [email-masked]

> On Sun, Jan 18, 2004 at 08:49:42AM -0800, [email-masked] wrote:

> > > Bruce Lilly wrote:
> >
> > > > X- is used as an indicator for experimental or private-use tags in
> > > > many places
> > > > in Internet message protocols other than field names. For example:
> > > [...]
> >  [...]
> > The bottom line is that the success of x- fields in other places has anywhere
> > from disasterous at worst to mixed at best. And to a large extent the lack of
> > harm in many spaces has come from lack of use. So if you're looking for support
> > for keeping the X- distinction from the successful use of X- in other contexts,
> > I'm afraid it is not forthcoming.

> I don't quite understand why the X- headers or other similar X- stuff
> must be successful.

Think for a moment about what you're saying. You're saying that it doesn't
matter if a particular feature of a specification has caused interoperability
problems in other contexts and is known to have caused severe interoperability
problems in the present context.

I'm sorry, but the entire point of this exercise is to produce interoperable
standards. The question was whether or not X- fields have succeeded in
not causing interoperability issues in other contexts. And the answer
is that in the cases where they have seen active use they have NOT
succeeded in this regard.

> Aren't these simply meant for 'private', experimental
> and test use? If I want to test out a new idea, if I want _my_ mailing
> system A and _my_ mailing system B and _my friends_ mailing system C to
> do something special, if I want to flag certain messages for special
> handling in _my_ environment, then I'll use X- headers. All this may be
> highly successful for me, but not for the world at large.

The problem is that often as not such experiments do not stay confined,
and instead deploy more widely.

> If some header-based feature or interoperation should work on a large
> scale, not just in the environment I control, then I will need to define
> and describe a non-X- header for it.

Which doesn't happen in practice.

> The X- prefix only gives me
> assurance that my experimental or internal stuff is not clashing with
> any currently defined, widely used header, now or in the future.  The X-
> header will not even guarantee that it will work through externally
> controlled gateways, even though most of the time it will.

No such guarantee is present in practice, since any number of
X- fields are now in wide use.

> Anybody who uses X- headers or tags and expects large scale
> interoperation is simply naive. I can easily make a rule that in my huge
> private garden cars are to drive backwards. If I want them to do the
> same on public roads, well, good luck.

On the contrary, the person who is being naive here is you. You're expecting
people to play by the rules when the rules require them to behave in seriously
inconvenient ways. We have ample evidence this is not how things play out. The
trick is to write rules that people will actually follow.

> I like the comparison between X- and 192.168.0.x. I can create a huge
> internal network with 192.168.0.x IP addresses, I can even tunnel the
> network over the public internet. This can all work perfectly well. But
> I can not expect it to work outside of my private network, neither is it
> accessible from the internet as is, nor is it guaranteed that it can
> work with another private network using the same address range.

First, its not a valid comparison since such addresses are unroutable and hence
are unusable on a wide scale. No such restrictions apply to X- fields; message
transfer agents do not drop them.

Second, the fact of the matter is that private use addresses of this sort
routinely leak outside of local networks and cause all sorts of problems when
they do. In particular, the number of UDP packets that arrrive at various
destinctions on the Internet with these sorts of IP addresses in the sender
field is enormous, as is the amount of bandwidth that they waste.

				Ned

Date: Thu, 22 Jan 2004 00:38:47 +0100
From: [email-masked]
To: [email-masked]
Subject: Re: X-* header fields
Mail-Followup-To: [email-masked], [email-masked]
Sender: [email-masked]

On Sun, Jan 18, 2004 at 08:49:42AM -0800, [email-masked] wrote:

> > Bruce Lilly wrote:
> 
> > > X- is used as an indicator for experimental or private-use tags in
> > > many places
> > > in Internet message protocols other than field names. For example:
> > [...]
>  [...]
> The bottom line is that the success of x- fields in other places has anywhere
> from disasterous at worst to mixed at best. And to a large extent the lack of
> harm in many spaces has come from lack of use. So if you're looking for support
> for keeping the X- distinction from the successful use of X- in other contexts,
> I'm afraid it is not forthcoming.

I don't quite understand why the X- headers or other similar X- stuff
must be successful. Aren't these simply meant for 'private', experimental
and test use? If I want to test out a new idea, if I want _my_ mailing
system A and _my_ mailing system B and _my friends_ mailing system C to
do something special, if I want to flag certain messages for special
handling in _my_ environment, then I'll use X- headers. All this may be
highly successful for me, but not for the world at large.

If some header-based feature or interoperation should work on a large
scale, not just in the environment I control, then I will need to define
and describe a non-X- header for it. The X- prefix only gives me
assurance that my experimental or internal stuff is not clashing with
any currently defined, widely used header, now or in the future.  The X-
header will not even guarantee that it will work through externally
controlled gateways, even though most of the time it will.

Anybody who uses X- headers or tags and expects large scale
interoperation is simply naive. I can easily make a rule that in my huge
private garden cars are to drive backwards. If I want them to do the
same on public roads, well, good luck.

I like the comparison between X- and 192.168.0.x. I can create a huge
internal network with 192.168.0.x IP addresses, I can even tunnel the
network over the public internet. This can all work perfectly well. But
I can not expect it to work outside of my private network, neither is it
accessible from the internet as is, nor is it guaranteed that it can
work with another private network using the same address range.

I thought RFC822 was quite clear about this:

4.7.4
  [...]
  To  provide
  **user-defined**
  fields  with  a  measure  of  safety,  in name selection, such
  **extension-fields will never**
  have names  that
  **begin  with  the string "X-".**
  [...]

4.7.5
  [...]
  Note:  The prefatory string "X-" will never  be  used  in  the
         names  of Extension-fields.  This provides user-defined
         fields with a protected set of names.

Kurt

Date: Wed, 21 Jan 2004 17:43:24 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: James M Galvin <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: cryptographically verifiable fields
Sender: [email-masked]

James M Galvin wrote:

>So, here's a suggestion for how to use encapsulation in a backwards
>compatible way.  Actually, it is transitionally backwards compatible
>since it presumes the ultimate goal is to "upgrade" to using
>encapsulation.
>
>Presumably the "backward" sites would not honor a Content-Type: header
>if it was present in a control message.  So, use the multipart/signed
>solution but duplicate the text in the signed object in the preamble of
>the body, i.e., before the first boundary marker.
>
>"Backward" sites get what they expect and "forward" sites use the
>enhanced features.
>  
>
In some cases that might work, however the "backward" software might not
stop
processing at the multipart boundary delimiter.

>    There are a number of other issues that would have to be addressed
>    in the Usenet examples:
>
>    1. Authorization; a mechanism for distributing and updating
>       authorization information (who is authorized to approve a
>       moderated article, or to issue control messages) is needed.
>
>Is this more complicated than adding a new protocol element to a control
>message, i.e., a new type control message?
>  
>
I think that becomes a "chicken-and-egg" problem: if a control message
provides authorization
information, why should I trust that control message -- is it
authorized?  Some root source(s) of
authorization information would have to be established by some other
mechanism.  Probably
not a big problem.

>    2. A standardized open mechanism for obtaining information necessary
>       to complete authentication.  Aside from cost issues, some people
>       are leery of dealing with CAs, especially after last year's
>       Verisign DNS shenanigans.
>
>So you're in need of a "PKI", essentially.  Well, if we had DNSSEC you
>would have one, ideally suited for this application since given an email
>address you've got all the information you need for a lookup and the
>machinery to do it.
>  
>
Yes.  Hmm...

Date: Wed, 21 Jan 2004 17:14:06 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: =?ISO-8859-1?Q?Claus_F=E4rber?= <[email-masked]>
CC: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

Claus Färber wrote:

>Bruce Lilly <[email-masked]> schrieb/wrote:
>  
>
>>Why treat 4-digit years with leading zeros specially?  I.e. that
>>allows: 0001
>>    
>>
>
>0001 is different from 01 (which means ??01).
>
>Claus
>  
>
True, but irrelevant (01 isn't a 4-digit number).

Date: Wed, 21 Jan 2004 14:41:39 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: cryptographically verifiable fields
To: ietf-822 <[email-masked]>
Sender: [email-masked]

On Mon, 19 Jan 2004, Bruce Lilly wrote:

    >This is where I think multipart/signed got it right and is a big win (he
    >says with all humility :-).  There are only two real requirements.
    >
    >    the object to be signed must be canonicalized
    >
    >    it must be universally representable
    >
    >
    [...]

    >Now, if you're looking to add a signature (or some other cryptographic
    >trace value) to the outermost headers and have it apply over the entire
    >message (including the outermost headers), then you've got a really hard
    >problem.
    >
    >Actually, I don't think "really hard problem" begins to describe just
    >how hard it would be.  I can not imagine why anyone would want to do
    >such a thing.  It just isn't practical in today's Internet email
    >environment.
    >
    >S/MIME and PGP/MIME inherit this to the extent they use
    >multipart/signed.

    One problem with encapsulation is that the signed information *is*
    encapsulated.  That's fine for an end-to-end application, but
    doesn't work well for a message that needs to be validated and
    interpreted at multiple points.  Examples of the latter would be a
    Usenet control message, or a Usenet article posted to a moderated
    group.  In each case, some of the message header fields and the body
    would be signed, while fields which are expected to change in
    transit (trace fields) would not be signed.  Avoiding encapsulation
    permits backwards compatibility; existing software can handle the
    message with the same lack of authentication that is currently the
    case, while newer software can verify authenticity.  With
    encapsulation, everything has to be rewritten to look inside the
    wrapper.

So, here's a suggestion for how to use encapsulation in a backwards
compatible way.  Actually, it is transitionally backwards compatible
since it presumes the ultimate goal is to "upgrade" to using
encapsulation.

Presumably the "backward" sites would not honor a Content-Type: header
if it was present in a control message.  So, use the multipart/signed
solution but duplicate the text in the signed object in the preamble of
the body, i.e., before the first boundary marker.

"Backward" sites get what they expect and "forward" sites use the
enhanced features.

    I think Keith's outline of a solution is viable.

    There are a number of other issues that would have to be addressed
    in the Usenet examples:

    1. Authorization; a mechanism for distributing and updating
       authorization information (who is authorized to approve a
       moderated article, or to issue control messages) is needed.

Is this more complicated than adding a new protocol element to a control
message, i.e., a new type control message?

    2. A standardized open mechanism for obtaining information necessary
       to complete authentication.  Aside from cost issues, some people
       are leery of dealing with CAs, especially after last year's
       Verisign DNS shenanigans.

So you're in need of a "PKI", essentially.  Well, if we had DNSSEC you
would have one, ideally suited for this application since given an email
address you've got all the information you need for a lookup and the
machinery to do it.

Key management and distribution is never easy.  Maybe we can "borrow"
something from other protocols that have had to deal with this
problem, or move forward with what you proposed as a basis.

Jim

Date: Wed, 21 Jan 2004 14:26:36 -0500
From: Keith Moore <[email-masked]>
To: James M Galvin <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: making mail traceable
Sender: [email-masked]

>     > If it's end-to-end, aka MUA <> sMTA <> rMTA <> MUA signatures
>     > are nice to have, but as I can trust my rMTA's Received: line
>     > about the sending MTA it is not of much additional information.
> 
>     it's a moot point.  it's much easier to make e2e sigs work than to
>     make hop-by-hop sigs work.
> 
> I don't think it's entirely moot.  In the context of traceability it
> is another tool worth exploring.

I don't mind if we find a way for MTAs to sign received fields (or
provide similar functionality in another way).  I just hope we can avoid
needing to do this in order to provide sender non-repudiation and
tracing of messagesto their source.

>     > I don't think it is an easy task to find information to add to
>     > the hash.
> 
>     offhand:
>     - subject field (perhaps truncated to XX bytes)
>     - message body
>     - source IP address and port
>     - precise date/time (not the Date header field)
>     - *maybe* some form of the envelope recipient list
> 
> I would include both the Date: in the message and the local Date.  I
> would also add the Message-ID:.

Okay, I can almost make a case for the Date field being a part of the 
content (IFF it's generated by the sender's MUA at the time of message
composition).  I have a harder time making the case for including 
message-id, because the content is (or isn't) spam independent of what
the message-id is, and message-ids do sometimes get changed in transit.
basically I'd like to include as little as we can get away with while 
still identifying the content.
 
>     > Adding the body of the email to the hash is also playing
>     > vabanque as e.g. mailing lists add trailers to the message and
>     > break the hash.
> 
>     interesting point that.  but if the body is the last thing to be
>     hashed you may be able to recheck the hash at every line boundary.
>     you might also have the id field include the number of bytes from
>     the body that are hashed.
> 
> All of this is why the object to be hashed needs to be pushed down a
> level and treated opaquely by intermediate MTAs.

That would be nice if we could do it, and if I thought we could do it in
a reasonable amount of time, I'd agree with you.  But I think it would
be easier to replace the entire mail system with something different
than to change the behavior of so many existing tools that don't treat
an embedded message the same as they treat one that isn't embedded. 
Which is another way of saying that even though I think it's a desirable
end-state, this particular approach would take a decade or more to deploy.
And I don't think we can afford to wait that long.

Date: Wed, 21 Jan 2004 14:09:59 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: Markus Stumpf <[email-masked]>, ietf-822 <[email-masked]>
Sender: [email-masked]

On Mon, 19 Jan 2004, Keith Moore wrote:

    > If it's end-to-end, aka MUA <> sMTA <> rMTA <> MUA signatures are nice
    > to have, but as I can trust my rMTA's Received: line about the sending
    > MTA
    > it is not of much additional information.

    it's a moot point.  it's much easier to make e2e sigs work than to make
    hop-by-hop sigs work.

I don't think it's entirely moot.  In the context of traceability it is
another tool worth exploring.

There is significant value in having each MTA add (conceptually as part
of a Received: header) a signature indicating it handled the message.
It does not have to be cumulative (although I would probably include the
signature of the prior hop in my signature calculation) nor does it have
to assert anything other than I handled the message and agree to be
accountable for having relayed it.

For example, this could be really useful for clients who trust their
delivering MTA.  More generally it provides the basis for sites to
create reliable whitelists.

    > I don't think it is an easy task to find information to add to the
    > hash.

    offhand:
    - subject field (perhaps truncated to XX bytes)
    - message body
    - source IP address and port
    - precise date/time (not the Date header field)
    - *maybe* some form of the envelope recipient list

I would include both the Date: in the message and the local Date.  I
would also add the Message-ID:.

    > Adding the body of the email to the hash is also playing vabanque as
    > e.g. mailing lists add trailers to the message and break the hash.

    interesting point that.  but if the body is the last thing to be
    hashed you may be able to recheck the hash at every line boundary.
    you might also have the id field include the number of bytes from
    the body that are hashed.

All of this is why the object to be hashed needs to be pushed down a
level and treated opaquely by intermediate MTAs.

In the case of the adding of a trailer by mailing lists, the
"correction" is to have the technology add a body part with the trailer
information as opposed to modifying the body part.

Jim

Date: Wed, 21 Jan 2004 13:12:53 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Nathaniel Borenstein <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On Mon, 19 Jan 2004, Nathaniel Borenstein wrote:

    Preserving anonymous email is very important to me, and a battle worth
    fighting very passionately in the long run.  The architecture of any
    anti-spam tracing facilities will be critical to that battle.

Just to be clear, when you use "anonymous" you truly mean anonymous in
the sense of the "remailers" that we used to hear so much about.

And further, anonymity in a context that is in conflict with
traceability.

Would you permit "site traceability" as compared to "author
traceability" or is that too in conflict with anonymity?

Jim

To: [email-masked]
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
Subject: Re: Revisiting RFC 2822 grammar
Date: 21 Jan 2004 19:08:00 +0100
Lines: 10
Sender: [email-masked]

Bruce Lilly <[email-masked]> schrieb/wrote:
> Why treat 4-digit years with leading zeros specially?  I.e. that
> allows: 0001

0001 is different from 01 (which means ??01).

Claus
-- 
http://www.faerber.muc.de

Date: Wed, 21 Jan 2004 12:59:41 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: Re: making mail traceable
To: Dave Crocker <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

I agree with this characterization.  It's a great criterion with which
to judge spam fighting tools, and to my mind clearly articulates the
downside of blacklists in a positive way.  The alternative is to speak
negatively in terms of "collateral damage" but that just sets up an
argument or debate.

Thanks,

Jim

On Mon, 19 Jan 2004, Dave Crocker wrote:

    Date: Mon, 19 Jan 2004 09:34:44 -0800
    From: Dave Crocker <[email-masked]>
    To: [email-masked]
    Subject: Re: Re: making mail traceable

    Folks,

    >     do we really care where the message was sent from, as opposed to who
    >     sent it?
    >
    > My current thinking is I don't want to tightly couple the message
    > direclty to a person, even in an ephemeral way.  I would rather note
    > sites, especially those that have multiple users.
    >
    > So, rather than even having a home user site create a "who sent it
    > identifier", I would prefer the ISP submission server create a "message
    > passed through my server identifier",

    We need to distinguish between having a mechanism with semantics about the
    MTA, versus one with semantics about the author of the message.  Which
    underlying semantic do we really care about and why?

    I believe the semantic we care about, for MTAs, is that they are part of
    well-behaved networks, not that they are "authorized" by the author, or that
    they in turn vouch directly for the author.  In other words, is the hosting
    ISP running a coherent, controlled MTA environment?  If the answer is yes, we
    are not certain that other MTA sources from that ISP are rogue, but we are
    certain they are not vouched for, by the ISP.

    I believe the biggest semantic we want to see is "this author is
    well-behaved".  Anything about the MTA is indirect.  It might be useful, but
    it's not core.  A step in that direction is to authenticate the author or to
    provide an assurance that they can be located.

    Schemes that involve MTA registration for/by the author (SPF, LMAP, RMX, ...)
    confuse these two semantics and they create very Procrustean usage and
    administration scenarios. For an operation with users that send from outside
    the ISP, they create problematic usage patterns.

    In contrast, schemes that focus on the author directly are more flexible.  It
    is quite straightforward for an ISP to operate that scheme on behalf of the
    author, in those environments that permit it.  This means that direct
    author-focused schemes permit a number of operational styles, one of which is
    equivalent to  the MTA registrations such as SPF, LMAP and RMX. This includes
    having the ISP mask the actual author, while retaining accountability for
    them.

    d/
    --
    Dave Crocker <dcrocker-at-brandenburg-dot-com>
    Brandenburg InternetWorking <http://brandenburg.com>

Date: Wed, 21 Jan 2004 12:53:45 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On Mon, 19 Jan 2004, Keith Moore wrote:

    > My current thinking is I don't want to tightly couple the message
    > direclty to a person, even in an ephemeral way.

    I don't really like doing that either.  But I don't think the
    granularity of "site" is good enough to identify and marginalize
    spammers - actually experience with trying to blacklist sites (or IP
    address blocks associated with sites) IMHO indicates that it is not
    good enough for this purpose.

    Also, for lots of reasons I don't think that giving law enforcement a
    way to track down spammers is a desirable way to solve the spam
    problem.  It would get the government too involved in mediating
    people's communications, it would invite favoritism, it would require
    too many LE resources, and for that reason it would be hard to limit
    abuse.  I'd far rather find a way for the net to be self-policing by
    allowing recipients (or recipient sites) to marginalize spammers
    without actually finding out who the spammers are.  Of course, for
    serious infractions of the law, LE will still be able to trace
    originator-ids to the actual people who sent the messages - I don't see
    any way of avoiding this.

I hadn't put the pieces together quite this way before but I like it.
It makes sense to me.

We agree.  Thanks,

Jim

Date: Wed, 21 Jan 2004 12:50:37 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

Just for the record, we're in agreement.  I could quibble about some
semantics but that's not productive right now.

Thanks,

Jim

On Mon, 19 Jan 2004, Keith Moore wrote:

    Date: Mon, 19 Jan 2004 12:18:14 -0500
    From: Keith Moore <[email-masked]>
    To: James M Galvin <[email-masked]>
    Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
    Subject: Re: making mail traceable

    >>> Doesn't RFC1847 (security multiparts) already provide both of these,
    >>> or at least the framework sans the actual algorithm?
    >>
    >>     Only if everyone's user agents treated a security multipart
    >>     containing a message semantically the same as they would the
    >>     original message.
    >>
    >> Sure, so your primary concern is that the use of security multiparts
    >> would be a less backwards compatible change than a change to the
    >> Received syntax or adding a new header.
    >
    >     That, and there is a semantic difference between a signed message
    > and a
    >     cryptographically verifiable trace field in a message.
    >
    > Agreed.  My point was that the same technology would be used but would
    > be interpreted differently.

    It's going to be very similar technology, but the actual protocol used
    needs to be different.  If submission servers start applying security
    multiparts to messages it's going to break things, and it's going to be
    misinterpreted.

    >> I can imagine it might be a local policy preference for all messages
    >> to
    >> be examined by a person instead of rejecting it if the hash validation
    >> fails.  However, it should be the case that except for human review
    >> the
    >> message is rejected (for whatever we decide rejection means).  There
    >> may be some edge cases where broken mailers do really obscure things
    >> to
    >> messages to break the hash, but either you are doing security right or
    >> you don't do it.
    >
    >     Actually I don't see this as a security measure.  It doesn't really
    >     protect systems from any kind of attack, and it doesn't
    > authenticate
    >     the message as being authored or witnessed by any publicly known
    >     principal name.  It doesn't even have to provide an unimpeachable
    >     assurance of non-repudiation -- it just has to be good enough that
    > it's
    >     infeasible to make large quantities of spam look as if it were
    >     associated with originator-ids other than those controlled by the
    >     spammers.
    >
    > This seems incongruous to me.
    >
    > Some principal is going to assign a hash.  There has to be a principal
    > because the hash has to be digitally signed.  If it's not there's no
    > point having it for the purpose as described.
    >
    > This means the message is at least witnessed and there's
    > non-repudiation
    > to the extent the principal can not deny having handled the message.
    > That is a security measure.

    What I was responding to is the "either you are doing security right or
    you don't do it".  I certainly agree that this should be "done right"
    (to the extent that we can figure out what "right" is) but I think that
    calling this a "security" measure invites confusion between this
    application of security technologies (public-key cryptography,
    noninvertable hash functions) and other applications of those
    technologies, for which "doing it right" means something different than
    it does here.

    > As to whether any of the content is impeachable, that depends on when
    > the hash is actually created.

    It also depends on what is included in the hash.  I believe it will be
    necessary to omit some information from the hash in order to get the
    hash to survive most existing mail transports.  I don't think this is a
    problem as long as we don't treat the originator-id tag as a digital
    signature.

    >>> I'm sorry but I just don't see what you have in mind that is worse
    >>> than it is today,
    >>
    >>     Anyway, the reason my proposal allows originator-ids to be
    >> ephemeral is to make it hard for ordinary people to track messages -
    >> I believe anonymous speech is important - and also out of recognition
    >> that spammers can probably get ephemeral accounts anyway.
    >>
    >> My point is that we are not that [far] from this already.  To stretch
    >> the example to the extreme, the US Government could make a law that
    >> ISPs have to make sure they can map a From: email address to an
    >> actual person before accepting submission of a message (in the US of
    >> course).  Why do we need a special identifier?
    >
    >     The US Congress might or might not care, but for a variety of
    > reasons it would not be appropriate to use From in this way.  There is
    > no defined header field that can be used for this purpose without
    > conflicting with valid and existing uses of the field.  (Sender would
    > have been the right thing, but it's too widely misused now.)
    >
    > What you say is true but it doesn't refute what I said.  Does this
    > mean you agree with me?

    I still think an Originator-id field would be at least somewhat easier
    for governments to abuse in this way than any of the existing fields.
    Variant use of From is sufficiently widespread that I suspect there
    would be some pushback against the government trying to constrain its
    use.

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Wed, 21 Jan 2004 12:50:28 -0500
To: Markus Stumpf <[email-masked]>
Sender: [email-masked]

>> either way, the spam is traceable to an account that is associated 
>> with
>> you.  recipients of the message can complain to whichever ISP issued
>> the originator-id field, and that ISP will figure out pretty quickly
>> that you're a spammer, and blacklist you.
>
> This is true. But how big a loss is it to loose one account at one
> public email provider when you can have the next account anonymous in 
> 10
> seconds.

If you ask the ISP about an originator-id that is associated with a 
spammer, the ISP will tell you that this account has been used for 
spamming.

Similarly if you ask the ISP about the originator-id associated with an 
anonymous account, the ISP will tell you that the account is anonymous. 
  (or more likely, that the ISP doesn't have any reliable information 
about who owns that account).

Of course this does depend on the ISP being willing to provide a query 
service that provides information about originators of mail.  mail from 
ISPs that don't do that (or that provide false information about 
senders) will continue to be dealt with by cruder mechanisms like site 
blacklisting. The intent of this proposal is to provide a more 
fine-grained way of marginalizing spammers than to blacklist the entire 
ISP or site where the message originated.

>> the real trick is to prevent the other kind of attack - some miscreant
>> wants to discredit some vendor, so they take a single message that the
>> vendor sent legitimately and re-send it to a few million people.  we
>> need to make sure that the message is traced to the miscreant, not the
>> vendor.
>
> Currently all discuss methods where a MTA builds a hash over a
> message and sends it on. How about if the MTA incorporates into the 
> hash
> the name/address of the MTA it will deliver the email to and adds this
> hash information to the message as it is sent?

I'd like to find a better way.

It's much more difficult to build a whole chain of verifiable postmarks 
than it is to verifiably associate a message with a sender.  If the 
scheme relies on every MTA in the path being upgraded in order to work, 
it's going to be very difficult to deploy this widely enough that any 
benefit will be realized.

Originator-id is really just a stronger version of what is already 
widespread practice - many ISPs add some tag or another to a message 
before forwarding it to the net just to make it easier for themselves 
to track down reports of spam.  And I don't think it would be difficult 
for tools like spamassassin to recognize originator-ids, validate them, 
and use this in their criteria for determining whether a message was 
spam.  Both of these are relatively simple changes to functionality 
that already exists.  OTOH upgrading every single MTA would be very, 
very difficult.

Keith

Date: Wed, 21 Jan 2004 18:26:16 +0100
From: Markus Stumpf <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Subject: Re: making mail traceable
Organization: SpaceNet AG, Muenchen, Germany
Sender: [email-masked]

Had to think about that for a while ;-)

On Mon, Jan 19, 2004 at 06:31:44PM -0500, Keith Moore wrote:
> either way, the spam is traceable to an account that is associated with 
> you.  recipients of the message can complain to whichever ISP issued 
> the originator-id field, and that ISP will figure out pretty quickly 
> that you're a spammer, and blacklist you.

This is true. But how big a loss is it to loose one account at one
public email provider when you can have the next account anonymous in 10
seconds.

> the real trick is to prevent the other kind of attack - some miscreant 
> wants to discredit some vendor, so they take a single message that the 
> vendor sent legitimately and re-send it to a few million people.  we 
> need to make sure that the message is traced to the miscreant, not the 
> vendor.

Currently all discuss methods where a MTA builds a hash over a
message and sends it on. How about if the MTA incorporates into the hash
the name/address of the MTA it will deliver the email to and adds this
hash information to the message as it is sent? That way a "look ahead"
information would be added and it would be a lot harder to fake the
MTA chain.
Maybe if we introduce Sent: lines with a semantic similar to
Received: lines it would be enough to build a hash over the Sent:
line that would be added and the Received(s) line the MTA adds when
receiving the message and the Sent: line from the MTA it received
the message from to build a trustable chain?

So sending a message to you, our mailserver would build a hash using e.g.

Sent: to smtp.cs.utk.edu (160.36.56.154)
  by mail.space.net with SMTP; 21 Jan 2004 16:00:28 -0000;
Received: (qmail 85650 invoked from network); 21 Jan 2004 16:00:27 -0000
Received: from moebius2.space.net (195.30.1.100)
  by mail.space.net with SMTP; 21 Jan 2004 16:00:27 -0000
Sent: to mail.space.net (195.30.0.8)
  by moebius2.space.net with SMTP; 21 Jan 2004 16:00:27 -0000; #moebius2_hash#

and insert the hash at the end of the second line above.

	\Maex

-- 
SpaceNet AG            | Joseph-Dollinger-Bogen 14 | Fon: +49 (89) 32356-0
Research & Development |       D-80807 Muenchen    | Fax: +49 (89) 32356-299
"The security, stability and reliability of a computer system is reciprocally
 proportional to the amount of vacuity between the ears of the admin"

From: "Al Costanzo" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Subject: Re: making mail traceable
Date: Wed, 21 Jan 2004 08:35:53 -0500
Sender: [email-masked]

> Bruce Lillywrote:
>
> >Because of this LOC is seldom used. So my first point is to get vaild
data
> >into the DNS that people can 'easily' get without a GPS unit.
> >
> >
> Longitude, latitude, and altitude of a given place tend to be reasonably
> stable. Many
> maps provide that information, and there are on-line resources available
> as well.
>
> Street addresses, ..., country names change at a politician's whim.
>
And your point is?  If this ever happended you just update the records of
course.

> >Next, different laws apply to different parts of the country and the
world.
> >I am not only speaking about spammers. I am talking about credit card
fraud,
> >identity theft, etc.
> >
> [...]
>
> >We have a person that just placed a credit card order on our website. The
> >cerdit card information is all vaild and the address is correct. All is
> >verified. This of course does not mean that the card is not stolen.  An
> >addtional step would be to know where the email oringinated from.
> >
> >
> Make up your mind -- a web site or email?

I was giving you examples of an email application as well as other useful
risk assesment uses.

>
> >It will prevent someone from Pakistan (for example) saying they live in
Ohio
> >USA and using a stolen credit card from ohio. ( I saw this example
myself)
> >Have the data avaliable will help both electronic mail as well as
e-commerce
> >applications.
> >
> >
> Will it not also prevent an Ohio resident traveling in Pakistan using
> his own non-stolen
> credit card from doing e-business (perhaps ordering urgently-needed
> medical supplies,
> for example -- consider a diabetic)?

No it will not. I will give you a merchant account example in response.
Currently if you place a phone order for medicine the merchant will ask you
for your name and address and the cvv code on the card.  If you do not have
the CVV code the merchant can still process the order if he desires to but
this is his choice. He also know there is more risk that this is fraud.

Also if the postal code or address is wrong the terminal will tell the
merchant and it is up to the merchant if he wants to process the order or
void it.

If the GL location matches the state and country the risk is lower for the
transaction. It is only a way to determine risk it does not stop a
transaction. It is a tool that can be used to help prevent fraud.

The use of Geographic Location has many uses, I have only named two.

>
> >I know it will not be perfect overnight but it will help greatly, if you
> >have a better solution let me know.
> >
> In the specific case of a stolen credit card, I would recommend
> reporting the card as
> stolen to the credit card company.  That way the card will be declined
> where ever it is
> used.
>

This works after it is reported.  A person from sweden had a AMEX card with
no limit and did not know someone cloned his number and placed a 2,000.00
order on his card. If there was a way to quickly determine where the email
came from it would have prevented a series of phone calls to determine that
this happened.

In this case, the order was actually processed and it was charged back after
the fact because the card was not reported lost. It was not lost the card
was cloned.

If you or anyone has a better idea how to track the geographic location of
email please let me know but I really think this is an important issue that
needs to be addressed.

Regards,

Al

Date: Wed, 21 Jan 2004 08:06:27 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Al Costanzo <[email-masked]>
CC: [email-masked]
Subject: Re: making mail traceable
Sender: [email-masked]

Al Costanzo wrote:

>Because of this LOC is seldom used. So my first point is to get vaild data
>into the DNS that people can 'easily' get without a GPS unit.
>  
>
Longitude, latitude, and altitude of a given place tend to be reasonably 
stable. Many
maps provide that information, and there are on-line resources available 
as well.

Street addresses, ..., country names change at a politician's whim.

>Next, different laws apply to different parts of the country and the world.
>I am not only speaking about spammers. I am talking about credit card fraud,
>identity theft, etc.
>
[...]

>We have a person that just placed a credit card order on our website. The
>cerdit card information is all vaild and the address is correct. All is
>verified. This of course does not mean that the card is not stolen.  An
>addtional step would be to know where the email oringinated from.
>  
>
Make up your mind -- a web site or email?

>It will prevent someone from Pakistan (for example) saying they live in Ohio
>USA and using a stolen credit card from ohio. ( I saw this example myself)
>Have the data avaliable will help both electronic mail as well as e-commerce
>applications.
>  
>
Will it not also prevent an Ohio resident traveling in Pakistan using 
his own non-stolen
credit card from doing e-business (perhaps ordering urgently-needed 
medical supplies,
for example -- consider a diabetic)?

>I know it will not be perfect overnight but it will help greatly, if you
>have a better solution let me know.
>
In the specific case of a stolen credit card, I would recommend 
reporting the card as
stolen to the credit card company.  That way the card will be declined 
where ever it is
used.

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

Date: Wed, 21 Jan 2004 07:38:15 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Graham Klyne <[email-masked]>
CC: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

Graham Klyne wrote:

>
> At 22:26 20/01/04 +0000, (someone) wrote:
>
>> >But maybe it should be " 4DIGIT / %x31-39 3*DIGIT / obs-year ", i.e.
>> >years with more than 4 digits must not have leading zeros.
>
>
> Er, shouldn't that be:
>
>   4DIGIT / %x31-39 4*DIGIT / obs-year
>
> to avoid syntactic ambiguity between the first and second options?
>

Why treat 4-digit years with leading zeros specially?  I.e. that allows:
0001
0099
0345

Why not:

    %x31-39 3*DIGIT / obs-year

Date: Wed, 21 Jan 2004 11:07:33 +0000
To: [email-masked]
From: Graham Klyne <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

At 22:26 20/01/04 +0000, (someone) wrote:
> >But maybe it should be " 4DIGIT / %x31-39 3*DIGIT / obs-year ", i.e.
> >years with more than 4 digits must not have leading zeros.

Er, shouldn't that be:

   4DIGIT / %x31-39 4*DIGIT / obs-year

to avoid syntactic ambiguity between the first and second options?

#g

------------
Graham Klyne
For email:
http://www.ninebynine.org/#Contact

From: Arnt Gulbrandsen <[email-masked]>
To: Al Costanzo <[email-masked]>
Subject: Re: making mail traceable
Cc: [email-masked], [email-masked]
Date: Wed, 21 Jan 2004 10:08:13 +0100
Sender: [email-masked]

Al Costanzo writes:
> This is not a perfect solution but it helps greatly. I know you will 
> retort it wont work with laptops and it wont work with yada yada 
> yada... but it will work in many cases.

If it won't work for the CEO sitting in his hotel room far away typing 
on his laptop, the company won't adopt it.

That is to say, "work in many cases" is not sufficient. It must also not 
break too many critical cases.

Arnt

To: [email-masked]
Xref: clerew local.mime:2986
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Date: Tue, 20 Jan 2004 22:26:04 GMT
Lines: 21
Sender: [email-masked]

In <[email-masked]> [email-masked] (Claus Färber) writes:

>But maybe it should be " 4DIGIT / %x31-39 3*DIGIT / obs-year ", i.e.  
>years with more than 4 digits must not have leading zeros.

Yes, I like that. There remains the issue of how big a field implementors
should provide to store such years, but if they do not anticipate their
implementations still being around when the Y10K bug looms, they have no
problem. If they do so anticipate, then they choose to be safe until the
Y100K bug looms, and so on.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "Al Costanzo" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Subject: Re: making mail traceable
Date: Tue, 20 Jan 2004 16:37:33 -0500
Sender: [email-masked]

Bruce,

Let me first speak about the re-invention of the wheel.  IMHO when a wheel
does not turn it is because it is square or flat and needs to be redesigned.

Walk up to someone in IT and ask the the Log & Lat of their office and see
if they know it.  Now ask them their offices postal address. I would say
100% of the people will know where they work but few if any will know their
log and lat position save the few who received a hand held GPS unit for the
holiday.

Because of this LOC is seldom used. So my first point is to get vaild data
into the DNS that people can 'easily' get without a GPS unit.

Next, different laws apply to different parts of the country and the world.
I am not only speaking about spammers. I am talking about credit card fraud,
identity theft, etc. There are many issues and unfortunately the world is
not one happy IETF meeting.

There needs to be a way to know where the mail is physically comming from
and what laws apply so that in the case of credit card fraud, it can quickly
be turned over to the proper authorities.

Lets look at a non-spam, non government application for a moment to peak
your interest in the concept.

We have a person that just placed a credit card order on our website. The
cerdit card information is all vaild and the address is correct. All is
verified. This of course does not mean that the card is not stolen.  An
addtional step would be to know where the email oringinated from.

This is not a perfect solution but it helps greatly. I know you will retort
it wont work with laptops and it wont work with yada yada yada... but it
will work in many cases.

It will prevent someone from Pakistan (for example) saying they live in Ohio
USA and using a stolen credit card from ohio. ( I saw this example myself)
Have the data avaliable will help both electronic mail as well as e-commerce
applications.

I know it will not be perfect overnight but it will help greatly, if you
have a better solution let me know.  But the thing that is needed to knaot
this work is NOT verifying an email address is valid but verifying a
location is valid and making the solution easy enough for the average joe
without a GPS to make it work.

If you have a better idea, PLEASE tell me.  But the solution is knowing the
physical addresss IMO.

Al Costanzo

"Bruce Lilly" wrote:
----- Original Message ----- 
From: "Bruce Lilly" <[email-masked]>
To: "Al Costanzo" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Monday, January 19, 2004 11:27 PM
Subject: Re: making mail traceable

>
> Al Costanzo wrote:
>
> >To the point IMO there needs to be a way to determine where mail came
from,
> >a physical location.  When I first became active in the IETF I worked at
a
> >college, since then I have worn many hats, as the owner of an ISP, I can
see
> >the need for this type of information.
> >
> >First of all this could be used to filter email, second, to determine
> >juristication of law, third allow an ISP decide what action it would /
> >should take against the SPAMMER or the ISP that is allowing the SPAM to
> >flood into his mail server or network.
> >
> >
> First I'll say that I agree with Keith's points.  But I want to add a
> different perspective.
>
> While *you* may think that you can filter email based on geographic
> location of the
> originator, assuming that you can accurately determine that, I cannot.
> I correspond
> with people around the world, and I cannot implement such a crude means
> of "filtering".
>
> Second, jurisdiction may depend on where intermediate servers are
> located. In any event,
> I don't hold much hope for solutions based on hordes of shysters.
> Moreover, knowing
> the location of the originator might not help; according to what I've
> heard the UselesS
> Congress' recent SPAMMERS-CAN-SPAM bill overrides tougher state laws. And
> obviously US law is ineffective w.r.t. spam originated outside of the USA.
>
> Third, if an ISP's customer is spamming, that ISP's terms of service
> should be sufficient to
> deal with the problem. If an ISPs networking peers are sources of spam,
> cutting off peering
> arrangements (or threats do do so) are a potential solution.  If an ISP
> is operating an open
> relay, he should purchase a clue.
>
>  From a practical point of view, it is unlikely that DNS can hold
> information about the
> location of portable machines (such as the laptop on which I'm composing
> this message,
> which has been in 5 states in the past month).  It also won't help for
> RFC 1918 IP
> addresses.
>
> >[...]
> >The physical location of the machine sending the email is an important
thing
> >to know to fight SPAM and fighting credit card fraud.
> >
> >
> >
> I disagree. The network topology (where the source connects to the
> Internet) is important;
> geographical location is largely irrelevant.
>
> >Creating the GL RR and using this in mail software will be a first step
in
> >solving a serious problem.
> >
> >
> For the moment assuming that geographical location has some value; LOC
> RRs already
> exist -- why reinvent the wheel?
>
> #################################################################
> #################################################################
> #################################################################
> #####
> #####
> #####
> #################################################################
> #################################################################
> #################################################################
>
> #################################################################
> #################################################################
> #################################################################
> #####
> #####
> #####
> #################################################################
> #################################################################
> #################################################################
>

Date: Mon, 19 Jan 2004 23:27:14 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Al Costanzo <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: making mail traceable
Sender: [email-masked]

Al Costanzo wrote:

>To the point IMO there needs to be a way to determine where mail came from,
>a physical location.  When I first became active in the IETF I worked at a
>college, since then I have worn many hats, as the owner of an ISP, I can see
>the need for this type of information.
>
>First of all this could be used to filter email, second, to determine
>juristication of law, third allow an ISP decide what action it would /
>should take against the SPAMMER or the ISP that is allowing the SPAM to
>flood into his mail server or network.
>  
>
First I'll say that I agree with Keith's points.  But I want to add a 
different perspective.

While *you* may think that you can filter email based on geographic 
location of the
originator, assuming that you can accurately determine that, I cannot.  
I correspond
with people around the world, and I cannot implement such a crude means 
of "filtering".

Second, jurisdiction may depend on where intermediate servers are 
located. In any event,
I don't hold much hope for solutions based on hordes of shysters.  
Moreover, knowing
the location of the originator might not help; according to what I've 
heard the UselesS
Congress' recent SPAMMERS-CAN-SPAM bill overrides tougher state laws. And
obviously US law is ineffective w.r.t. spam originated outside of the USA.

Third, if an ISP's customer is spamming, that ISP's terms of service 
should be sufficient to
deal with the problem. If an ISPs networking peers are sources of spam, 
cutting off peering
arrangements (or threats do do so) are a potential solution.  If an ISP 
is operating an open
relay, he should purchase a clue.

 From a practical point of view, it is unlikely that DNS can hold 
information about the
location of portable machines (such as the laptop on which I'm composing 
this message,
which has been in 5 states in the past month).  It also won't help for 
RFC 1918 IP
addresses.

>[...]
>The physical location of the machine sending the email is an important thing
>to know to fight SPAM and fighting credit card fraud.
>
>  
>
I disagree. The network topology (where the source connects to the 
Internet) is important;
geographical location is largely irrelevant.

>Creating the GL RR and using this in mail software will be a first step in
>solving a serious problem.
>  
>
For the moment assuming that geographical location has some value; LOC 
RRs already
exist -- why reinvent the wheel?

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

Date: Mon, 19 Jan 2004 21:38:46 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: James M Galvin <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: cryptographically verifiable fields
Sender: [email-masked]

James M Galvin wrote:

>This is where I think multipart/signed got it right and is a big win (he
>says with all humility :-).  There are only two real requirements.
>
>    the object to be signed must be canonicalized
>
>    it must be universally representable
>  
>
[...]

>Now, if you're looking to add a signature (or some other cryptographic
>trace value) to the outermost headers and have it apply over the entire
>message (including the outermost headers), then you've got a really hard
>problem.
>
>Actually, I don't think "really hard problem" begins to describe just
>how hard it would be.  I can not imagine why anyone would want to do
>such a thing.  It just isn't practical in today's Internet email
>environment.
>
>S/MIME and PGP/MIME inherit this to the extent they use
>multipart/signed.
>  
>
One problem with encapsulation is that the signed information *is* 
encapsulated. That's
fine for an end-to-end application, but doesn't work well for a message 
that needs to be
validated and interpreted at multiple points.  Examples of the latter 
would be a Usenet
control message, or a Usenet article posted to a moderated group.  In 
each case, some of
the message header fields and the body would be signed, while fields 
which are expected
to change in transit (trace fields) would not be signed.  Avoiding 
encapsulation permits
backwards compatibility; existing software can handle the message with 
the same lack of
authentication that is currently the case, while newer software can 
verify authenticity.
With encapsulation, everything has to be rewritten to look inside the 
wrapper.

I think Keith's outline of a solution is viable.

There are a number of other issues that would have to be addressed in 
the Usenet examples:
1. Authorization; a mechanism for distributing and updating 
authorization information
    (who is authorized to approve a moderated article, or to issue 
control messages) is needed.
2. A standardized open mechanism for obtaining information necessary to 
complete
    authentication.  Aside from cost issues, some people are leery of 
dealing with CAs,
    especially after last year's Verisign DNS shenanigans. Some people 
distribute public
    keys via finger service or via a web site, but that generally 
involves manual intervention
    to make the keys available to authentication software.  If an email 
address serves as an
    identifier for the purpose of authorization, then perhaps an SMTP 
extension could be
    used to retrieve a public key corresponding to that email address -- 
but that's just a
    thought.  Of course, given a distributed authorization database, 
public keys could be
    part of that.  The SMTP extension idea might be useful where 
authentication is needed
    but authorization is not.

The same sort of solution might be useful in email.  Assuming a viable 
means of signing
non-trace header fields plus message body, and availability of an open 
automated mechanism
for obtaining public keys, signed email which is fully compatible with 
existing MTAs and MUAs
could quickly become popular. Without a multipart/signed wrapper, such a 
message could
be read by existing MUAs.  Enhanced MUAs/MTAs could handle signed 
messages as follows:
1. author composes message, MUA (or possibly submission MTA) computes a 
hash over non-trace
    header fields plus body and adds a header field consisting of that 
hash encrypted with the
    user's private key (and possibly transfer-encoded for robustness).
2. MTAs transfer message as usual, adding trace fields.
3. recipient's MUA checks for header field containing encrypted hash:
   a) no field: message is not signed (or hash field has been lost in 
transit)
   b) encrypted hash field is found: MUA queries DNS for MX records 
corresponding to sender
       envelope address, then opens connection to an SMTP server to 
retrieve public key, finally
       attempts authentication by decrypting hash and verifying that 
hash is valid
       i) if hash is valid, message is authentic and unchanged (with a 
high degree of probability)
       ii) if hash is not valid, message is forged or has been modified 
in transit
       iii) in case of DNS or SMTP problems, authenticity cannot be 
established.

Obviously such a scheme requires support in MUAs (possibly submission 
MTAs), and SMTP
servers.  If submission MTAs are used for generation of the encrypted 
hash, ISPs could
implement the necessary mechanisms with the exception of MUA 
authentication support
(the ISP would generate a key pair for each email address, using the 
private key for
encryption and returning the public one in response to the SMTP 
extension request; all
of that could be handled transparently to computer-illiterate end users).

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

From: "Al Costanzo" <[email-masked]>
To: <[email-masked]>
Cc: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 21:31:24 -0500
Sender: [email-masked]

Keith,

Normally I can see your point of view, but not in this case.  I find your
response containing a gaping hole in  logic. Allowing someone to know what
City or State an email  came from does NOTHING to give away a persons actual
location only the location of the machine that sent it.

And if a person is worried about being stalked they should contact the
police. If they are afraid that email is going to be giving the location
away they should not be sending email.

Lets be realistic here. Using your logic the peson should not use a phone
either because I stalker may have caller ID. Getting back to reality and the
problem at hand:

To the point IMO there needs to be a way to determine where mail came from,
a physical location.  When I first became active in the IETF I worked at a
college, since then I have worn many hats, as the owner of an ISP, I can see
the need for this type of information.

First of all this could be used to filter email, second, to determine
juristication of law, third allow an ISP decide what action it would /
should take against the SPAMMER or the ISP that is allowing the SPAM to
flood into his mail server or network.

The only thing that completely anonymous email does is cause people to send
volumes of SPAM and waste millions and millions of dollars in credit card
fraud.  We receive thousands of dollars worth of orders from people using
stolen credit cards with anonyous email accounts that cause chargebacks all
the time.

Because of all this fraud, most of our customers have to make hard choices,
such as, ignore orders from Yahoo! and HotMail accounts or accept the
possibility that these orders will result in charge backs.

Credit card owners want to report the card the lost and want to prosecute
the theif but frankly views like yours are going to stifle the abilty to
bring thiefs to justice.

I am sure it is not your intention to help people commit credit card fraud
or send SPAM but realistically there needs to be some type of responsibility
in email assigned to email accounts and thier ISPs and I think it is high
time that this is done.

I do not mean to offend you in any way but on this topic I think you are
totally wrong and hope you re-consider your thoughts on this subject.  Can
tell you have a strong view on this subject, but I would like your help in a
solution and not just say this will be done "over my dead body".  Statements
such as this are not constructive and your argument has no merit.

The physical location of the machine sending the email is an important thing
to know to fight SPAM and fighting credit card fraud.

By not considering this, I do not think you are helping solve a serious
problem that I see existing RIGHT NOW and getting worse every day.

Creating the GL RR and using this in mail software will be a first step in
solving a serious problem.

With Regards,

Al Costanzo

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Al Costanzo" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; <[email-masked]>
Sent: Monday, January 19, 2004 12:49 PM
Subject: Re: making mail traceable

> > I agree with you on this 100%.
> >
> > The exact location may be a disaster to publish but precision of the RR
> > record is configurable.
> >
> > You could just list the Country or Country & State or Country & State &
> > City.
>
> you've obviously never known someone well who was stalked by her
> nutcase ex-husband.
>
> Keith
>
>

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Mon, 19 Jan 2004 18:31:44 -0500
To: Markus Stumpf <[email-masked]>
Sender: [email-masked]

>>> I don't think it is an easy task to find information to add to the
>>> hash.
>> offhand:
>> - subject field (perhaps truncated to XX bytes)
>> - message body
>> - source IP address and port
>> - precise date/time (not the Date header field)
>> - *maybe* some form of the envelope recipient list
>
> But all of this can be used for replay attacks. Get an account at
> big-email-provider-1 and send the spam mail to your address at
> big-email-provider-2.

either you strip off the originator-id field after it arrives at 
big-email-provider-2, or you don't.

if you don't strip it off, then the original originator-id is still 
there, and it associates your spam with your account at 
big-email-provider-1.

if you do strip it off, another originator-id field gets added by 
big-email-provider-2, and your spam gets associated with that account.

either way, the spam is traceable to an account that is associated with 
you.  recipients of the message can complain to whichever ISP issued 
the originator-id field, and that ISP will figure out pretty quickly 
that you're a spammer, and blacklist you.  when other recipients 
inquire about that originator-id (or even a different originator-id 
that maps to the same account), they'll find out that you're 
blacklisted.

> Take the message and reinject it via a proxy
> server by adding a fake Received: line and using a faked envelope 
> sender
> to make it look like a forward and a consitent chain of mailservers.
> You can use the same message some thousand times with any of some
> thousand open proxy servers and for any envelope recipient you like
> (replay attack).

if I understand what you're saying,  it shouldn't matter.  the number 
of hops that the message takes, and the depth of branching in the tree, 
should be irrelevant.  what really matters is that you can't claim that 
you didn't send the message.

the real trick is to prevent the other kind of attack - some miscreant 
wants to discredit some vendor, so they take a single message that the 
vendor sent legitimately and re-send it to a few million people.  we 
need to make sure that the message is traced to the miscreant, not the 
vendor.

> With Bcc mimic and multiple RCPT TOs in one stream it might be 
> dangerous
> or impossible to use some form of the envelope recipient list.

there are a number of problems with using the recipient list.  that's 
why I said "maybe".

Keith

Date: Tue, 20 Jan 2004 00:15:44 +0100
From: Markus Stumpf <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Subject: Re: making mail traceable
Organization: SpaceNet AG, Muenchen, Germany
Sender: [email-masked]

On Mon, Jan 19, 2004 at 05:57:53PM -0500, Keith Moore wrote:
> >I don't think it is an easy task to find information to add to the 
> >hash.
> offhand:
> - subject field (perhaps truncated to XX bytes)
> - message body
> - source IP address and port
> - precise date/time (not the Date header field)
> - *maybe* some form of the envelope recipient list

But all of this can be used for replay attacks. Get an account at
big-email-provider-1 and send the spam mail to your address at
big-email-provider-2. Take the message and reinject it via a proxy
server by adding a fake Received: line and using a faked envelope sender
to make it look like a forward and a consitent chain of mailservers.
You can use the same message some thousand times with any of some
thousand open proxy servers and for any envelope recipient you like
(replay attack).

With Bcc mimic and multiple RCPT TOs in one stream it might be dangerous
or impossible to use some form of the envelope recipient list.

Maybe I am missing something, but I can't see how this helps making
the trace of the message more trustworthy than the Received lines only.

	\Maex

-- 
SpaceNet AG            | Joseph-Dollinger-Bogen 14 | Fon: +49 (89) 32356-0
Research & Development |       D-80807 Muenchen    | Fax: +49 (89) 32356-299
"The security, stability and reliability of a computer system is reciprocally
 proportional to the amount of vacuity between the ears of the admin"

Cc: Keith Moore <[email-masked]>, James M Galvin <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Mon, 19 Jan 2004 18:01:19 -0500
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

Nathaniel,

I don't think you and I are in significant disagreement. I do think 
we're probably talking about slightly different things.

Keith

>> I'm interested in solving a problem, not in trying to make the US 
>> government look good.  Now if it turns out that, by some accident or 
>> divine intervention, CANSPAM really does turn out to be useful, 
>> that's fine with me.  But it doesn't seem wise to depend on it 
>> working.
>
> If you believe that CANSPAM might be merely "useless" then I think you 
> are a wild-eyed optimist.  This isn't primarily about doing good -- 
> although that would be nice -- it's about mitigating damage.  You're 
> in for a rude awakening when the government starts regulating the 
> contents of your sendmail.cf file.
>
>> I think some of us who understand the complexities should try to play 
>> a constructive role by solving the problem as best as we can - but 
>> that doesn't include taking direction from a technically inept, 
>> parochial legislative body, that is easily swayed by direct marketing 
>> proponents into acting contrary to the public interest.
>
> I'm not advocating your taking direction from anyone, Keith.  I'm 
> proposing that we put our heads together and give them the best 
> *advice* we can collectively come up with on *how* to do some things 
> they are determined to do one way or another.
>
>> I believe the least harmful way of doing this is to provide 
>> traceability in such a way that government intervention isn't needed 
>> to solve most of the spam problem.
>
> Sounds perfect.  Unfortunately the train has left the station -- the 
> government *is* intervening, and we'd better be prepared to suggest a 
> strategy for them, or they'll do something unnecessarily stupid, I 
> promise.  While we discuss these issues, the Beltway traffic is 
> snarled by software vendors pitching spam "solutions" to the feds.   
> I'm sure some of them are much worse than others; it would help if the 
> ietf could provide a guiding vision.  (But then again, I'm not sure I 
> know why I would expect the ietf-822 list to succeed where the ASRG 
> has failed, alas.)  -- Nathaniel
>

Cc: Keith Moore <[email-masked]>, James M Galvin <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Mon, 19 Jan 2004 17:57:53 -0500
To: Markus Stumpf <[email-masked]>
Sender: [email-masked]

On Jan 19, 2004, at 4:41 PM, Markus Stumpf wrote:

> On Mon, Jan 19, 2004 at 12:18:14PM -0500, Keith Moore wrote:
>> It also depends on what is included in the hash.  I believe it will be
>> necessary to omit some information from the hash in order to get the
>> hash to survive most existing mail transports.  I don't think this is 
>> a
>> problem as long as we don't treat the originator-id tag as a digital
>> signature.
>
> IMHO it is a question of what one wants to accomplish with the 
> signature.
> If it's end-to-end, aka MUA <> sMTA <> rMTA <> MUA signatures are nice
> to have, but as I can trust my rMTA's Received: line about the sending 
> MTA
> it is not of much additional information.

it's a moot point.  it's much easier to make e2e sigs work than to make 
hop-by-hop sigs work.

> I don't think it is an easy task to find information to add to the 
> hash.

offhand:
- subject field (perhaps truncated to XX bytes)
- message body
- source IP address and port
- precise date/time (not the Date header field)
- *maybe* some form of the envelope recipient list

> Adding the body of the email to the hash is also playing vabanque as
> e.g. mailing lists add trailers to the message and break the hash.

interesting point that.  but if the body is the last thing to be hashed 
you may be able to
recheck the hash at every line boundary.  you might also have the id 
field include the
number of bytes from the body that are hashed.

Date: Mon, 19 Jan 2004 17:43:37 -0500
Subject: Re: making mail traceable
Cc: James M Galvin <[email-masked]>, [email-masked]
To: Keith Moore <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

On Monday, January 19, 2004, at 03:03  PM, Keith Moore wrote:

> I'm interested in solving a problem, not in trying to make the US 
> government look good.  Now if it turns out that, by some accident or 
> divine intervention, CANSPAM really does turn out to be useful, that's 
> fine with me.  But it doesn't seem wise to depend on it working.

If you believe that CANSPAM might be merely "useless" then I think you 
are a wild-eyed optimist.  This isn't primarily about doing good -- 
although that would be nice -- it's about mitigating damage.  You're in 
for a rude awakening when the government starts regulating the contents 
of your sendmail.cf file.

> I think some of us who understand the complexities should try to play 
> a constructive role by solving the problem as best as we can - but 
> that doesn't include taking direction from a technically inept, 
> parochial legislative body, that is easily swayed by direct marketing 
> proponents into acting contrary to the public interest.

I'm not advocating your taking direction from anyone, Keith.  I'm 
proposing that we put our heads together and give them the best 
*advice* we can collectively come up with on *how* to do some things 
they are determined to do one way or another.

> I believe the least harmful way of doing this is to provide 
> traceability in such a way that government intervention isn't needed 
> to solve most of the spam problem.

Sounds perfect.  Unfortunately the train has left the station -- the 
government *is* intervening, and we'd better be prepared to suggest a 
strategy for them, or they'll do something unnecessarily stupid, I 
promise.  While we discuss these issues, the Beltway traffic is snarled 
by software vendors pitching spam "solutions" to the feds.   I'm sure 
some of them are much worse than others; it would help if the ietf 
could provide a guiding vision.  (But then again, I'm not sure I know 
why I would expect the ietf-822 list to succeed where the ASRG has 
failed, alas.)  -- Nathaniel

Date: Mon, 19 Jan 2004 22:41:37 +0100
From: Markus Stumpf <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: James M Galvin <[email-masked]>, ietf-822 <[email-masked]>
Subject: Re: making mail traceable
Organization: SpaceNet AG, Muenchen, Germany
Sender: [email-masked]

On Mon, Jan 19, 2004 at 12:18:14PM -0500, Keith Moore wrote:
> It also depends on what is included in the hash.  I believe it will be 
> necessary to omit some information from the hash in order to get the 
> hash to survive most existing mail transports.  I don't think this is a 
> problem as long as we don't treat the originator-id tag as a digital 
> signature.

IMHO it is a question of what one wants to accomplish with the signature.
If it's end-to-end, aka MUA <> sMTA <> rMTA <> MUA signatures are nice
to have, but as I can trust my rMTA's Received: line about the sending MTA
it is not of much additional information.

The more interesting situation is if the message is multi hop, like with
forwards. Then we have a situation of MUA <> sMTA <> forw MTA <> rMTA <> MUA
and one has to trust "forw MTA" to add correct trace information about
sMTA. It is desirable to have some signature that allows my rMTA to
verify that the envelope sender (aka the originator address) and the
trace information provided by fMTA are correct and also to protect
against replay attacks, where a signature derived from MUA <> sMTA
is intercepted and abused by evil MTA to send spam and forge sender
addresses and trace information.

I don't think it is an easy task to find information to add to the hash.
Even if the Date field looks kinda sexy to be added it will cause all
sorts of problems, as even today a not too small amount of emails take 
a week or more to reach their rMTA and there are a lot of MTAs with
wrong time out there. So the acceptance window has to be quite big and
that leaves enough time to abuse such a hash for replay attacks.

Adding the body of the email to the hash is also playing vabanque as
e.g. mailing lists add trailers to the message and break the hash.

	\Maex

-- 
SpaceNet AG            | Joseph-Dollinger-Bogen 14 | Fon: +49 (89) 32356-0
Research & Development |       D-80807 Muenchen    | Fax: +49 (89) 32356-299
"The security, stability and reliability of a computer system is reciprocally
 proportional to the amount of vacuity between the ears of the admin"

Cc: Keith Moore <[email-masked]>, James M Galvin <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Mon, 19 Jan 2004 15:03:47 -0500
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

> Keith, I fear you're still fighting last year's battle when you write:
>
>> Also, for lots of reasons I don't think that giving law enforcement a 
>> way to track down spammers is a desirable way to solve the spam 
>> problem.  It would get the government too involved in mediating 
>> people's communications, it would invite favoritism, it would require 
>> too many LE resources, and for that reason it would be hard to limit 
>> abuse.  I'd far rather find a way for the net to be self-policing 
>> by....
>
> I don't find it desirable either, and I too would rather have seen the 
> community find a self-policing solution.  (For the record, I would 
> also prefer for people to be so loving and compassionate that violence 
> never reached the level of requiring police or governments.)  However, 
> CANSPAM represents the official rejection of our preferences by the US 
> government,

I'm interested in solving a problem, not in trying to make the US 
government look good.  Now if it turns out that, by some accident or 
divine intervention, CANSPAM really does turn out to be useful, that's 
fine with me.  But it doesn't seem wise to depend on it working.

> Washington today is full of people trying to figure out their new 
> mandate of regulating email.  They're going to do it, regardless of 
> our preferences, and I think some of us who understand the 
> complexities should try to play a constructive role in fleshing out 
> that mandate.

I think some of us who understand the complexities should try to play a 
constructive role by solving the problem as best as we can - but that 
doesn't include taking direction from a technically inept, parochial 
legislative body, that is easily swayed by direct marketing proponents 
into acting contrary to the public interest.

>  I consider it virtually inevitable that they will require some form 
> of enhanced tracability in the name of stopping spam -- it's just 
> about the very first instinct of a bureaucracy -- so the question I am 
> trying to pose is, essentially, what is the least harmful way of doing 
> this?

I believe the least harmful way of doing this is to provide 
traceability in such a way that government intervention isn't needed to 
solve most of the spam problem.  Then there will be strong arguments 
for restricting government intervention to more dangerous problems of 
email abuse - e.g. mailing of child porn and bomb threats - and to 
impose judicial review and auditing on such interventions as safeguards 
against abuse.

Keith

Date: Mon, 19 Jan 2004 13:46:21 -0500
Subject: Re: making mail traceable
Cc: James M Galvin <[email-masked]>, [email-masked]
To: Keith Moore <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

Keith, I fear you're still fighting last year's battle when you write:

> Also, for lots of reasons I don't think that giving law enforcement a 
> way to track down spammers is a desirable way to solve the spam 
> problem.  It would get the government too involved in mediating 
> people's communications, it would invite favoritism, it would require 
> too many LE resources, and for that reason it would be hard to limit 
> abuse.  I'd far rather find a way for the net to be self-policing 
> by....

I don't find it desirable either, and I too would rather have seen the 
community find a self-policing solution.  (For the record, I would also 
prefer for people to be so loving and compassionate that violence never 
reached the level of requiring police or governments.)  However, 
CANSPAM represents the official rejection of our preferences by the US 
government, and I see little gain in dwelling on battles already lost.

Washington today is full of people trying to figure out their new 
mandate of regulating email.  They're going to do it, regardless of our 
preferences, and I think some of us who understand the complexities 
should try to play a constructive role in fleshing out that mandate.  I 
consider it virtually inevitable that they will require some form of 
enhanced tracability in the name of stopping spam -- it's just about 
the very first instinct of a bureaucracy -- so the question I am trying 
to pose is, essentially, what is the least harmful way of doing this?

Preserving anonymous email is very important to me, and a battle worth 
fighting very passionately in the long run.  The architecture of any 
anti-spam tracing facilities will be critical to that battle.  I think 
we need to accept that we've lost the "whether" battle -- e.g. whether 
or not there will be police on the net trying to trace email --  and 
try to respond in a strategically intelligent way, by helping define a 
tracing architecture that is anonymity-friendly.  I expect that some 
people I respect enormously may choose to fall on their swords in total 
opposition to the "email police."  But I hope that some of the many 
great minds of the IETF will focus on the challenging design question 
of how to preserve interpersonal anonymity while facilitating spam 
tracing.   I'm not sure I would want to live with the result of forcing 
that to be an either/or choice.  -- Nathaniel
---
Nathaniel S. Borenstein <[email-masked]>
President, Computer Professionals for Social Responsibility
http://www.cpsr.org

Date: Mon, 19 Jan 2004 11:17:47 -0800 (PST)
From: [email-masked]
To: Keith Moore <[email-masked]>
cc: [email-masked]
Subject: Re: making mail traceable
Sender: [email-masked]

Perhaps there exists a way to combine mail trace field with dns 
(in part to reduce size of the has hash necessary for email). 

The way it could work is that public key is available for verification 
from dns (this must include version information allowing company to 
use multiple keys at the same time when moving from one to another or
just when its convinient, i.e. when it has multiple servers). This public 
key can be used to verify the private verificaion id embedded in the email
as well as parameters of the actual email as it was being processed.
My thinking is that good parameters for hash are RCPT TO and MAIL FROM, 
date and message id (unfortunetly RCPT TO can have multiple addresses in 
fact it can be very long list with current SMTP standard).

The best would actually be if this could be done as part of "Received" 
information where each mail server on the way could add additional 
"Received" header and this header would be in such standard format as to 
be easily parsed by automated systems, something like:

> Received: FROM above.proper.com (above.proper.com [208.184.76.39])
          BY sokol.elan.net (8.12.5/8.12.5)
          FOR [email-masked] ENVELOPE-FROM [email-masked]
          SENT-ON 200301191237 MESSAGE-ID i0JIT3lQ001354
          VERIFY-DSN key-name#mail.imc.org BY-USING algorithm-name
          VERIFICATION-ID xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
	  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The systems concerned about message origin and its path would then use
the known parameters of the message already included in the received 
header (FOR, FROM, MESSAGE-ID) and VERIFICATION-ID and use algorithm 
specified (this should perhaps be embedded as part of verification-id)
and check if such verification id could only have been created by somebody 
who used private key which corresponds to public key available from
VERIFY-DSN location. I'm however concerned that standard X.509 or similar
algorithms would create and ID that is too long, but 2-4 lines as specified
above is probably ok which gives us up to 256 bytes. Another thing to be 
concerned about is how quickly (and how much cpu resources are needed) 
when creating such verification ids and when verifiying them.

-- 
William Leibzon
Elan Networks
[email-masked]

Date: Mon, 19 Jan 2004 13:10:14 -0500
Subject: Re: Revisiting RFC 2822 grammar
Cc: Charles Lindsey <[email-masked]>
To: [email-masked]
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

Personally I see no reason to place any limit on the number of digits 
in the year, but would not object to anything that took us past the 
expected heat death of the universe.  However, I do think a four digit 
limitation would be pointlessly short-sighted.  -- Nathaniel

On Monday, January 19, 2004, at 07:53  AM, Bruce Lilly wrote:

>
> Charles Lindsey wrote:
>
>> So you can a year with 17 digits in it? I don't think that is a good 
>> idea.
>>
>>
> Pete didn't want to have to revise the document in the year 9999 :-).
>
>

Cc: Keith Moore <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 12:49:36 -0500
To: "Al Costanzo" <[email-masked]>
Sender: [email-masked]

> I agree with you on this 100%.
>
> The exact location may be a disaster to publish but precision of the RR
> record is configurable.
>
> You could just list the Country or Country & State or Country & State &
> City.

you've obviously never known someone well who was stalked by her 
nutcase ex-husband.

Keith

From: "Al Costanzo" <[email-masked]>
To: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 12:40:08 -0500
Sender: [email-masked]

Keith,

I agree with you on this 100%.

The exact location may be a disaster to publish but precision of the RR
record is configurable.

You could just list the Country or Country & State or Country & State &
City.

So I don't think a nutcase is an issue here.

Al Costanzo

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Al Costanzo" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; <[email-masked]>
Sent: Monday, January 19, 2004 12:19 PM
Subject: Re: making mail traceable

> > Could you expand on your reason for this a bit? "over my dead body"
> > does not
> > let me see why you are so opposed to this idea.
> >
> > I want to know where the email is comming from.
>
> there are too many nutcases in the world who would physically attack
> people if they knew the actual, physical source of their email.
>
>

From: Dave Crocker <[email-masked]>
To: <[email-masked]>
Date: Mon, 19 Jan 2004 09:34:44 -0800
Subject: Re: Re: making mail traceable
Sender: [email-masked]

Folks,

>     do we really care where the message was sent from, as opposed to who
>     sent it?
> 
> My current thinking is I don't want to tightly couple the message
> direclty to a person, even in an ephemeral way.  I would rather note
> sites, especially those that have multiple users.
> 
> So, rather than even having a home user site create a "who sent it
> identifier", I would prefer the ISP submission server create a "message
> passed through my server identifier", 

We need to distinguish between having a mechanism with semantics about the 
MTA, versus one with semantics about the author of the message.  Which 
underlying semantic do we really care about and why?

I believe the semantic we care about, for MTAs, is that they are part of 
well-behaved networks, not that they are "authorized" by the author, or that 
they in turn vouch directly for the author.  In other words, is the hosting 
ISP running a coherent, controlled MTA environment?  If the answer is yes, we 
are not certain that other MTA sources from that ISP are rogue, but we are 
certain they are not vouched for, by the ISP.

I believe the biggest semantic we want to see is "this author is 
well-behaved".  Anything about the MTA is indirect.  It might be useful, but 
it's not core.  A step in that direction is to authenticate the author or to 
provide an assurance that they can be located.

Schemes that involve MTA registration for/by the author (SPF, LMAP, RMX, ...) 
confuse these two semantics and they create very Procrustean usage and 
administration scenarios. For an operation with users that send from outside 
the ISP, they create problematic usage patterns.

In contrast, schemes that focus on the author directly are more flexible.  It 
is quite straightforward for an ISP to operate that scheme on behalf of the 
author, in those environments that permit it.  This means that direct 
author-focused schemes permit a number of operational styles, one of which is 
equivalent to  the MTA registrations such as SPF, LMAP and RMX. This includes 
having the ISP mask the actual author, while retaining accountability for 
them.

d/
--
Dave Crocker <dcrocker-at-brandenburg-dot-com>
Brandenburg InternetWorking <http://brandenburg.com>

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Mon, 19 Jan 2004 12:30:46 -0500
To: James M Galvin <[email-masked]>
Sender: [email-masked]

> My current thinking is I don't want to tightly couple the message
> direclty to a person, even in an ephemeral way.

I don't really like doing that either.  But I don't think the 
granularity of "site" is good enough to identify and marginalize 
spammers - actually experience with trying to blacklist sites (or IP 
address blocks associated with sites) IMHO indicates that it is not 
good enough for this purpose.

Also, for lots of reasons I don't think that giving law enforcement a 
way to track down spammers is a desirable way to solve the spam 
problem.  It would get the government too involved in mediating 
people's communications, it would invite favoritism, it would require 
too many LE resources, and for that reason it would be hard to limit 
abuse.  I'd far rather find a way for the net to be self-policing by 
allowing recipients (or recipient sites) to marginalize spammers 
without actually finding out who the spammers are.  Of course, for 
serious infractions of the law, LE will still be able to trace 
originator-ids to the actual people who sent the messages - I don't see 
any way of avoiding this.

> So, rather than even having a home user site create a "who sent it
> identifier", I would prefer the ISP submission server create a "message
> passed through my server identifier", along with additional private log
> entries that make it possible to trace the message back to its actual
> origin.  Those log entries can then be retrieved through ordinary law
> enforcement means.

This would be an acceptable implementation of originator-id.  But to 
make the system complete ordinary users would need to be able to query 
the ISP to learn something about the sender's reputation.

>     Also, just as I don't want to expose the actual identity of every
>     message sender, I wouldn't want to expose the location of every 
> message
>     sender.  (I can see it now - send an e-mail critical of Dubya, and 
> a
>     G-man knocks on your door within the hour...)
>
> Yes, well, any security measure that involves a known principle carries
> that risk so I don't view it as a reason not to do this.

Knowing who you are and knowing where you are are different things.  If 
they were the same thing Usama bin Laden would be dead now.

Keith

Cc: Keith Moore <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 12:19:47 -0500
To: "Al Costanzo" <[email-masked]>
Sender: [email-masked]

> Could you expand on your reason for this a bit? "over my dead body" 
> does not
> let me see why you are so opposed to this idea.
>
> I want to know where the email is comming from.

there are too many nutcases in the world who would physically attack 
people if they knew the actual, physical source of their email.

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Mon, 19 Jan 2004 12:18:14 -0500
To: James M Galvin <[email-masked]>
Sender: [email-masked]

>>> Doesn't RFC1847 (security multiparts) already provide both of these,
>>> or at least the framework sans the actual algorithm?
>>
>>     Only if everyone's user agents treated a security multipart
>>     containing a message semantically the same as they would the
>>     original message.
>>
>> Sure, so your primary concern is that the use of security multiparts
>> would be a less backwards compatible change than a change to the
>> Received syntax or adding a new header.
>
>     That, and there is a semantic difference between a signed message 
> and a
>     cryptographically verifiable trace field in a message.
>
> Agreed.  My point was that the same technology would be used but would
> be interpreted differently.

It's going to be very similar technology, but the actual protocol used 
needs to be different.  If submission servers start applying security 
multiparts to messages it's going to break things, and it's going to be 
misinterpreted.

>> I can imagine it might be a local policy preference for all messages 
>> to
>> be examined by a person instead of rejecting it if the hash validation
>> fails.  However, it should be the case that except for human review 
>> the
>> message is rejected (for whatever we decide rejection means).  There
>> may be some edge cases where broken mailers do really obscure things 
>> to
>> messages to break the hash, but either you are doing security right or
>> you don't do it.
>
>     Actually I don't see this as a security measure.  It doesn't really
>     protect systems from any kind of attack, and it doesn't 
> authenticate
>     the message as being authored or witnessed by any publicly known
>     principal name.  It doesn't even have to provide an unimpeachable
>     assurance of non-repudiation -- it just has to be good enough that 
> it's
>     infeasible to make large quantities of spam look as if it were
>     associated with originator-ids other than those controlled by the
>     spammers.
>
> This seems incongruous to me.
>
> Some principal is going to assign a hash.  There has to be a principal
> because the hash has to be digitally signed.  If it's not there's no
> point having it for the purpose as described.
>
> This means the message is at least witnessed and there's 
> non-repudiation
> to the extent the principal can not deny having handled the message.
> That is a security measure.

What I was responding to is the "either you are doing security right or 
you don't do it".  I certainly agree that this should be "done right" 
(to the extent that we can figure out what "right" is) but I think that 
calling this a "security" measure invites confusion between this 
application of security technologies (public-key cryptography, 
noninvertable hash functions) and other applications of those 
technologies, for which "doing it right" means something different than 
it does here.

> As to whether any of the content is impeachable, that depends on when
> the hash is actually created.

It also depends on what is included in the hash.  I believe it will be 
necessary to omit some information from the hash in order to get the 
hash to survive most existing mail transports.  I don't think this is a 
problem as long as we don't treat the originator-id tag as a digital 
signature.

>>> I'm sorry but I just don't see what you have in mind that is worse 
>>> than it is today,
>>
>>     Anyway, the reason my proposal allows originator-ids to be 
>> ephemeral is to make it hard for ordinary people to track messages - 
>> I believe anonymous speech is important - and also out of recognition 
>> that spammers can probably get ephemeral accounts anyway.
>>
>> My point is that we are not that [far] from this already.  To stretch 
>> the example to the extreme, the US Government could make a law that 
>> ISPs have to make sure they can map a From: email address to an 
>> actual person before accepting submission of a message (in the US of 
>> course).  Why do we need a special identifier?
>
>     The US Congress might or might not care, but for a variety of 
> reasons it would not be appropriate to use From in this way.  There is 
> no defined header field that can be used for this purpose without 
> conflicting with valid and existing uses of the field.  (Sender would
> have been the right thing, but it's too widely misused now.)
>
> What you say is true but it doesn't refute what I said.  Does this 
> mean you agree with me?

I still think an Originator-id field would be at least somewhat easier 
for governments to abuse in this way than any of the existing fields.  
Variant use of From is sufficiently widespread that I suspect there 
would be some pushback against the government trying to constrain its 
use.

Date: Mon, 19 Jan 2004 12:17:47 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: cryptographically verifiable fields
To: ietf-822 <[email-masked]>
Sender: [email-masked]

On Sun, 18 Jan 2004, Bruce Lilly wrote:

    Could somebody outline a process whereby a single field or group of
    fields in a message could be signed, with the following conditions:

    1. the mechanism is robust w.r.t. common types of message munging
       (reordered fields, possible dropping of fields (obviously, let's
       assume that the field that is signed isn't dropped), addition of
       trailing whitespace, etc.)

This is where I think multipart/signed got it right and is a big win (he
says with all humility :-).  There are only two real requirements.

    the object to be signed must be canonicalized

    it must be universally representable

There's a couple details in each of those but basically you create a
7bit opaque object to sign and transfer from the originator to the
recipient.  That object is pushed one level inside the message, inside
the outermost multipart/signed.

Now, if you're looking to add a signature (or some other cryptographic
trace value) to the outermost headers and have it apply over the entire
message (including the outermost headers), then you've got a really hard
problem.

Actually, I don't think "really hard problem" begins to describe just
how hard it would be.  I can not imagine why anyone would want to do
such a thing.  It just isn't practical in today's Internet email
environment.

S/MIME and PGP/MIME inherit this to the extent they use
multipart/signed.

    2. the mechanism is not subject to replay attacks (e.g. copying the
       signed field from one message to another)

This is a separate requirement, easily included with a protected Date:
and/or nonce available.

    I believe that S/MIME and PGP/MIME signed messages are robust w.r.t.
    those criteria, since the signed message is itself transfer encoded
    (if necessary) and encapsulated via MIME;

They can be robust with respect to replay but are not by default.

Jim

Date: Mon, 19 Jan 2004 12:03:33 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On Sun, 18 Jan 2004, Keith Moore wrote:

    > To this point I would say, we need to know the physical location of the
    > machine (node) on the Internet sending the email.

    do we really care where the message was sent from, as opposed to who
    sent it?

This is a really good question.

My current thinking is I don't want to tightly couple the message
direclty to a person, even in an ephemeral way.  I would rather note
sites, especially those that have multiple users.

So, rather than even having a home user site create a "who sent it
identifier", I would prefer the ISP submission server create a "message
passed through my server identifier", along with additional private log
entries that make it possible to trace the message back to its actual
origin.  Those log entries can then be retrieved through ordinary law
enforcement means.

Now, perhaps from an implementation point of view that's not much
different than an emphemeral original identifier, but I believe it will
be perceived differently and the distinction is important.

    Also, just as I don't want to expose the actual identity of every
    message sender, I wouldn't want to expose the location of every message
    sender.  (I can see it now - send an e-mail critical of Dubya, and a
    G-man knocks on your door within the hour...)

Yes, well, any security measure that involves a known principle carries
that risk so I don't view it as a reason not to do this.

Jim

From: "Al Costanzo" <[email-masked]>
To: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 12:02:44 -0500
Sender: [email-masked]

Could you expand on your reason for this a bit? "over my dead body" does not
let me see why you are so opposed to this idea.

I want to know where the email is comming from.

Al Costanzo
----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Al Costanzo" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; <[email-masked]>
Sent: Monday, January 19, 2004 11:17 AM
Subject: Re: making mail traceable

>
> "over my dead body".
>
> Keith
>
> On Jan 19, 2004, at 11:05 AM, Al Costanzo wrote:
>
> > Keith,
> >
> > I think making mail traceable is the very best thing that the working
> > group
> > can do.  People are sick of SPAM and they need a solution.
> >
> > I already spoke on the reasons why believe this information needs to
> > be in
> > DNS in a previous post but I did not go into the detals as to what this
> > could do to help the situation.
> >
> > In the example below, the infomation that would be used by Client's
> > mail
> > software or the MTA is the portion before the double quote, the
> > information
> > after the doulble quote are for ''non-machine viewing''.
> >
> > After GL RR identifier you have the two char country code followed by
> > the
> > postal code.
> >
> > GL US.45420.1910
> >
> > We now have states that have enacted laws concerning spam and now the
> > federal government as well.  But even if you put this as a reason to
> > use
> > this idea aside, having this information would allow people to
> > configure
> > email software they are using to block emails from a country they do
> > not
> > wish to receive email from as well as being able to trace where the
> > email
> > came from.
> >
> > In addtion, just as there is no rule forcing an ISP to use this RR,
> > mail
> > software could be set to block email from an address that do not
> > contain the
> > information in the same way the ISPs block email that is comming from
> > an A
> > record that do not have a PTR record now.
> >
> >
> >                 IN      NS      ns.akc.net.
> >
> > uspring         IN      A       192.188.192.2
> >                 IN      MX      5       mail
> >                 IN      HINFO   Vax VMS
> >                 IN      GL US.45420.1910 "1425 Arbor Avenue, Dayton OH"
> > ftp             IN      CNAME   uspring
> >
> > I will re-post the draft shortly
> >
> > Al Costanzo
> >
> > ----- Original Message -----
> > From: "Keith Moore" <[email-masked]>
> > To: "Graham Klyne" <[email-masked]>
> > Cc: "Keith Moore" <[email-masked]>; "ietf-822" <[email-masked]>
> > Sent: Monday, January 19, 2004 9:34 AM
> > Subject: Re: making mail traceable
> >
> >
> >>
> >>>> But do we really want traceability?  Or to put it another way, do we
> >>>> really want to put hooks in the mail system that make mass
> >>>> surveillance (by governments, or perhaps even by large companies or
> >>>> unscrupulous ISPs) that much easier?
> >>>
> >>> Speaking for myself, I really would like the option to not use
> >>> bandwidth receiving email from previously unknown senders unless its
> >>> source can be traced.
> >>
> >> well I suppose you could say that the reason I'm publicly discussing
> >> these ideas now is that spam has gotten bad enough that extreme
> >> measures seem to be required.
> >>
> >> also, earlier versions of this scheme made surveillance easier than
> >> the
> >> one I just proposed.
> >>
> >> Keith
> >>
> >>
> >
>
>

Date: Mon, 19 Jan 2004 11:52:13 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

On Sun, 18 Jan 2004, Keith Moore wrote:

    >> Doesn't RFC1847 (security multiparts) already provide both of these,
    >> or
    >> at least the framework sans the actual algorithm?
    >
    >     Only if everyone's user agents treated a security multipart
    >     containing a message semantically the same as they would the
    >     original message.
    >
    > Sure, so your primary concern is that the use of security multiparts
    > would be a less backwards compatible change than a change to the
    > Received syntax or adding a new header.

    That, and there is a semantic difference between a signed message and a
    cryptographically verifiable trace field in a message.

Agreed.  My point was that the same technology would be used but would
be interpreted differently.

    >>     - If you really wanted to, you could augment Received or add a new
    >>     trace field that recomputed the hash at each hop (to show if and
    >> where
    >>     the message was corrupted in transport)
    >>
    >> Why would you forward a message that was discovered to be corrupted?
    >
    >     Probably because the message might have been corrupted in a way
    > that
    >     makes the hash invalid without actually changing the content of the
    >     message.  Something tells me it will be difficult to write a
    >     canonicalization function that accommodates all of the various
    > kinds of
    >     message and header munging that is out there.
    >
    > I can imagine it might be a local policy preference for all messages to
    > be examined by a person instead of rejecting it if the hash validation
    > fails.  However, it should be the case that except for human review the
    > message is rejected (for whatever we decide rejection means).  There
    > may
    > be some edge cases where broken mailers do really obscure things to
    > messages to break the hash, but either you are doing security right or
    > you don't do it.

    Actually I don't see this as a security measure.  It doesn't really
    protect systems from any kind of attack, and it doesn't authenticate
    the message as being authored or witnessed by any publicly known
    principal name.  It doesn't even have to provide an unimpeachable
    assurance of non-repudiation -- it just has to be good enough that it's
    infeasible to make large quantities of spam look as if it were
    associated with originator-ids other than those controlled by the
    spammers.

This seems incongruous to me.

Some principal is going to assign a hash.  There has to be a principal
because the hash has to be digitally signed.  If it's not there's no
point having it for the purpose as described.

This means the message is at least witnessed and there's non-repudiation
to the extent the principal can not deny having handled the message.
That is a security measure.

As to whether any of the content is impeachable, that depends on when
the hash is actually created.  Certainly if it's created after a "Date:"
is added or after the local "Received:" is added, then those fields are
impeachable.  Presumably the rest of the content is opaque to the
creation of the hash so it's content may not be impeachable against the
principal, but with the Date: field you've got a security service.

Did you have something different in mind?

    >> I'm sorry but I just don't see what you have in mind that is worse
    >> than
    >> it is today,
    >
    >     Anyway, the reason my proposal allows originator-ids to be
    > ephemeral is
    >     to make it hard for ordinary people to track messages - I believe
    >     anonymous speech is important - and also out of recognition that
    >     spammers can probably get ephemeral accounts anyway.
    >
    > My point is that we are not that [far] from this already.  To stretch the
    > example to the extreme, the US Government could make a law that ISPs
    > have to make sure they can map a From: email address to an actual
    > person
    > before accepting submission of a message (in the US of course).  Why do
    > we need a special identifier?

    The US Congress might or might not care, but for a variety of reasons
    it would not be appropriate to use From in this way.  There is no
    defined header field that can be used for this purpose without
    conflicting with valid and existing uses of the field.  (Sender would
    have been the right thing, but it's too widely misused now.)

What you say is true but it doesn't refute what I said.  Does this mean
you agree with me?

Jim

Date: Mon, 19 Jan 2004 11:38:28 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Al Costanzo <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

I agree with you about noting the site sending the message.  It should
be a "local" issue as to how that is translated into a local user.  This
gets to the point someone (sorry I forget who) made about requiring some
legal recourse (search warrant or discovery in US terms) for the last
bit of information, thus providing some privacy protection.

And while I agree the DNS is probably a natural place for the
information, I feel obliged to note the problems that exist getting
reverse lookup entries properly configured in the DNS.

Also, the DNS might seem a "secure" location for information, surely we
all realize the DNS is just a "major security incident waiting to
happen."  But then maybe this is the "killer app" for DNSSEC?

Jim

On Sat, 17 Jan 2004, Al Costanzo wrote:

    Date: Sat, 17 Jan 2004 17:08:55 -0500
    From: Al Costanzo <[email-masked]>
    To: James M Galvin <[email-masked]>
    Cc: [email-masked]
    Subject: Re: making mail traceable

    To this point I would say, we need to know the physical location of the
    machine (node) on the Internet sending the email.

    I would say that  DNS is the perfect location for storing this information,
    since unlike mail headers it is more difficult to mung around with by the
    casual user and usually administrated properly by ISPs.

    With the information stored in the DNS IMO, it gives us a second level of
    repudiation to protect us all from spammers and help the US inforce the new
    law.

    Al

    ----- Original Message -----
    From: "James M Galvin" <[email-masked]>
    To: "Al Costanzo" <[email-masked]>
    Cc: <[email-masked]>
    Sent: Saturday, January 17, 2004 3:55 PM
    Subject: Re: making mail traceable

    >
    > To the extent it is "information" we agree we need for email to be
    > traceable and we agree that having it in the DNS is the right place for
    > it, then it would be helpful.  Dave's point was that we should be
    > discussing what information we need and agree on that before we try to
    > agree on where to put it.
    >
    > Jim
    >
    >
    >
    >
    > On Fri, 16 Jan 2004, Al Costanzo wrote:
    >
    >     Date: Fri, 16 Jan 2004 19:42:09 -0500
    >     From: Al Costanzo <[email-masked]>
    >     To: James M Galvin <[email-masked]>, [email-masked]
    >     Subject: Re: making mail traceable
    >
    >
    >
    >     I may be missing the point here but about a year ago I wrote a draft
    for a
    >     DNS RR record to keep track of the physical location of the A or MX
    record
    >     as a "physical postal address" just for this purpose, with the
    intention of
    >     being able to track back the location of SPAM.
    >
    >     With new state laws I thought it was the way to go,  and then use MTA
    use
    >     this information.
    >
    >     Would this not help?  If so I have a copy of the draft I am currently
    >     revising.
    >
    >     Al Costanzo
    >     ----- Original Message -----
    >     From: "James M Galvin" <[email-masked]>
    >     To: "Dave Crocker" <[email-masked]>
    >     Cc: "ietf-822" <[email-masked]>
    >     Sent: Friday, January 16, 2004 5:02 PM
    >     Subject: Re: making mail traceable
    >
    >
    >     >
    >     >
    >     > On Fri, 16 Jan 2004, Dave Crocker wrote:
    >     >
    >     >     >     >  The "Received" header is woefully inadequate for spam
    tracing
    >     >     >
    >     >     >     True.  Then again, so is the rest of the message format
    and mail
    >     >     >     transport.
    >     >     >
    >     >     > I don't agree it's "woefully inadequate."
    >     >
    >     >     When we have some agreement on the information that is needed to
    >     >     facilitate spam tracing, then we can decide whether it is better
    to
    >     >     add it to Received or create a new header.
    >     >
    >     > Absolutely.
    >     >
    >     > And let us not forget some means to validate that the information we
    do
    >     > have or get is accurate and correct.  Or is that what Nathaniel and
    >     > Keith meant by "traceable?"
    >     >
    >     > Jim
    >     >
    >     >
    >
    >
    >
    >

Cc: Keith Moore <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 11:17:58 -0500
To: "Al Costanzo" <[email-masked]>
Sender: [email-masked]

"over my dead body".

Keith

On Jan 19, 2004, at 11:05 AM, Al Costanzo wrote:

> Keith,
>
> I think making mail traceable is the very best thing that the working 
> group
> can do.  People are sick of SPAM and they need a solution.
>
> I already spoke on the reasons why believe this information needs to 
> be in
> DNS in a previous post but I did not go into the detals as to what this
> could do to help the situation.
>
> In the example below, the infomation that would be used by Client's 
> mail
> software or the MTA is the portion before the double quote, the 
> information
> after the doulble quote are for ''non-machine viewing''.
>
> After GL RR identifier you have the two char country code followed by 
> the
> postal code.
>
> GL US.45420.1910
>
> We now have states that have enacted laws concerning spam and now the
> federal government as well.  But even if you put this as a reason to 
> use
> this idea aside, having this information would allow people to 
> configure
> email software they are using to block emails from a country they do 
> not
> wish to receive email from as well as being able to trace where the 
> email
> came from.
>
> In addtion, just as there is no rule forcing an ISP to use this RR, 
> mail
> software could be set to block email from an address that do not 
> contain the
> information in the same way the ISPs block email that is comming from 
> an A
> record that do not have a PTR record now.
>
>
>                 IN      NS      ns.akc.net.
>
> uspring         IN      A       192.188.192.2
>                 IN      MX      5       mail
>                 IN      HINFO   Vax VMS
>                 IN      GL US.45420.1910 "1425 Arbor Avenue, Dayton OH"
> ftp             IN      CNAME   uspring
>
> I will re-post the draft shortly
>
> Al Costanzo
>
> ----- Original Message -----
> From: "Keith Moore" <[email-masked]>
> To: "Graham Klyne" <[email-masked]>
> Cc: "Keith Moore" <[email-masked]>; "ietf-822" <[email-masked]>
> Sent: Monday, January 19, 2004 9:34 AM
> Subject: Re: making mail traceable
>
>
>>
>>>> But do we really want traceability?  Or to put it another way, do we
>>>> really want to put hooks in the mail system that make mass
>>>> surveillance (by governments, or perhaps even by large companies or
>>>> unscrupulous ISPs) that much easier?
>>>
>>> Speaking for myself, I really would like the option to not use
>>> bandwidth receiving email from previously unknown senders unless its
>>> source can be traced.
>>
>> well I suppose you could say that the reason I'm publicly discussing
>> these ideas now is that spam has gotten bad enough that extreme
>> measures seem to be required.
>>
>> also, earlier versions of this scheme made surveillance easier than 
>> the
>> one I just proposed.
>>
>> Keith
>>
>>
>

From: "Al Costanzo" <[email-masked]>
To: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 11:05:32 -0500
Sender: [email-masked]

Keith,

I think making mail traceable is the very best thing that the working group
can do.  People are sick of SPAM and they need a solution.

I already spoke on the reasons why believe this information needs to be in
DNS in a previous post but I did not go into the detals as to what this
could do to help the situation.

In the example below, the infomation that would be used by Client's mail
software or the MTA is the portion before the double quote, the information
after the doulble quote are for ''non-machine viewing''.

After GL RR identifier you have the two char country code followed by the
postal code.

GL US.45420.1910

We now have states that have enacted laws concerning spam and now the
federal government as well.  But even if you put this as a reason to use
this idea aside, having this information would allow people to configure
email software they are using to block emails from a country they do not
wish to receive email from as well as being able to trace where the email
came from.

In addtion, just as there is no rule forcing an ISP to use this RR, mail
software could be set to block email from an address that do not contain the
information in the same way the ISPs block email that is comming from an A
record that do not have a PTR record now.

                IN      NS      ns.akc.net.

uspring         IN      A       192.188.192.2
                IN      MX      5       mail
                IN      HINFO   Vax VMS
                IN      GL US.45420.1910 "1425 Arbor Avenue, Dayton OH"
ftp             IN      CNAME   uspring

I will re-post the draft shortly

Al Costanzo

----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Graham Klyne" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; "ietf-822" <[email-masked]>
Sent: Monday, January 19, 2004 9:34 AM
Subject: Re: making mail traceable

>
> >> But do we really want traceability?  Or to put it another way, do we
> >> really want to put hooks in the mail system that make mass
> >> surveillance (by governments, or perhaps even by large companies or
> >> unscrupulous ISPs) that much easier?
> >
> > Speaking for myself, I really would like the option to not use
> > bandwidth receiving email from previously unknown senders unless its
> > source can be traced.
>
> well I suppose you could say that the reason I'm publicly discussing
> these ideas now is that spam has gotten bad enough that extreme
> measures seem to be required.
>
> also, earlier versions of this scheme made surveillance easier than the
> one I just proposed.
>
> Keith
>
>

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: Will case of non-ascii character be preserved by IDNA?
Date: Mon, 19 Jan 2004 16:11:33 +0100
Lines: 33
Cancel-Lock: sha1:tuSOqwH7bdFUxSSgnGDS5rOBC6s=
Sender: [email-masked]

"Charles Lindsey" <[email-masked]> writes:

> In <[email-masked]> Simon Josefsson <[email-masked]> writes:
>
>>Similar in this case; you invoke IDNA on the IDN and then put the
>>output in the header.
>
>>MÜNCHEN.NET => xn--mnchen-3ya.NET
>
>>Thus, you will send:
>
>>To: MARIA <[email-masked]>
>
>>Which is translated, by the receiver, into:
>
>>To: Maria <[email-masked]>
>
> That doesn't seem right. How did the local-part maria come to be in lower
> case?

A cut and paste error by the author.  Unfortunately, IDNA doesn't
protect against that.

Seriously, the local part and text name is not relevant in the
example.  For completeness, though, the last header should have been:

To: MARIA <[email-masked]>

I hope she is running a spam filter...  (It is with mixed feelings I
have noticed that I get spam for non-ASCII domains.)

Regards,
Simon

To: [email-masked]
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
Subject: Re: Revisiting RFC 2822 grammar
Date: 19 Jan 2004 14:00:00 +0100
Lines: 13
Sender: [email-masked]

Charles Lindsey <[email-masked]> schrieb/wrote:
> So you can a year with 17 digits in it? I don't think that is a good idea.

In the year 999999999999999, it will be a good idea. Don't contribute to  
the Y10P problem. ;-)

But maybe it should be " 4DIGIT / %x31-39 3*DIGIT / obs-year ", i.e.  
years with more than 4 digits must not have leading zeros.

Claus
-- 
http://www.faerber.muc.de

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable 
Date: Mon, 19 Jan 2004 09:34:20 -0500
To: Graham Klyne <[email-masked]>
Sender: [email-masked]

>> But do we really want traceability?  Or to put it another way, do we 
>> really want to put hooks in the mail system that make mass 
>> surveillance (by governments, or perhaps even by large companies or 
>> unscrupulous ISPs) that much easier?
>
> Speaking for myself, I really would like the option to not use 
> bandwidth receiving email from previously unknown senders unless its 
> source can be traced.

well I suppose you could say that the reason I'm publicly discussing 
these ideas now is that spam has gotten bad enough that extreme 
measures seem to be required.

also, earlier versions of this scheme made surveillance easier than the 
one I just proposed.

Keith

Date: Mon, 19 Jan 2004 13:52:11 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Will case of non-ascii character be preserved by IDNA?
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Charles Lindsey <[email-masked]> wrote:

> In <[email-masked]> Simon Josefsson
> <[email-masked]> writes:
> 
> > Thus, you will send:
> >
> > To: MARIA <[email-masked]>
> >
> > Which is translated, by the receiver, into:
> >
> > To: Maria <[email-masked]>
> 
> That doesn't seem right.  How did the local-part maria come to be in
> lower case?

Good catch.  That must have been a typo.  IDNA has nothing to say about
(and no effect on) local parts.

AMC

Date: Mon, 19 Jan 2004 08:14:42 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Sender: [email-masked]

Pete Resnick wrote:

> On 1/18/04 at 8:56 AM -0500, Bruce Lilly wrote:
>
>> According to the latest rfc-index, 1036 has not been obsoleted.   It 
>> might well be your opinion that it should be obsoleted or 
>> reclassified, but that does not affect its official status. You are 
>> of course free to petition for reclassification of RFC 1036 to 
>> historic status (as RFCs 3166, 3638, etc. have done for other RFCs).
>
>
> Oy. If I go ahead and get 1036 moved to historic, can we stop having 
> this discussion?

Well, if you can get 1036's requirements for parsing structure of an 
unstructured
field (Subject) repudiated by any means, yes, we can stop having this 
discussion.

I don't think moving 1036 to historic at this point would be wise, as 
1036 does define
several extension fields.

But as the cognizant WG has not seen fit to repudiate the conflicting 
sections of 1036,
it probably should be done by 2822's successor as the text message 
format is affected.
On the other hand, if you can prompt the cognizant WG to either issue a 
1036 amendment
that specifically addresses the conflicting issues or to issue a draft 
of a 1036 successor that
has a non-zero chance of becoming an RFC, that would work also.  To date 
that WG has
produced neither (though at one point about a year ago there was a draft 
proferred by
Dan Kohn that showed considerable promise).

Date: Mon, 19 Jan 2004 07:53:35 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

Charles Lindsey wrote:

>So you can a year with 17 digits in it? I don't think that is a good idea.
>
>  
>
Pete didn't want to have to revise the document in the year 9999 :-).

Date: Mon, 19 Jan 2004 07:41:46 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Sender: [email-masked]

Russ Allbery wrote:

>I didn't mean to do that.  I am actually arguing the exact opposite by
>saying that RFC 1036 is *also* not a well-defined standard to follow since
>it won't interoperate with Usenet as it exists today.  I am arguing
>exactly that Usenet is simply not well-defined, by RFC 1036 or anything
>else.
>  
>
OK, I see.  Presumably 1036 was reasonably accurate when issued, since 
it described
the operation of a widely used implementation.  The fact that actual use 
has deviated
with no revision to the documentation says more about Usenet than about 
1036.

>You're saying that if you ignore RFC 1036 you have to ignore Usenet.  This
>is obvious nonsense, since there are reams of software written to work
>with Usenet articles that ignore RFC 1036.  My point is twofold:  one
>cannot simply write to something that claims to be a standard because
>sometimes it's badly obsolete, and it is actually possible to write
>interoperable software without a standard (it's just very annoying).
>  
>
I'm saying it's one option. Picking and choosing is another. Making 
things up with
no supporting standard is another  -- and you're right about that being 
annoying.

>>It would be difficult to present a convincing argument that Usenet isn't
>>the least important application that uses the text message
>>format.
>>
[...]

>This is a completely different discussion that I'm not going to have on
>this mailing list.
>
It simply means that given the alternatives (pick and choose, and 
justify the choices made,
or make things up and justify them, it's far simpler to say "Usenet? 
Sorry, not supported."
That isn't to say that that's a great choice; it's the least of evils.

>  I would prefer that this group *did* ignore Usenet
>except for the small and very specific places where exactly following RFC
>2822 makes it very difficult to use with Usenet-like applications, such as
>allowing whitespace in the middle of message IDs.  Other than those very
>specific issues, RFC 2822 work would be best served by ignoring Usenet
>because Usenet would be best served by ceasing its attempts to tread off
>into uncharted territory and moving back to using the same message format
>as e-mail.
>  
>
I agree that there are some issues where 2822 could make some things 
easier for
some uses of the text message format.  And I wholeheartedly agree that 
Usenet
should continue to use the common text message format (indeed, RFC 1036
(section 2) goes to great lengths to indicate that is is in fact the 
same format).

Unfortunately, some provisions of RFC 1036 have an impact on that common
message format, and the Subject field hacks are such provisions.

To: [email-masked]
Xref: clerew local.mime:2946
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Date: Mon, 19 Jan 2004 11:49:13 GMT
Lines: 94
Sender: [email-masked]

In <p06100910bc2d18902e44@[216.43.25.67]> Pete Resnick <[email-masked]> writes:

>On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:

>>Now we come to the obs- syntax, where there are still many 
>>ambiguities. As things stand, sometimes the obs- syntax allows 
>>something that is already in the regular syntax (that is ambiguous). 
>>OTOH, sometimes it does not, and sometimes it allows only a part of 
>>what is in the regular syntax, all of which can be very confusing to 
>>the reader who tries to work out exactly how the obs- syntax differs 
>>from the regular.

>I disagree completely. I think it's much easier for the reader to 
>have some complete pieces in the obs- syntax even if there is 
>redundancy. For example, I think the horrible hoop you have to jump 
>through below for obs-local-part:

Yes, that syntax was pretty ugly, but most of them come out reasonably clean.

But I think the real point is that you need to be consistent. Either the
obs- version of a rule should consistenly include the regular version of
the same rule, or they should be consistently disjoint. Problem is that at
the moment you have neither of these situation.

But whatever is done, I think the text needs to make it clear what policy
has been followed.

>>>obs-phrase      =       word *(word / ("." [CFWS]))
>>
>>OK, but that is not "obsolete". It is intended as an extension to be 
>>allowed sometime in the future on a "MUST accept, SHOULD NOT 
>>generate yet" basis. So please can we rename it as 'extended-phrase' 
>>(which is what I have currently put in Usefor).

>I am not convinced this is worth it. It's explained perfectly well in the text.

Yes, but it is too easily missed if it is hidden away in the obsolete
stuff, which is how I cam to miss it when constructing the Usefor syntax
until Bruce pointed it out.

It is an extension to the preceding standards. It should be in a section
entitled "extensions", or some such.

>>Currently, RFC2822 requires:
>>
>>1. Return-Path
>>2. 1*Received
>>3. *Resent-xxx
>>4. Other headers

>No, it doesn't. Look at the parens and the repeats. It requires:

OK.

>>Here is another example (a real one this time, which some readers of 
>>uk.net.news.management may recognize):
>>
>>Received:  from lon-mail-1.gradwell.net (localhost [127.0.0.1])
>>  by clerew.man.ac.uk (8.11.7+Sun/8.11.7) with ESMTP id i05HCjF01021
>>  for <[email-masked]>; Mon, 5 Jan 2004 17:12:45 GMT
>>Delivered-To:	postmaster@A
>>Received:  (qmail 81124 invoked by uid 800); 5 Jan 2004 12:54:22 -0000
>[...]

>You're right, that's not allowed, and I think that is a bug that 
>needs to be fixed.

>>Now there are all sorts of perfectly genuine "tracing headers" in 
>>there, all added in transit, and all useful.

>So, likely we need optional-field to appear in trace. I think that's 
>the logical answer.

OK. But I still think that the syntax is the wrong place to enforce this.
The number of way of including tracing information is going to increase
(see separate thread currently running), so you need to allow maximum
flexibility for future developments.

Also, my example included a Reply-To header inserted my a mailing list
expander in the midst of the tracings. Was that right or wrong?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2950
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: cryptographically verifiable fields
Date: Mon, 19 Jan 2004 12:08:48 GMT
Lines: 41
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Keith Moore wrote:

>> That, and there is a semantic difference between a signed message and 
>> a cryptographically verifiable trace field in a message.

>Could somebody outline a process whereby a single field or group of 
>fields in a
>message could be signed, with the following conditions:
>1. the mechanism is robust w.r.t. common types of message munging 
>(reordered fields,
>    possible dropping of fields (obviously, let's assume that the field 
>that is signed isn't
>    dropped), addition of trailing whitespace, etc.)
>2. the mechanism is not subject to replay attacks (e.g. copying the 
>signed field from one
>    message to another)

Usefor did some work on this (but for other reasons). However, it was
decided it was a step too far for the current draft, but a possibility for
a later 'security' document.

The internet-draft has expired now, but you can still find it on
   <http://www.landfiled.com/usefor/drafts/draft-lindsey-usefor-signed-01.txt>

It included a very elaborate canonicalization. Even so, I would write it
differently now. And worse things have happened since I wrote it, for
example how do you canonicalize IDNA domain names, as UTF-8 or as
punycode?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2948
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Date: Mon, 19 Jan 2004 12:00:24 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Pete Resnick wrote:

>> I agree. These should be removed.

>IMO, Subject should be unstructured, period.  Unfortunately, RFC 1036 
>introduced two
>hacks:
>1. "Re: " (which it got wrong!), which *requires* certain actions when 
>encountered, i.e. it is
>    effectively part of the syntax.  See RFC 1036 sections 2.1.4, 2.2.5.
>2. "cmsg", which *requires* certain actions when encountered.  See RFC 
>1035 sections 2.2.6
>    and 3 (including subsections).

>This is the same "Subject" field described in RFCs 822 and 2822.

>Now, if an RFC 2822 successor were to repudiate those requirements 
>spelled out in RFC 1036,
>that would be fine.

No! If they are to be repudiated, then it is for an RFC 1036 successor to
do so. Indeed, Usefor is taking care of both of these (though the exact
form of such care is not in concrete yet).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2947
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (obs-utext)
Date: Mon, 19 Jan 2004 11:56:30 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Further note that this is obs-syntax, which is not for generation of new 
>messages.

>Also note that ASCII NUL is included in obs-char.

Note also that NULs and naked CR and LF in headers will not be transported
correctly by many transports, and there is nothing corresponding to CTE
Q-P to tunnel them through as there is in the case of bodies.

Which also raises the issue of which parts of the obs-syntax are there
because there exist (or once existed) agents that routinely generated such
things, and which parts are there because they were strictly legal under
RFC 822, but nobody ever actually generated them (except those
participating in obfuscated header contests)?

I have this feeling that we are requiring agents to recognize obs-
features that have never ever been seen in the wild.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2949
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Will case of non-ascii character be preserved by IDNA?
Date: Mon, 19 Jan 2004 12:02:03 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>Similar in this case; you invoke IDNA on the IDN and then put the
>output in the header.

>MÜNCHEN.NET => xn--mnchen-3ya.NET

>Thus, you will send:

>To: MARIA <[email-masked]>

>Which is translated, by the receiver, into:

>To: Maria <[email-masked]>

That doesn't seem right. How did the local-part maria come to be in lower
case?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2945
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Date: Mon, 19 Jan 2004 11:36:38 GMT
Lines: 21
Sender: [email-masked]

In <p0610090fbc2d1655a86d@[216.43.25.67]> Pete Resnick <[email-masked]> writes:

>On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:

>>3. The date syntax is reordered, and four-digit years seem to have 
>>disappeared from the obsolete syntax.

>Nope. 2*DIGIT means 2 or more digits, which is correct.

So you can a year with 17 digits in it? I don't think that is a good idea.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 19 Jan 2004 07:10:59 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: cryptographically verifiable fields
Sender: [email-masked]

Keith Moore wrote:

[...]

> I don't think it would be easy to work out the details; but I do 
> believe it's feasible.

OK, thanks.  I'm not sure that messing with address local-parts is the 
right thing to do
(local-parts can be case-sensitive), but I get the idea.  Sounds quite 
complex, to the
point that I have some doubts about how well different implementations 
would work.

I think there are also a few missing bits and pieces which are necessary 
to make
such verification useful from a practical (i.e. minimizing manual setup 
and intervention)
point of view, but that's a separate issue.

Thanks again.

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

Date: Fri, 16 Jan 2004 18:02:49 +0000
To: Keith Moore <[email-masked]>
From: Graham Klyne <[email-masked]>
Subject: Re: making mail traceable 
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

At 11:47 16/01/04 -0500, Keith Moore wrote:
>But do we really want traceability?  Or to put it another way, do we 
>really want to put hooks in the mail system that make mass surveillance 
>(by governments, or perhaps even by large companies or unscrupulous ISPs) 
>that much easier?

Speaking for myself, I really would like the option to not use bandwidth 
receiving email from previously unknown senders unless its source can be 
traced.

I think the damage being done by spam flooding my mailbox is somewhat 
greater than the damage caused by other agencies monitoring its 
content.  I'm not inclined to trust my secrets to email, in any case.

#g

------------
Graham Klyne
For email:
http://www.ninebynine.org/#Contact

Date: Sun, 18 Jan 2004 16:35:02 -0600
To: ietf-822 <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Cc: Bruce Lilly <[email-masked]>
Sender: [email-masked]

On 1/18/04 at 8:56 AM -0500, Bruce Lilly wrote:

>According to the latest rfc-index, 1036 has not been obsoleted.   It 
>might well be your opinion that it should be obsoleted or 
>reclassified, but that does not affect its official status. You are 
>of course free to petition for reclassification of RFC 1036 to 
>historic status (as RFCs 3166, 3638, etc. have done for other RFCs).

Oy. If I go ahead and get 1036 moved to historic, can we stop having 
this discussion?

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

To: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sun, 18 Jan 2004 13:10:13 -0800
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> And pretending that there is a well-defined "current practice" does no
> favors either.

I didn't mean to do that.  I am actually arguing the exact opposite by
saying that RFC 1036 is *also* not a well-defined standard to follow since
it won't interoperate with Usenet as it exists today.  I am arguing
exactly that Usenet is simply not well-defined, by RFC 1036 or anything
else.

>>> a) ignore RFC 1036, which means ignoring Usenet, since 1036 is THE RFC
>>>   dealing with Usenet message format

>> This is simply nonsensical garbage.  The world does not magically stop
>> working when you don't have an RFC to work from.

> Did I say the world would stop working?  Did I say that there isn't an
> applicable RFC?  News flash: Usenet is not the world.  In the grand
> scheme of things, it isn't even close to being important.

You missed my point.

You're saying that if you ignore RFC 1036 you have to ignore Usenet.  This
is obvious nonsense, since there are reams of software written to work
with Usenet articles that ignore RFC 1036.  My point is twofold:  one
cannot simply write to something that claims to be a standard because
sometimes it's badly obsolete, and it is actually possible to write
interoperable software without a standard (it's just very annoying).

> It would be difficult to present a convincing argument that Usenet isn't
> the least important application that uses the text message
> format. Certainly email, which is legitimately used (i.e. excluding
> spam) to a much larger extent than Usenet, and which has become an
> essential tool for commerce, is far more important than Usenet.  Voice
> mail, fax, EDI, and even SIP could be argued as more important than
> Usenet.  Not to mention the fact that Usenet's historically appallingly
> low signal-to-noise ratio (which seems to keep falling to new lows) is
> making Usenet largely irrelevant, it having been supplanted by weblogs
> and the like to a large extent.

This is a completely different discussion that I'm not going to have on
this mailing list.  I would prefer that this group *did* ignore Usenet
except for the small and very specific places where exactly following RFC
2822 makes it very difficult to use with Usenet-like applications, such as
allowing whitespace in the middle of message IDs.  Other than those very
specific issues, RFC 2822 work would be best served by ignoring Usenet
because Usenet would be best served by ceasing its attempts to tread off
into uncharted territory and moving back to using the same message format
as e-mail.

> That amounts to precisely the same thing as c, viz. picking and choosing
> bits and pieces, and has the predictable and observable consequences
> noted, viz. incompatibilities and interoperability problems.

Okay, fair enough.

> Writing code which attempts to "work with the Usenet messages in the
> wild" amounts to picking and choosing from a plethora of incompatible
> options;

Yes.  And that's how you have to write Usenet software right now.

> Where are all of these supposedly well-defined issues of "current
> practice" actually defined?

I did not mean to claim they were well-defined.

> Has the cognizant WG come to rapid agreement on any of those issues?

No.  The cognizant WG is completely disfunctional.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sun, 18 Jan 2004 13:00:22 -0800
Sender: [email-masked]

Arnt Gulbrandsen <[email-masked]> writes:
> Russ Allbery writes:

>> This is simply nonsensical garbage. The world does not magically stop
>> working when you don't have an RFC to work from.

> There is no shortage of implementors who can be misled into thinking that
> 1036 is the thing to support.

I know.  Which is why it would be really good to replace it.  But when
they discover that their software doesn't interoperate, they generally
shift to supporting what's actually present in the wild.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Cc: Keith Moore <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: cryptographically verifiable fields
Date: Sun, 18 Jan 2004 15:41:14 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

>> That, and there is a semantic difference between a signed message and 
>> a cryptographically verifiable trace field in a message.
>
> Could somebody outline a process whereby a single field or group of 
> fields in a
> message could be signed, with the following conditions:
> 1. the mechanism is robust w.r.t. common types of message munging 
> (reordered fields,
>    possible dropping of fields (obviously, let's assume that the field 
> that is signed isn't
>    dropped), addition of trailing whitespace, etc.)

- You only include specific fields in the hash, fields that are likely 
to be maintained end-to-end.
- You hash the fields in a predetermined order.
- Before hashing each of those fields, you delete preceding and 
trailing whitespace, and you replace all internal whitespace (including 
CRLF-space) with a single space.  you canonicalize the field-name (say, 
all lower case letters).
- Some fields might have specific rules for canonicalization before 
hashing.   e.g. For fields containing addresses you would remove all 
phrases and comments, leaving only the addresses; you would then 
canonicalize each of the addresses (e.g. any local part that isn't a 
quoted string would be enclosed in double quotes, contents of all local 
parts would be lower-cased, all domains would be lower-cased;  < and > 
enclosing addresses would be removed); the resulting addresses would be 
sorted; and any white space next to a special would be removed.  (I'm 
not sure yet whether I think address fields should be included in the 
hash, but this is how you might do it if such fields were included)

I don't think it would be easy to work out the details; but I do 
believe it's feasible.

A few systems would still mung headers to the point that the 
originator-id verification would fail.  The consequences would mostly 
be these: For systems that munged outgoing mail, their recipients' mail 
would not be verifiable. For systems munging incoming mail, their 
recipients would not be able to verify originator-ids on mail that they 
received or on mail that they forwarded to other systems.  In either 
case there would be an incentive to fix the implementations that munged 
mail.

it might also be desirable to impose some constraints on messages that 
are originated with an originator-id, such as limiting the permissible 
variation on the format of address fields, date fields, etc.

Note again, the purpose isn't to prove that the sender sent the exact 
message that the recipient received.  The purpose is to make it 
difficult for the sender to credibly say the "I didn't send anything 
resembling that message"  for large numbers of messages.  A sender 
should not be penalized for a small number of messages considered 
offensive to recipients regardless of whether the sender claims that he 
didn't send them.

> 2. the mechanism is not subject to replay attacks (e.g. copying the 
> signed field from one
>    message to another)

You include the message-id and/or the message body in the fields to be 
hashed.

Date: Sun, 18 Jan 2004 08:49:42 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Bruce Lilly <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Bruce Lilly wrote:

> >
> > Here's another take on the subject (I hope it doesn't open up several
> > cans
> > of worms, but here goes...)
> >
> > X- is used as an indicator for experimental or private-use tags in
> > many places
> > in Internet message protocols other than field names. For example:

> [...]

> > So what's so abhorrent about top-level X- fields (RFC 822) that
> > doesn't also apply to
> > charsets, media types, language tags, etc.?

> I didn't intend that to stifle discussion, but I haven't seen any
> responses from the anti-X-
> community.  I did intend it to be a moderately strong case in support of
> X- fields (by
> way of asking what's so bad about them that differs from charsets,
> language tags, etc.).

In the case of media types, x- values have proved to be a very poor idea, in
that their use has in some cases prevented registration of types that really
needed to be properly registered. And note that media types are, generally
speaking, something that implementations support through the use of a
configurable table, which means the adverse impact on interoperability of
multiple labels is much,  much lower than something that tends to be embedded
in program code. However, abandoning the x- distinction in the case of media
types is highly problematic at this point because of the political issues
surrounding the registration process and the history of its development. Header
fields do not have a registry at present and hence do not have this problem.

I haven't seen significant use of x- types in either charsets or language tags.
In the case of charsets the early registration of a massive number of charsets
is likely what prevented this from  happening, along with the willingness,
right or wrong, to simply use non-x- names without registration. Language tags
started out with a reasonably comprehensive set of initial registrations and a
reasonably timely and fair review and approval process. (Of course there's also
the fact that defining a new language is a much bigger deal than coming up with
new item for most of of the other registries we have.)

I can argue the effects of x- tags in the content-transfer-encoding
space either way. On the one hand, the ability to use various x- tags
for various uuencode variants made it possible to use uuencode as
transitional encoding on the way to getting base64 support in place. On
the other hand, the lack of a standarized label did cause fairly
nontrivial interoperability problems along the way. Does the end justify
the means?

The bottom line is that the success of x- fields in other places has anywhere
from disasterous at worst to mixed at best. And to a large extent the lack of
harm in many spaces has come from lack of use. So if you're looking for support
for keeping the X- distinction from the successful use of X- in other contexts,
I'm afraid it is not forthcoming.

				Ned

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Date: Sun, 18 Jan 2004 18:05:49 +0100
Sender: [email-masked]

Russ Allbery writes:
> Bruce Lilly <[email-masked]> writes:
>>  a) ignore RFC 1036, which means ignoring Usenet, since 1036 is THE RFC
>>     dealing with Usenet message format
>
> This is simply nonsensical garbage. The world does not magically stop 
> working when you don't have an RFC to work from.

There is no shortage of implementors who can be misled into thinking 
that 1036 is the thing to support.

Arnt

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Date: Sun, 18 Jan 2004 18:02:02 +0100
Sender: [email-masked]

Russ Allbery writes:
> Bruce Lilly <[email-masked]> writes:
>>  Now, if an RFC 2822 successor were to repudiate those requirements 
>>  spelled out in RFC 1036, that would be fine.
>
> RFC 1036 is not a standards track document and therefore should not 
> need to be explicitly repudiated by another document.

I think this is a bit too legalistic. RFC 1036 is the best available 
document for a widely used and well-known system, which gives it some 
standing in practice, and which makes a textual repudiation reasonable 
(even if not strictly necessary).

IMO it would be better for an RFC 1036 successor to repudiate this, but 
I've heard they're waiting to get RFC number 10036, so that won't 
happen soon.

Arnt

Date: Sun, 18 Jan 2004 11:21:01 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

Bruce Lilly wrote:

>
> Here's another take on the subject (I hope it doesn't open up several 
> cans
> of worms, but here goes...)
>
> X- is used as an indicator for experimental or private-use tags in 
> many places
> in Internet message protocols other than field names. For example: 

[...]

> So what's so abhorrent about top-level X- fields (RFC 822) that 
> doesn't also apply to
> charsets, media types, language tags, etc.?

I didn't intend that to stifle discussion, but I haven't seen any 
responses from the anti-X-
community.  I did intend it to be a moderately strong case in support of 
X- fields (by
way of asking what's so bad about them that differs from charsets, 
language tags, etc.).
I don't want to read too much into that dearth of responses -- any comments?

Date: Sun, 18 Jan 2004 09:44:53 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: IETF RFC-822 list <[email-masked]>
Organization: Bruce Lilly
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (quoted-pair)
Sender: [email-masked]

Adam M. Costello wrote:

>Bruce Lilly <[email-masked]> wrote:
>
>  
>
>>Adam M. Costello wrote:
>>
>>    
>>
>>>Consider the following example:
>>>
>>>From: "foo\
>>>bar" <blah@example>
>>>      
>>>
>
>You have misquoted me (or more likely, your MUA has).  The example was:
>
>From: "foo\
> bar" <blah@example>
>
>The space before "bar" is crucial, because the alternative (without the
>space) is another equally interesting example:
>  
>
Yes, my MUA (Mozilla 1.6) did it.

>In fact, when you presented your own example, I thought you omitted the
>space deliberately:
>
>From: "Foo Bar" <"foo\
>bar"@example>
>
>  
>
I did. It was intended that the CRLF was part of the local-part 
(ignoring whether
or not that was sensible or advisable). And I didn't want there to be 
confusion about
whether it was CRLF or CRLFSP.

>I don't expect us to be able to settle this.  I think RFC 822 is not
>self-consistent on this issue.
>
>  
>
Agreed, and that's an indication in favor of deprecation.

>>If backslash-escaped CR and LF (ignoring NUL for the moment) were
>>permitted, one could have:
>>
>>From: "foo\CR\LFbar" <blah@example>
>>
>>etc., which ought to present no problems;
>>    
>>
>
>Until it gets converted to the local line-ending conventions.  Your
>example contains all three possibilities: CR not followed by LF, LF not
>preceeded by CR, and CRLF (terminating the field).  Imagine saving this
>message to an mbox file on a Unix machine, where lines are terminated
>by LF.  How will you do it?  Normally CRLF gets translated to LF, but
>that's not reversible if the input already contains LF not preceeded by
>CR.
>  
>
In this case, there's no reason to unescape before saving, and therefore 
there's no
CRLF (as opposed to \CR\LF) to convert.

Conversion of line endings is tricky business, and I'd recommend against 
it except when
saving an attachment of text type.  I suspect there would be problems on 
many systems
saving a message with structure:

multipart/mixed, content-transfer-encoding 8bit
    application/octet-stream, content-transfer-encoding 8bit
        some binary content including 0x0a 0x0d 0x0a 0x0d

Assuming line endings must be converted, doing it correctly requires 
parsing the
MIME structure -- the embedded 0x0d 0x0a in the binary content must not be
altered.  In the example if 0x0d 0x0a (i.e. CRLF) is converted to a lone 
0x0a, the
four-octet sequence becomes the three-octet sequence 0x0a 0x0a 0x0d.  As you
noted, that's not a reversible transformation; it's likely to yield the 
five-octet
sequence 0x0d 0x0a 0x0d 0x0a 0x0d.

>Other problems:  How would this field display?  Could it be cut and
>pasted?
>
>  
>
Display is one issue. Cut and paste should be verbatim, i.e. using the 
on-the-wire
representation.

>I think any sort of control characters in header fields, other than
>CRLF (as a unit) and maybe TAB, is asking for headaches.  Even TAB is
>somewhat troublesome.
>  
>
True, including security implications for some control characters.  
Probably the
safest for display purposes is to use a textual representation for 
control characters,
possibly with some form of highlighting to avoid confusion with literal 
text.

>P.S. The tendency of your MUA to drop spaces at the beginnings of lines
>is probably related to its use (or misuse) of format=flowed.
>
>I notice that each of your paragraphs consists of multiple "paragraphs"
>in the
>format=flowed sense, so that they don't actually flow, but instead end
>up looking
>like this paragraph.
>  
>
format=flowed is another issue, regarding which I haven't yet added my 2 
cents worth.
So here it is, FWIW.  Format=flowed is far too complex for text/plain -- 
it amounts to
a markup language (a minimalist one, perhaps, but markup nevertheless).  
Implementation
differences (and/or lack of implementation support) is probably 
indicative of the complexity
and incompatibility with text/plain.  IMO it would be best for it to 
have its own subtype,
just as other markup languages do (e.g. text/html, text/richtext).

Date: Sun, 18 Jan 2004 08:56:18 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
CC: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Sender: [email-masked]

Russ Allbery wrote:

>Bruce Lilly <[email-masked]> writes:
>
>  
>
>>RFC 1036's title is "Standard for Interchange of USENET Messages".
>>While it has no standing as an Internet Standard it certainly claims to
>>be a standard, and moreover it is current and is THE RFC addressing
>>message format w.r.t. Usenet (RFC 850 having been superseded by 1036,
>>and 1036 not having been amended or superseded).
>>    
>>
>
>You're simply wrong about this.
>  
>
Perhaps, but it will take more than a bald assertion to convince me.

>There is no standard document governing the message format of Usenet.
>  
>
RFC 1036 says otherwise.

>There is one obsolete informational RFC that does not agree with current
>practice,
>
According to the latest rfc-index, 1036 has not been obsoleted.   It 
might well be
your opinion that it should be obsoleted or reclassified, but that does 
not affect its
official status. You are of course free to petition for reclassification 
of RFC 1036
to historic status (as RFCs 3166, 3638, etc. have done for other RFCs).

> and an effort to write a standard which is completely stalled.
>  
>
It seems to be stalled for lack of agreement on just what comprises 
"current practice",
not to mention what *should* comprise best practice.  That and the 
mantra that "the
draft just needs a few minor tweaks" that has been repeated for 
literally years.  Unless
I've missed something (entirely possible), in spite of new leadership 
(as of about a year
ago) and a planned rechartering of the WG that was set to work on a 
successor to 1036
some eight years ago in order to deal with "urgent issues", that WG in 
fact has no new
charter.  That's not a good sign. I share your frustration, and I wish 
1036 had long been
obsoleted by an update that incorporated those things upon which the WG 
could agree
(as I had suggested quite some time ago).  But it hasn't happened, and 
the sad fact is
that 1036 is still the current standard.

>Presenting the obsolete informational RFC as a standard because it
>contains the word "standard" in the title and no one has yet written
>anything better is not doing anyone any favors.
>  
>
And pretending that there is a well-defined "current practice" does no 
favors either.

>>a) ignore RFC 1036, which means ignoring Usenet, since 1036 is THE RFC
>>   dealing with Usenet message format
>>    
>>
>
>This is simply nonsensical garbage.  The world does not magically stop
>working when you don't have an RFC to work from.
>  
>
Did I say the world would stop working?  Did I say that there isn't an 
applicable RFC?
News flash: Usenet is not the world.  In the grand scheme of things, it 
isn't even close
to being important.  It would be difficult to present a convincing 
argument that Usenet
isn't the least important application that uses the text message format. 
Certainly email,
which is legitimately used (i.e. excluding spam) to a much larger extent 
than Usenet, and
which has become an essential tool for commerce, is far more important 
than Usenet.
Voice mail, fax, EDI, and even SIP could be argued as more important 
than Usenet.  Not to
mention the fact that Usenet's historically appallingly low 
signal-to-noise ratio (which
seems to keep falling to new lows) is making Usenet largely irrelevant, 
it having been
supplanted by weblogs and the like to a large extent.

>c) pick and choose bits and pieces from the various standards
>  
>
>
>It's incomprehensible to me how you can omit the choice that essentially
>all Usenet implementors actually take, which is:
>
> d) write code which works with the Usenet messages in the wild and
>    follows commonly accepted best practice, using RFC 1036 as one of many
>    guides but not giving it all that much weight
>
>  
>
That amounts to precisely the same thing as c, viz. picking and choosing 
bits and pieces,
and has the predictable and observable consequences noted, viz. 
incompatibilities and
interoperability problems.  Writing code which attempts to "work with 
the Usenet
messages in the wild" amounts to picking and choosing from a plethora of 
incompatible
options; is a header field restricted to the set of octets prescribed by 
RFCs 1036, 850, 822,
and 2822, or is it "anything goes"? Are charsets other than the default 
properly tagged
per the MIME RFCs or left up to the recipient to guess?  And exactly 
which charset *is*
the default?  Does the existence of the magic incantation " Re: " at the 
start of a
Subject field body define a message as a reply, or is it the presence of 
a References field
that defines a message as a reply?    Where are all of these supposedly 
well-defined issues
of "current practice" actually defined?  Has the cognizant WG come to 
rapid agreement
on any of those issues?

Date: Sun, 18 Jan 2004 13:29:57 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (quoted-pair)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> Adam M. Costello wrote:
> 
> >Consider the following example:
> >
> >From: "foo\
> >bar" <blah@example>

You have misquoted me (or more likely, your MUA has).  The example was:

From: "foo\
 bar" <blah@example>

The space before "bar" is crucial, because the alternative (without the
space) is another equally interesting example:

From: "foo\
bar" <blah@example>

In fact, when you presented your own example, I thought you omitted the
space deliberately:

From: "Foo Bar" <"foo\
bar"@example>

But now I suspect that your MUA ate the space, and what you originally
intended was:

From: "Foo Bar" <"foo\
 bar"@example>

But in any case both are worth considering.

> I'd interpret that example as a field which is not folded (the
> backslash-escaped CRLF is not line folding because it is escaped) in
> which the quoted-string contains a CRLF.

We can certainly find evidence in RFC 822 to support that
interpretation, but we can also find evidence to doubt it.  As I
mentioned, 3.4.8 says "Each header field may be represented on exactly
one line consisting of the name of the field and its body, and
terminated by a CRLF; this is what the parser sees."  The proposed
interpretation has the parser seeing the field as two lines (because
the parser sees the line break inside the quoted-string).  Also, what
could 3.4.5 be talking about when it says "Quoted CRLFs (i.e., a
backslash followed by a CR followed by a LF) are also subject to rules
of folding"?

Finally, is it valid not to have a space after \CRLF?  As in:

From: "Foo Bar" <"foo\
bar"@example>

The grammar allows it, and if we go with the proposed interpretation
that \CRLF is not an instance of folding, then what would require a
space after the \CRLF?

There is a related question for the other example:

From: "Foo Bar" <"foo\
 bar"@example>

Is the space part of the local part?  3.4.5 says "Stripping off the
first following LWSP-char is also appropriate when parsing quoted CRLFs"
(the "also" means "similar to the case of unquoted CRLFs").  But why?
Why put the space in only to have it stripped out again?  The only
reasonable explanation is that the space is required (this gets back to
the previous question).  Only if it's required do you need a rule about
stripping it, so that you can have a quoted-string whose meaning is
fooCRLFbar with no space.

We now have two arguments that space is required after \CRLF, but that
in turn argues that \ CR LF LWSP-char is indeed some sort of folding.
Maybe it's a fold that cannot be unfolded.  Except there's still that
pesky statement in 3.4.8 that "Each header field may be represented on
exactly one line".

I don't expect us to be able to settle this.  I think RFC 822 is not
self-consistent on this issue.

> If backslash-escaped CR and LF (ignoring NUL for the moment) were
> permitted, one could have:
>
> From: "foo\CR\LFbar" <blah@example>
>
> etc., which ought to present no problems;

Until it gets converted to the local line-ending conventions.  Your
example contains all three possibilities: CR not followed by LF, LF not
preceeded by CR, and CRLF (terminating the field).  Imagine saving this
message to an mbox file on a Unix machine, where lines are terminated
by LF.  How will you do it?  Normally CRLF gets translated to LF, but
that's not reversible if the input already contains LF not preceeded by
CR.

Other problems:  How would this field display?  Could it be cut and
pasted?

I think any sort of control characters in header fields, other than
CRLF (as a unit) and maybe TAB, is asking for headaches.  Even TAB is
somewhat troublesome.

> But let's clearly document the changes from 822!

Agreed.

AMC

P.S. The tendency of your MUA to drop spaces at the beginnings of lines
is probably related to its use (or misuse) of format=flowed.

I notice that each of your paragraphs consists of multiple "paragraphs"
in the
format=flowed sense, so that they don't actually flow, but instead end
up looking
like this paragraph.

To: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sun, 18 Jan 2004 04:14:18 -0800
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> RFC 1036's title is "Standard for Interchange of USENET Messages".
> While it has no standing as an Internet Standard it certainly claims to
> be a standard, and moreover it is current and is THE RFC addressing
> message format w.r.t. Usenet (RFC 850 having been superseded by 1036,
> and 1036 not having been amended or superseded).

You're simply wrong about this.

There is no standard document governing the message format of Usenet.
There is one obsolete informational RFC that does not agree with current
practice, and an effort to write a standard which is completely stalled.
Presenting the obsolete informational RFC as a standard because it
contains the word "standard" in the title and no one has yet written
anything better is not doing anyone any favors.

> As a self-proclaimed standard for message format, and with a number of
> discrepancies with other message format RFCs, 1036 presents a problem
> for implementors; an implementor apparently has the following choices:

> a) ignore RFC 1036, which means ignoring Usenet, since 1036 is THE RFC
>    dealing with Usenet message format

This is simply nonsensical garbage.  The world does not magically stop
working when you don't have an RFC to work from.

> b) take 1036 into account, which imposes structure on the supposedly
>    unstructured Subject field

> c) pick and choose bits and pieces from the various standards

It's incomprehensible to me how you can omit the choice that essentially
all Usenet implementors actually take, which is:

 d) write code which works with the Usenet messages in the wild and
    follows commonly accepted best practice, using RFC 1036 as one of many
    guides but not giving it all that much weight

Whether you think this is a viable choice or not, it clearly is because
this is what people actually do.  Reality is at odds with your opinions.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sun, 18 Jan 2004 06:45:13 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Sender: [email-masked]

Russ Allbery wrote:

>RFC 1036 is not a standard and this is not best practice for Usenet
>software.  Please do not include this in the grammar.
>  
>
RFC 1036's title is "Standard for Interchange of USENET Messages".  While it
has no standing as an Internet Standard it certainly claims to be a 
standard, and
moreover it is current and is THE RFC addressing message format w.r.t. 
Usenet
(RFC 850 having been superseded by 1036, and 1036 not having been amended or
superseded).

I agree that a number of things in 1036 are questionable practice; 
Subject field
hacks are one of those questionable practices.  But questionable or not, 
that's
what 1036 requires.

>RFC 1036 is not a standards track document and therefore should not need
>to be explicitly repudiated by another document.
>  
>
As a self-proclaimed standard for message format, and with a number of 
discrepancies
with other message format RFCs, 1036 presents a problem for implementors; an
implementor apparently has the following choices:
a) ignore RFC 1036, which means ignoring Usenet, since 1036 is THE RFC 
dealing
    with Usenet message format
b) take 1036 into account, which imposes structure on the supposedly 
unstructured
     Subject field
c) pick and choose bits and pieces from the various standards

Option a is certainly a viable choice.  Option b leads to the present 
discussion.
Option c can lead to incompatibilities and lack of interoperability; it 
is the option of
last resort.

Repudiation of the questionable items may provide an out for the hapless 
implementor.
Maybe not [old messages still exist, and may need to be parsed in 
accordance with old
standards].

One of the lessons to be learned is that it can be extremely difficult 
to recover from
poor design decisions.

Date: Sun, 18 Jan 2004 05:52:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: IETF RFC-822 list <[email-masked]>
Organization: Bruce Lilly
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (quoted-pair)
Sender: [email-masked]

Adam M. Costello wrote:

>Bruce Lilly <[email-masked]> wrote:
>
>  
>
>>Moreover, the semantics of \CRLF as given in RFC 822 are quite
>>different from \CR followed by a lone LF.
>>    
>>
>
>I'm having trouble making sense of \CRLF in RFC 822.
>
>Consider the following example:
>
>From: "foo\
> bar" <blah@example>
>
>This could not have been created by starting with a one-line field
>and then folding it, because 3.1.1 says that folding may happen
>"wherever there may be linear-white-space (NOT simply LWSP-chars)".
>Linear-white-space is allowed in qtext, but not in quoted-pair, so there
>would be no way for the folding process to wedge a CRLF between the
>backslash and the next CHAR.
>
>The only way to parse this field, according to the grammar, is as:
>
>"From" ":" <"> CHAR CHAR CHAR quoted-pair CHAR CHAR CHAR CHAR CHAR <"> ...
> From   :   "   f    o    o       \CR      LF   SP   b    a    r    "  ...
>
>There is no way to parse the field in a way that involves the
>linear-white-space token, and no way to parse it in a way that involves
>the CRLF token.
>
>Question:  Can this field be unfolded?
>  
>
I'd interpret that example as a field which is not folded (the 
backslash-escaped CRLF
is not line folding because it is escaped) in which the quoted-string 
contains a CRLF. That
CRLF is escaped so that it is preserved for the application (an 
unescaped CRLF would be
part of line folding, which would not be visible to the application).  
Consider as an
alternative example:

From: "Foo Bar" <"foo\
bar"@example>

[ignoring for the moment whether or not a CRLF in a local-part is either 
sensible or
advisable]

>This all seems like a big mess that should be deprecated.  And indeed,
>in RFC 2822, lone CR, lone LF, and \CR are all relegated to obsolete
>syntax.  I'm thinking that was a good move.
>  
>
Perhaps it should be deprecated; on the other hand, 2822 provides no 
mechanism for
passing CR, LF, or NUL at the application layer, as quoting is not 
permitted for
those octets (except via obs- constructs, which may not be used for 
message generation).
If backslash-escaped CR and LF (ignoring NUL for the moment) were 
permitted, one
could have:

From: "foo\CR\LFbar" <blah@example>

etc., which ought to present no problems; there is no explicit CRLF pair 
on the wire, so
folding/unfolding isn't an issue, the application layer still gets the 
CR and LF octets when
parsing the quoted string, and it is backwards-compatible (i.e. that was 
legal in 822 and
semantics are unchanged).

But let's clearly document the changes from 822!

>As for the obsolete grammar, parsing \CRLF as \CR followed by LF is
>consistent with the 822 grammar, even if it doesn't seem to jibe with
>the phrase "quoted CRLF" in the prose.
>
Maybe; handling of WSP after \CRLF would seem to be somewhat different 
(I agree with
you that 822 isn't quite clear about that)

Date: Sun, 18 Jan 2004 05:15:34 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
CC: ietf-822 <[email-masked]>
Subject: cryptographically verifiable fields
Sender: [email-masked]

Keith Moore wrote:

> That, and there is a semantic difference between a signed message and 
> a cryptographically verifiable trace field in a message.

Could somebody outline a process whereby a single field or group of 
fields in a
message could be signed, with the following conditions:
1. the mechanism is robust w.r.t. common types of message munging 
(reordered fields,
    possible dropping of fields (obviously, let's assume that the field 
that is signed isn't
    dropped), addition of trailing whitespace, etc.)
2. the mechanism is not subject to replay attacks (e.g. copying the 
signed field from one
    message to another)

I believe that S/MIME and PGP/MIME signed messages are robust w.r.t. 
those criteria, since
the signed message is itself transfer encoded (if necessary) and 
encapsulated via MIME; if the
MIME wrapper's header fields are munged, the signed message may still be 
valid -- it is
protected from transport issues via the wrapper and encoding.  But I 
don't see how one can
do the same for a single header field that is not subject to a replay 
attack.

Date: Sun, 18 Jan 2004 07:37:22 +0000
From: "Adam M. Costello" <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (quoted-pair)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> Moreover, the semantics of \CRLF as given in RFC 822 are quite
> different from \CR followed by a lone LF.

I'm having trouble making sense of \CRLF in RFC 822.

Consider the following example:

From: "foo\
 bar" <blah@example>

This could not have been created by starting with a one-line field
and then folding it, because 3.1.1 says that folding may happen
"wherever there may be linear-white-space (NOT simply LWSP-chars)".
Linear-white-space is allowed in qtext, but not in quoted-pair, so there
would be no way for the folding process to wedge a CRLF between the
backslash and the next CHAR.

The only way to parse this field, according to the grammar, is as:

"From" ":" <"> CHAR CHAR CHAR quoted-pair CHAR CHAR CHAR CHAR CHAR <"> ...
 From   :   "   f    o    o       \CR      LF   SP   b    a    r    "  ...

There is no way to parse the field in a way that involves the
linear-white-space token, and no way to parse it in a way that involves
the CRLF token.

Question:  Can this field be unfolded?

3.1.1 says "Unfolding is accomplished by regarding CRLF immediately
followed by a LWSP-char as equivalent to the LWSP-char."  But the CRLF
token is not present in the parsing of the field, so we cannot unfold
it.

On the other hand, 3.1.2 says "The field-body may be composed of any
ASCII characters, except CR or LF.  (While CR and/or LF may be present
in the actual text, they are removed by the action of unfolding the
field.)"  Therefore, we ought to be able to remove the CR and the LF by
unfolding the field.

On the other hand, the grammar clearly allows bare LF, implicitly in
qtext/dtext/ctext and explicitly in text, contradicting the previous
statement from 3.1.2.

On the other hand, 3.4.8 says "Each header field may be represented
on exactly one line consisting of the name of the field and its body,
and terminated by a CRLF; this is what the parser sees."  Therefore,
we ought to be able to remove the internal line break by unfolding the
field.

On the other hand, 3.4.5 says "the presence of the quoting character
(backslash) explicitly indicates that the CRLF is data to the quoted
string."  How can the line break be data to the quoted string if the
parser doesn't even see the line break as stated in 3.4.8?

By the way, 3.4.3 says that \CRLF within a comment "must be followed
by at least one LWSP-char."  The same statement is not made regarding
quoted-strings, but could be inferred.  But nothing in the grammar
enforces this.  According to the grammar, this is a valid field:

From: "foo\
bar" <blah@example> (hi\
there)

This all seems like a big mess that should be deprecated.  And indeed,
in RFC 2822, lone CR, lone LF, and \CR are all relegated to obsolete
syntax.  I'm thinking that was a good move.

As for the obsolete grammar, parsing \CRLF as \CR followed by LF is
consistent with the 822 grammar, even if it doesn't seem to jibe with
the phrase "quoted CRLF" in the prose.

AMC

Cc: Keith Moore <[email-masked]>, "James M Galvin" <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Sun, 18 Jan 2004 01:33:01 -0500
To: "Al Costanzo" <[email-masked]>
Sender: [email-masked]

> To this point I would say, we need to know the physical location of the
> machine (node) on the Internet sending the email.

do we really care where the message was sent from, as opposed to who 
sent it?

I could see it being useful in the case of mail that was sent from a 
compromised machine (if we could locate the machine very precisely), or 
in the case of mail that was sent from a laptop that accessed a random 
802.11 network ("drive-by spamming").  But I have a hard time 
understanding how DNS could help us get this information, especially in 
the latter case.

Also, just as I don't want to expose the actual identity of every 
message sender, I wouldn't want to expose the location of every message 
sender.  (I can see it now - send an e-mail critical of Dubya, and a 
G-man knocks on your door within the hour...)

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Sun, 18 Jan 2004 01:13:23 -0500
To: James M Galvin <[email-masked]>
Sender: [email-masked]

>> What I'm after is a means of automating tracing for abuse complaints.
>
>     me too.  but I don't see how we can do that without providing
>     non-repudiation.
>
> It is not clear to me that by "hash" you meant digital signature, but
> clearly you need a signature for non-repudiation.

you have to have something to sign that is derived from the message in 
a repeatable fashion.  actually it's not the hash function that needs 
to be defined (SHA-1 would work fine), rather, it's the 
canonicalization function that is applied to a message before computing 
the hash.

>     otherwise it becomes easy to DoS somebody by forging mail as if it
>     were from them and generating lots of complaints about it.
>
> We need to be careful to avoid getting too wrapped up in DoS attacks.
> The problem is that it's possible with or without a hash and a
> signature.

yes, but once the complaint systems are automated then attacks on a 
sender using fake reports of abuse from that sender become more 
feasible.  and chances are the complaints that have originator-id 
fields are the ones that will be automated.

one nice thing - if the complaints themselves are required to have 
verifiable originator-id fields then attacking a user by sending fake 
abuse reports exposes the attacker :)

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Sun, 18 Jan 2004 01:08:46 -0500
To: James M Galvin <[email-masked]>
Sender: [email-masked]

>> Doesn't RFC1847 (security multiparts) already provide both of these, 
>> or
>> at least the framework sans the actual algorithm?
>
>     Only if everyone's user agents treated a security multipart
>     containing a message semantically the same as they would the
>     original message.
>
> Sure, so your primary concern is that the use of security multiparts
> would be a less backwards compatible change than a change to the
> Received syntax or adding a new header.

That, and there is a semantic difference between a signed message and a 
cryptographically verifiable trace field in a message.

>>     - A way to ensure that messages get tagged with originator-id when
>> they
>>     are injected into the mail systems
>>
>> I agree in principle, I think.  But couldn't a Received already 
>> provide
>> this information if everyone would just do it, i.e., it's optional 
>> now?
>
>     The two are fairly different.  You want to capture the 
> originator-id at
>     submission time, not at every hop.  And you need protocol elements 
> that
>     Received (not being extensible) cannot convey.  To me it makes more
>     sense to start from whole cloth.
>
> I agree with the principle.  As to whether it's in a Received header or
> in something different I also agree with Dave Crocker's point that we
> really need to examine what we want to capture before we can know for
> sure where to put it.

Fine with me.  In general I agree with the approach that you decide 
what the data model should be before you pick the presentation layer.  
(wish more IETF efforts would do the same...)

>>     - If you really wanted to, you could augment Received or add a new
>>     trace field that recomputed the hash at each hop (to show if and
>> where
>>     the message was corrupted in transport)
>>
>> Why would you forward a message that was discovered to be corrupted?
>
>     Probably because the message might have been corrupted in a way 
> that
>     makes the hash invalid without actually changing the content of the
>     message.  Something tells me it will be difficult to write a
>     canonicalization function that accommodates all of the various 
> kinds of
>     message and header munging that is out there.
>
> I can imagine it might be a local policy preference for all messages to
> be examined by a person instead of rejecting it if the hash validation
> fails.  However, it should be the case that except for human review the
> message is rejected (for whatever we decide rejection means).  There 
> may
> be some edge cases where broken mailers do really obscure things to
> messages to break the hash, but either you are doing security right or
> you don't do it.

Actually I don't see this as a security measure.  It doesn't really 
protect systems from any kind of attack, and it doesn't authenticate 
the message as being authored or witnessed by any publicly known 
principal name.  It doesn't even have to provide an unimpeachable 
assurance of non-repudiation -- it just has to be good enough that it's 
infeasible to make large quantities of spam look as if it were 
associated with originator-ids other than those controlled by the 
spammers.

>> I'm sorry but I just don't see what you have in mind that is worse 
>> than
>> it is today,
>
>     Given the apparent aspirations of the current occupant of the US 
> White
>     House, things could get far, far worse than they are now.  I'd far
>     rather have spammers than Big Brother George any day.
>
>> Can you please elaborate on the "mass surveillance" you fear?
>
>     Have you read the Patriot Act lately?
>
>     Okay, I'll be more specific.    If every message has an 
> originator-id
>     tag that can be traced to the origin, it becomes fairly simple for 
> the
>     US Government to insist that all US ISPs (and perhaps, all foreign 
> ISPs
>     peering with US ISPs) give them a list of mappings from tags to 
> more
>     recognizable identifiers (such as credit card #s), or that the ISPs
>     generate tags in such a way that the USG can simply decrypt them to
>     obtain those identifiers.  Given the kind of stuff that is already
>     authorized by existing laws, (and if not authorized, obtained by
>     coercion of various kinds) this isn't much of a stretch.  And 
> everybody
>     knows that terrorists use email...
>
>     Anyway, the reason my proposal allows originator-ids to be 
> ephemeral is
>     to make it hard for ordinary people to track messages - I believe
>     anonymous speech is important - and also out of recognition that
>     spammers can probably get ephemeral accounts anyway.
>
> My point is that we are not that from this already.  To stretch the
> example to the extreme, the US Government could make a law that ISPs
> have to make sure they can map a From: email address to an actual 
> person
> before accepting submission of a message (in the US of course).  Why do
> we need a special identifier?

The US Congress might or might not care, but for a variety of reasons 
it would not be appropriate to use From in this way.  There is no 
defined header field that can be used for this purpose without 
conflicting with valid and existing uses of the field.  (Sender would 
have been the right thing, but it's too widely misused now.)

> And as you suggest, anonymity is achieved by using a throw-away account
> with one of the free providers.  We just need to know that we can 
> filter
> on such origins if we don't want "trust" or otherwise want them.

To: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sat, 17 Jan 2004 17:23:32 -0800
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> IMO, Subject should be unstructured, period.  Unfortunately, RFC 1036
> introduced two hacks:

[...]

> 2. "cmsg", which *requires* certain actions when encountered.  See RFC
>    1035 sections 2.2.6 and 3 (including subsections).

RFC 1036 is not a standard and this is not best practice for Usenet
software.  Please do not include this in the grammar.

> Now, if an RFC 2822 successor were to repudiate those requirements
> spelled out in RFC 1036, that would be fine.

RFC 1036 is not a standards track document and therefore should not need
to be explicitly repudiated by another document.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sat, 17 Jan 2004 16:21:26 -0800 (PST)
From: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
To: Bruce Lilly <[email-masked]>
Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
Sender: [email-masked]

> Keith Moore wrote:

> >
> >> IMO, Subject should be unstructured, period.  Unfortunately, RFC 1036
> >> introduced two
> >> hacks:
> >
> >
> > Usenet is not email, and email should not be expected to inherit every
> > feature (or mistake) of Usenet.

> Transfer details are irrelevant to the message format.

But protocols and applications are not.

> The format is essentially the same,

Similarity does not make them equivalent. Nor does it mean, as Keith
points out, that one must cover all aspects of the other.

> and
> in many contexts (e.g. via IMAP) a message is just a message -- there is
> no way to differentiate
> "Usenet" and "email".

Which at most makes it convenient for the grammers to be equivalent. It does
not rise to the level of requiring it.

> "Re: " is a mistake of Usenet that RFC 2822 has
> picked up and run with.

Only to the extent of describing a prose convention. It does not appear in the
ABNF, nor should it IMO.

> ("cmsg" can for the moment be ignored here, since it is only significant
> for Usenet transport
> software)  From the point of view of library code handling the message
> format, the fact that
> 1036 requires "Re: " (and cmsg) to be recognized effectively makes that
> part of the syntax;
> library code has no way of knowing whether the calling application is
> Usenet-specific software,
> an MTA, an email-only MUA, a combined email/news UA, or an IMAP client
> (for which there
> is no Usenet/email distinction).

You're making all sorts of assumptions here that just do not hold water: That
it is necessary to handle "Re:" at the same syntactic level, that a library
even needs to concern itself with this detail, that all implementations must
allow for the conflation of netnews material and email material.

The bottom line as far as I'm concerned is that I see no requirement
that "Re:" be handled in the 2822 ABNF. And since it is not a requirement,
it becomes a matter of costs versus benefits. And from what I've seen
so far the costs far exceed the benefits.

				Ned

From: "Al Costanzo" <[email-masked]>
To: "James M Galvin" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: making mail traceable
Date: Sat, 17 Jan 2004 17:08:55 -0500
Sender: [email-masked]

To this point I would say, we need to know the physical location of the
machine (node) on the Internet sending the email.

I would say that  DNS is the perfect location for storing this information,
since unlike mail headers it is more difficult to mung around with by the
casual user and usually administrated properly by ISPs.

With the information stored in the DNS IMO, it gives us a second level of
repudiation to protect us all from spammers and help the US inforce the new
law.

Al

----- Original Message ----- 
From: "James M Galvin" <[email-masked]>
To: "Al Costanzo" <[email-masked]>
Cc: <[email-masked]>
Sent: Saturday, January 17, 2004 3:55 PM
Subject: Re: making mail traceable

>
> To the extent it is "information" we agree we need for email to be
> traceable and we agree that having it in the DNS is the right place for
> it, then it would be helpful.  Dave's point was that we should be
> discussing what information we need and agree on that before we try to
> agree on where to put it.
>
> Jim
>
>
>
>
> On Fri, 16 Jan 2004, Al Costanzo wrote:
>
>     Date: Fri, 16 Jan 2004 19:42:09 -0500
>     From: Al Costanzo <[email-masked]>
>     To: James M Galvin <[email-masked]>, [email-masked]
>     Subject: Re: making mail traceable
>
>
>
>     I may be missing the point here but about a year ago I wrote a draft
for a
>     DNS RR record to keep track of the physical location of the A or MX
record
>     as a "physical postal address" just for this purpose, with the
intention of
>     being able to track back the location of SPAM.
>
>     With new state laws I thought it was the way to go,  and then use MTA
use
>     this information.
>
>     Would this not help?  If so I have a copy of the draft I am currently
>     revising.
>
>     Al Costanzo
>     ----- Original Message -----
>     From: "James M Galvin" <[email-masked]>
>     To: "Dave Crocker" <[email-masked]>
>     Cc: "ietf-822" <[email-masked]>
>     Sent: Friday, January 16, 2004 5:02 PM
>     Subject: Re: making mail traceable
>
>
>     >
>     >
>     > On Fri, 16 Jan 2004, Dave Crocker wrote:
>     >
>     >     >     >  The "Received" header is woefully inadequate for spam
tracing
>     >     >
>     >     >     True.  Then again, so is the rest of the message format
and mail
>     >     >     transport.
>     >     >
>     >     > I don't agree it's "woefully inadequate."
>     >
>     >     When we have some agreement on the information that is needed to
>     >     facilitate spam tracing, then we can decide whether it is better
to
>     >     add it to Received or create a new header.
>     >
>     > Absolutely.
>     >
>     > And let us not forget some means to validate that the information we
do
>     > have or get is accurate and correct.  Or is that what Nathaniel and
>     > Keith meant by "traceable?"
>     >
>     > Jim
>     >
>     >
>
>
>
>

Date: Sat, 17 Jan 2004 16:11:11 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

On Sat, 17 Jan 2004, Keith Moore wrote:

    > What I'm after is a means of automating tracing for abuse complaints.

    me too.  but I don't see how we can do that without providing
    non-repudiation.

It is not clear to me that by "hash" you meant digital signature, but
clearly you need a signature for non-repudiation.

    otherwise it becomes easy to DoS somebody by forging mail as if it
    were from them and generating lots of complaints about it.

We need to be careful to avoid getting too wrapped up in DoS attacks.
The problem is that it's possible with or without a hash and a
signature.

So, I agree we should understand how any proposal permits or supports
DoS attacks, if it does, and we should certainly avoid any amplification
opportunities, but we're not going to prevent DoS with hash validation.

Jim

Date: Sat, 17 Jan 2004 15:55:47 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Al Costanzo <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

To the extent it is "information" we agree we need for email to be
traceable and we agree that having it in the DNS is the right place for
it, then it would be helpful.  Dave's point was that we should be
discussing what information we need and agree on that before we try to
agree on where to put it.

Jim

On Fri, 16 Jan 2004, Al Costanzo wrote:

    Date: Fri, 16 Jan 2004 19:42:09 -0500
    From: Al Costanzo <[email-masked]>
    To: James M Galvin <[email-masked]>, [email-masked]
    Subject: Re: making mail traceable

    I may be missing the point here but about a year ago I wrote a draft for a
    DNS RR record to keep track of the physical location of the A or MX record
    as a "physical postal address" just for this purpose, with the intention of
    being able to track back the location of SPAM.

    With new state laws I thought it was the way to go,  and then use MTA use
    this information.

    Would this not help?  If so I have a copy of the draft I am currently
    revising.

    Al Costanzo
    ----- Original Message -----
    From: "James M Galvin" <[email-masked]>
    To: "Dave Crocker" <[email-masked]>
    Cc: "ietf-822" <[email-masked]>
    Sent: Friday, January 16, 2004 5:02 PM
    Subject: Re: making mail traceable

    >
    >
    > On Fri, 16 Jan 2004, Dave Crocker wrote:
    >
    >     >     >  The "Received" header is woefully inadequate for spam tracing
    >     >
    >     >     True.  Then again, so is the rest of the message format and mail
    >     >     transport.
    >     >
    >     > I don't agree it's "woefully inadequate."
    >
    >     When we have some agreement on the information that is needed to
    >     facilitate spam tracing, then we can decide whether it is better to
    >     add it to Received or create a new header.
    >
    > Absolutely.
    >
    > And let us not forget some means to validate that the information we do
    > have or get is accurate and correct.  Or is that what Nathaniel and
    > Keith meant by "traceable?"
    >
    > Jim
    >
    >

Date: Sat, 17 Jan 2004 15:51:10 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

On Fri, 16 Jan 2004, Keith Moore wrote:

    > Doesn't RFC1847 (security multiparts) already provide both of these, or
    > at least the framework sans the actual algorithm?

    Only if everyone's user agents treated a security multipart
    containing a message semantically the same as they would the
    original message.

Sure, so your primary concern is that the use of security multiparts
would be a less backwards compatible change than a change to the
Received syntax or adding a new header.

    >     - A way to ensure that messages get tagged with originator-id when
    > they
    >     are injected into the mail systems
    >
    > I agree in principle, I think.  But couldn't a Received already provide
    > this information if everyone would just do it, i.e., it's optional now?

    The two are fairly different.  You want to capture the originator-id at
    submission time, not at every hop.  And you need protocol elements that
    Received (not being extensible) cannot convey.  To me it makes more
    sense to start from whole cloth.

I agree with the principle.  As to whether it's in a Received header or
in something different I also agree with Dave Crocker's point that we
really need to examine what we want to capture before we can know for
sure where to put it.

    >     - If you really wanted to, you could augment Received or add a new
    >     trace field that recomputed the hash at each hop (to show if and
    > where
    >     the message was corrupted in transport)
    >
    > Why would you forward a message that was discovered to be corrupted?

    Probably because the message might have been corrupted in a way that
    makes the hash invalid without actually changing the content of the
    message.  Something tells me it will be difficult to write a
    canonicalization function that accommodates all of the various kinds of
    message and header munging that is out there.

I can imagine it might be a local policy preference for all messages to
be examined by a person instead of rejecting it if the hash validation
fails.  However, it should be the case that except for human review the
message is rejected (for whatever we decide rejection means).  There may
be some edge cases where broken mailers do really obscure things to
messages to break the hash, but either you are doing security right or
you don't do it.

    > I'm sorry but I just don't see what you have in mind that is worse than
    > it is today,

    Given the apparent aspirations of the current occupant of the US White
    House, things could get far, far worse than they are now.  I'd far
    rather have spammers than Big Brother George any day.

    > Can you please elaborate on the "mass surveillance" you fear?

    Have you read the Patriot Act lately?

    Okay, I'll be more specific.    If every message has an originator-id
    tag that can be traced to the origin, it becomes fairly simple for the
    US Government to insist that all US ISPs (and perhaps, all foreign ISPs
    peering with US ISPs) give them a list of mappings from tags to more
    recognizable identifiers (such as credit card #s), or that the ISPs
    generate tags in such a way that the USG can simply decrypt them to
    obtain those identifiers.  Given the kind of stuff that is already
    authorized by existing laws, (and if not authorized, obtained by
    coercion of various kinds) this isn't much of a stretch.  And everybody
    knows that terrorists use email...

    Anyway, the reason my proposal allows originator-ids to be ephemeral is
    to make it hard for ordinary people to track messages - I believe
    anonymous speech is important - and also out of recognition that
    spammers can probably get ephemeral accounts anyway.

My point is that we are not that from this already.  To stretch the
example to the extreme, the US Government could make a law that ISPs
have to make sure they can map a From: email address to an actual person
before accepting submission of a message (in the US of course).  Why do
we need a special identifier?

And as you suggest, anonymity is achieved by using a throw-away account
with one of the free providers.  We just need to know that we can filter
on such origins if we don't want "trust" or otherwise want them.

Jim

Date: Sat, 17 Jan 2004 15:00:45 -0500
Subject: Re: making mail traceable (was Re: Received header Considered Pathetic)
Cc: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

I think this is a good discussion, so I'll do my best to fan the 
flames.  :-)

Keith correctly and clearly stated the core problem: automating abuse 
tracing without providing non-repudiation.  It's an impossible goal if 
stated in terms of absolutes, though, so let me restate my own vision 
of what the goal should be:  providing as much automated abuse tracing 
as possible without non-repudiation, or (in less technical terms) to 
provide the best possible spam tracing without eliminating anonymous 
email.

This is a problem that needs to balance privacy rights with law 
enforcement.  Historically, the way that open societies have typically 
dealt with this kind of issue is via the checks and balances of 
distributed control.  We can go a long way towards that by using 
cryptographic tokens to validate ISP's, but requiring legal procedures 
to access ISP's records in the course of a spam investigation.

Thus, getting a million complaints about spam messages that were 
cryptographically shown to come through an ISP should be enough to get 
a warrant to trace the senders, but one or a few complaints should be 
subjected to a much higher standard, to prevent police "fishing trips" 
in the name of spam control.  Automated traceability is a key to 
distinguishing between the two cases.

Of course, the previous paragraph would be more precise if, instead of 
"ISP" I had said "privacy-sensitive administrative domain."  Any 
domain-administering entity could choose to make tracing information 
from within its domain completely private, as long as it took 
responsibility for working with the authorities when it proved to be 
the terminal publicly-ascertainable node to which major spam could be 
traced.  Such domains would likely include large corporations and other 
institutions, including those that choose to run anonymous email 
gateways and fight spammers in their own ways (for example with 
computationally intensive challenge-response systems).

And yes, I realize that the above discussion completely ignores the 
jurisdictional issues, but I doubt that it will take more than a decade 
or two to work them out, and we need to think in the long term if we 
really want to control spam.  -- Nathaniel

PS -- Am I afraid that all of this will help Big Brother?   You bet.  
That's why I want to design as many checks & balances into the system 
as we can come up with.  But I think the CANSPAM act has made it very 
clear that there are likely to be ever more detailed regulations 
governing email systems, and I think it would be a losing battle (and 
therefore arguably irresponsible) to oppose any particular proposal 
without having a more moderate alternative proposal to endorse.  The US 
congress passed this law, however ill-informedly, because they 
correctly perceived a public demand to fix the problem of spam.  
Enabling Big Brother is, quite simply, the easiest, laziest solution to 
the problem, and therefore it is precisely what will happen unless we 
go to the effort of designing a less objectionable one.  -- Nathaniel

Date: Sat, 17 Jan 2004 12:44:54 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: making mail traceable (was Re: Received header Considered Pathetic)
Sender: [email-masked]

Keith Moore wrote:

>> I'm not quite ready to abandon it, primarily for practical reasons:
>> + it's already widely implemented
>> + it's already used for spam tracking
>>
>> Any alternative will require substantial time to pick up enough momentum
>> to be useful.
>
>
> that's true of any new information that might be provided regardless 
> of whether it uses a new header field or if the new information 
> somehow gets stuffed into the received field.  the point is, the 
> existing information is unreliable and inadequate.

Use of HELO/EHLO names is unreliable and inadequate. Use of domain names or
domain literals isn't new information.

>> What I'm after is a means of automating tracing for abuse complaints.
>
>
> me too.  but I don't see how we can do that without providing 
> non-repudiation.  otherwise it becomes easy to DoS somebody by forging 
> mail as if it were from them and generating lots of complaints about it.

Tracing back through Received fields, after my ISP's fields are 
accounted for,
the host named (if named reliably, i.e. not via HELO/EHLO name) as the 
source
is one of:
a) the sender's machine
b) one of the sender's ISPs' machines
c) an open relay
d) a resender, such as a mailing list expander

In case a or b, a complaint to the sender's ISP is appropriate; in case 
c the operator
of the open relay is essentially the spammer's accomplice and should be 
the recipient
of a complaint.  In case d, one can trace back further.

> nor do I think it's sufficient to get ISPs to terminate spammers.  
> what we need is a way to find out if a message is spam (or if the 
> sender is a spammer) after the message is sent, but before it is 
> delivered or read.

Tracing after the fact need not be the only tool used to fight spam; 
however it is a
valuable tool and its value could be increased by making it more 
amenable to automation.

> it's pretty hard to behave in the community interest when Big Brother 
> is twisting your arm.

Yes, if Big Brother is forcing you to behave against the community 
interest.  I know of
no coercion against ISPs that would prevent an ISP from enforcing its 
own terms of
service regarding unsolicited bulk email, do you?

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

Date: Sat, 17 Jan 2004 12:29:14 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Sender: [email-masked]

Keith Moore wrote:

>
>> IMO, Subject should be unstructured, period.  Unfortunately, RFC 1036 
>> introduced two
>> hacks:
>
>
> Usenet is not email, and email should not be expected to inherit every 
> feature (or mistake) of Usenet.

Transfer details are irrelevant to the message format. The format is 
essentially the same, and
in many contexts (e.g. via IMAP) a message is just a message -- there is 
no way to differentiate
"Usenet" and "email".  "Re: " is a mistake of Usenet that RFC 2822 has 
picked up and run with.
("cmsg" can for the moment be ignored here, since it is only significant 
for Usenet transport
software)  From the point of view of library code handling the message 
format, the fact that
1036 requires "Re: " (and cmsg) to be recognized effectively makes that 
part of the syntax;
library code has no way of knowing whether the calling application is 
Usenet-specific software,
an MTA, an email-only MUA, a combined email/news UA, or an IMAP client 
(for which there
is no Usenet/email distinction).

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

Cc: Keith Moore <[email-masked]>, Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Date: Sat, 17 Jan 2004 11:48:22 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

> IMO, Subject should be unstructured, period.  Unfortunately, RFC 1036 
> introduced two
> hacks:

Usenet is not email, and email should not be expected to inherit every 
feature (or mistake) of Usenet.

Cc: Keith Moore <[email-masked]>, "James M Galvin" <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Sat, 17 Jan 2004 11:46:53 -0500
To: "Al Costanzo" <[email-masked]>
Sender: [email-masked]

it might or might not help.  but it's orthogonal to what I was 
proposing.

Keith

> I may be missing the point here but about a year ago I wrote a draft 
> for a
> DNS RR record to keep track of the physical location of the A or MX 
> record
> as a "physical postal address" just for this purpose, with the 
> intention of
> being able to track back the location of SPAM.
>
> With new state laws I thought it was the way to go,  and then use MTA 
> use
> this information.
>
> Would this not help?  If so I have a copy of the draft I am currently
> revising.

Cc: Keith Moore <[email-masked]>, Nathaniel Borenstein <[email-masked]>, Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable (was Re: Received header Considered Pathetic)
Date: Sat, 17 Jan 2004 11:45:02 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

>> The "Received" header is woefully inadequate for spam tracing
>
> I'm not quite ready to abandon it, primarily for practical reasons:
> + it's already widely implemented
> + it's already used for spam tracking
>
> Any alternative will require substantial time to pick up enough 
> momentum
> to be useful.

that's true of any new information that might be provided regardless of 
whether it uses a new header field or if the new information somehow 
gets stuffed into the received field.  the point is, the existing 
information is unreliable and inadequate.

> What I'm after is a means of automating tracing for abuse complaints.

me too.  but I don't see how we can do that without providing 
non-repudiation.  otherwise it becomes easy to DoS somebody by forging 
mail as if it were from them and generating lots of complaints about 
it.

nor do I think it's sufficient to get ISPs to terminate spammers.  what 
we need is a way to find out if a message is spam (or if the sender is 
a spammer) after the message is sent, but before it is delivered or 
read.

> And no, I certainly don't want a Big Brother mechanism.  I expect ISPs 
> to behave
> responsibly and in the community interest (and for the most part, they 
> do -- they
> aren't any happier about having their resources wasted by spammers 
> than anybody
> else).

it's pretty hard to behave in the community interest when Big Brother 
is twisting your arm.

Date: Sat, 17 Jan 2004 11:31:55 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: Nathaniel Borenstein <[email-masked]>, ietf-822 <[email-masked]>, Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
Subject: Re: making mail traceable (was Re: Received header Considered Pathetic)
Sender: [email-masked]

Keith Moore wrote:

[quoting Nathaniel Borenstein]

> The "Received" header is woefully inadequate for spam tracing

I'm not quite ready to abandon it, primarily for practical reasons:
+ it's already widely implemented
+ it's already used for spam tracking

Any alternative will require substantial time to pick up enough momentum
to be useful.

Granted, there are some issues that need to be resolved:
- the machine-readable parts should have reliable information
   RFC 821 as amended by 1123 seems to permit this (relevant verbiage having
   been copied, warts and all, into 2821), though not specified clearly 
enough
   to ensure interoperable machine-readable implementations.
- of course, some time will be required to migrate existing implementations,
   however, I expect that to happen reasonably quickly, certainly more 
quickly
   than a ground-up replacement

> If we really want to make mail traceable, we need to do a bit more 
> than fix Received.  As I see it, we need:
>
> [...]
> But do we really want traceability?  Or to put it another way, do we 
> really want to put hooks in the mail system that make mass 
> surveillance (by governments, or perhaps even by large companies or 
> unscrupulous ISPs) that much easier?

What I'm after is a means of automating tracing for abuse complaints.
I don't expect:
a) to be able to use Received field content on-the-fly to reject spam
b) to be able to personally identify the individual responsible -- I'll 
leave that
    to his ISP's abuse department
c) the legal system to be of any use whatsoever regarding spam; nothing 
significant
    has changed since Jonathan Swift wrote "Gulliver's Travels" (q.v.; 
e.g. see
    http://swift.thefreelibrary.com/Gullivers-Travels/4-5 and search for 
"perplexed")
    in 1726.

And no, I certainly don't want a Big Brother mechanism.  I expect ISPs 
to behave
responsibly and in the community interest (and for the most part, they 
do -- they
aren't any happier about having their resources wasted by spammers than 
anybody
else).

Here's a concrete example of a recent spam message's Received fields, 
manually traced,
with comments interspersed.  Given the appropriate information in 
machine-readable
form, there's no reason why this couldn't be automated.

> Received: from mr06.mrf.mail.rcn.net (207.172.4.25 [207.172.4.25])
    by ms07.mrf.mail.rcn.net (Mirapoint Messaging Server MOS 3.2.2-GA 
FastPath)
    with ESMTP id CDL85988;
    Mon, 12 Jan 2004 06:14:33 -0500 (EST)

That's my ISP's mail servers (I recognize both the domain name and IP 
address).

> Received: from mx03.mrf.mail.rcn.net (mx03.mrf.mail.rcn.net [207.172.4.52])
    by mr06.mrf.mail.rcn.net (Mirapoint Messaging Server MOS 3.3.5-GR)
    with ESMTP id BGS07944;
    Mon, 12 Jan 2004 06:14:31 -0500 (EST)

Another of my ISP's mail servers; note that mr06.mrf.mail.rcn.net in the by
component matches the from component of the Received field which was added
after this one.  Note also that the timestamps are close in time and 
flowing in the
correct direction.

> Received: from pool-207-68-110-60.alt.east.verizon.net ([207.68.110.60])
    by mx03.mrf.mail.rcn.net with smtp (Exim 3.35 #4)
    id 1Ag01f-0006Td-00; Mon, 12 Jan 2004 06:14:31 -0500

That records whence my ISP's MX receiver received the message; a Verizon 
customer.
Had I only the IP address, a reverse lookup would have told me that.  
N.B. the time stamp
remains consistent.

> Received: from [132.214.45.221] by 
pool-207-68-110-60.alt.east.verizon.net; Tue, 13 Jan 2004 07:15:55 +0600

That one's bogus; at the time of receipt, the timestamp was in the future.

The spam complaint went to [email-masked].

[BTW, w.r.t. other trace fields inserted by ISPs, you might well see 
some in the header
of this message...]

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

Date: Sat, 17 Jan 2004 09:50:59 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Revisiting RFC 2822 grammar (date)
Sender: [email-masked]

Charles Lindsey wrote:

>I don't think 4DIGIT was ever in obs-year. Nor even in RFC 822?
>  
>
2822 obs-year provides for 2+ DIGITs.  RFC 822's error in omitting 
4-digit years
was amended by RFC 1123.

Date: Sat, 17 Jan 2004 09:45:44 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

Pete Resnick wrote:

>
> On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:
>
>>> obs-phrase      =       word *(word / ("." [CFWS]))
>>
>>
>> OK, but that is not "obsolete". It is intended as an extension to be 
>> allowed sometime in the future on a "MUST accept, SHOULD NOT generate 
>> yet" basis. So please can we rename it as 'extended-phrase' (which is 
>> what I have currently put in Usefor).
>
>
> I am not convinced this is worth it. It's explained perfectly well in 
> the text.

I believe that the complaint is not about the explanation, but rather 
about the label,
which labels as "obs[olete]" something which has never been legal, and which
might at some point in the future be legalized.  I.e., a different, more 
accurately
descriptive label is being requested.

Date: Sat, 17 Jan 2004 09:36:08 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (Subject field)
Sender: [email-masked]

Pete Resnick wrote:

> On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:
>
>> 6. I don't like Bruce's changes to subject. They mix in RFC 1036 
>> syntax, which IMO does not belong in 2822.
>
>
> I agree. These should be removed.

IMO, Subject should be unstructured, period.  Unfortunately, RFC 1036 
introduced two
hacks:
1. "Re: " (which it got wrong!), which *requires* certain actions when 
encountered, i.e. it is
    effectively part of the syntax.  See RFC 1036 sections 2.1.4, 2.2.5.
2. "cmsg", which *requires* certain actions when encountered.  See RFC 
1035 sections 2.2.6
    and 3 (including subsections).

This is the same "Subject" field described in RFCs 822 and 2822.

Now, if an RFC 2822 successor were to repudiate those requirements 
spelled out in RFC 1036,
that would be fine.

As things stand now, it is necessary to recognize the "re: " and "cmsg" 
hacks in order to comply
with RFC 1036, and there is at least one draft nearing RFC status that 
would add "Auto: " to the
list of Subject field hacks.

RFC 2822 currently has verbiage (section 3.6.5) which falls short of a 
requirement, however it
does nothing to remove the RFC 1036 requirements.  The verbiage also:
1. implies that presence of "Re: " indicates that a message is a reply, 
which is incorrect (I daily
    receive several spam messages having Subject fields beginning with 
"Re: " which are not replies).
2. seems wholly unnecessary. It might as well state that when not used 
in a reply, Subject MAY
    begin with "Re: ", when used in a reply, Subject MAY begin with 
"Qwerty%$@&^%#:_",
    when not used in a reply, Subject MAY begin with "Qwerty%$@&^%#:_", etc.

If an RFC 2822 successor were to repudiate the RFC 1036 Subject hack 
requirements, then
subject = "Subject:" unstructured CRLF
would be accurate (provided no other hacks are introduced in the interim).

Date: Sat, 17 Jan 2004 09:05:01 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (quoted-pair)
Sender: [email-masked]

Pete Resnick wrote:

> Yup, you're right, but that wasn't intended by 2822. Or more to the 
> point, it's not obvious to me that it was intended that bare CR or 
> bare LF should appear in the obs- version of body, which is the result.

OK.  This looks like it might need some discussion and non-trivial work 
to fix.

>> As an aside, note that obs-text may consist of multiple octets, so 
>> "\foo" could be considered a
>> quoted-pair ("foo" matches obs-text via *(obs-char *LF *CR)).  I 
>> believe that's a problem with 2822.
>
>
> I agree that it is a bug.

I think the revised grammar corrects that.

>> On the other hand, 822 specifically mentioned the multi-character 
>> \CRLF and gave its semantics, but 2822 doesn't seem to permit that.
>
>
> \CRLF is a quoted pair with CR followed by a bare LF. The current 
> syntax of 2822 permits that, leaving aside the question of whether or 
> not it should.

Well, 2822 provides for parsing as described, but as both \CR and lone 
LF are obs- constructs,
it does not permit generation.  Moreover, the semantics of \CRLF as 
given in RFC 822 are quite
different from \CR followed by a lone LF.

If indeed the RFC 822 construct and its semantics are to be deemed 
obsolete, there at least ought
to be mention of that fact in the "Differences from Earlier Standards" 
appendix.  OTOH if it's
to be continued, there ought to be provision for it as a single entity 
in the grammar.

Date: Sat, 17 Jan 2004 05:39:41 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF mailing list on MIME and e-mail <[email-masked]>
Subject: Re: Will case of non-ascii character be preserved by IDNA?
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Jacob Palme <[email-masked]> wrote:

> Will case of non-ascii character be preserved by IDNA?

Simon gave a perfect explanation of what will happen in practice.

> Characters (ASCII and non-ASCII) are case-folded into lowercase by
> Nameprep, iff the string contain _any_ non-ASCII character.

That is indeed what will happen if the sender applies ToASCII, which is
almost certainly what all existing implementations of IDNA do.

In theory, a conformant IDNA implementation need not apply ToASCII,
but could instead apply some other operation that returns equivalent
results.  The requirement in the IDNA spec is:

    Whenever a domain name is put into an IDN-unaware domain name slot,
    it MUST contain only ASCII characters.  Given an internationalized
    domain name (IDN), an equivalent domain name satisfying this
    requirement can be obtained by applying the ToASCII operation to
    each label...

In other words, ToASCII is sufficient, but is a little stricter than
necessary.  For example, given the IDN München.Net, a sender that simply
uses ToASCII will convert it to xn--mnchen-3ya.Net, but it would also
be permissible to send XN--MNCHEN-3YA.Net or Xn--MnChEn-3Ya.NeT or
xn--Mnchen-3ya.Net.  That last possibility is interesting because when
the receiver applies ToUnicode to it, the result will be München.Net.
Thus it would be trivial for senders to preserve the case of ASCII
letters without relying on any special cooperation from receivers.

Non-ASCII letters are another story.  Although the underlying Punycode
encoding is capable of carrying case information, IDNA makes no use of
that capability (IDNA is complex enough without it).  For a receiver
that uses ToUnicode to display IDNs (as all existing implementations
surely do), there is no way to make it output uppercase Ü.

In theory, receivers need not use ToUnicode, but could instead use some
other operation that returns equivalent results.  The requirement in the
IDNA spec is:

    ACE labels obtained from domain name slots SHOULD be hidden from
    users...  Given an internationalized domain name, an equivalent
    domain name containing no ACE labels can be obtained by applying the
    ToUnicode operation to each label.

Therefore, it is conceivable that someday a pair of operations
newToASCII and newToUnicode could be implemented that return results
equivalent to those of ToASCII and ToUnicode, but which work together to
preserve the case of non-ASCII letters.  For example, given the label
MÜNCHEN, newToASCII might return xN--MNCHEN-3yA, which a receiver using
newToUnicode would display as MÜNCHEN, and which a receiver using the
old ToUnicode would display as MüNCHEN.  All of these are equivalent
(IDNs are case-insensitive).  Working out the details of newToASCII and
newToUnicode would be non-trivial.

But it's not clear to what extent case preservation is demanded, or
even expected.  I've noticed that email addresses often get coerced to
all-caps, even the local part (in blatant disregard of the standards,
which say that local parts can be case sensitive).

AMC

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: Will case of non-ascii character be preserved by IDNA?
Date: Sat, 17 Jan 2004 03:55:39 +0100
Lines: 51
Cancel-Lock: sha1:19Wwp1ILm6ZS1JPOZvwEJsrTWME=
Sender: [email-masked]

Jacob Palme <[email-masked]> writes:

> Will case of non-ascii character be preserved by IDNA?

No.  Characters (ASCII and non-ASCII) are case-folded into lowercase
by Nameprep, iff the string contain _any_ non-ASCII character.

> If I send a message with the following header:
> To: Maria <[email-masked]>
> Then the mail will arrive to all recipients with this header
> exacly as when I sent it as shown above, i.e. in lower case.
> "münchen" will be translated to IDNA format, and then back
> to user-friendly format at receipt again.

Headers cannot contain non-ASCII, so you must use IDNA before putting
the data in the header, but that's a technicality.

münchen.net => xn--mnchen-3ya.net

Thus, you will send:

To: Maria <[email-masked]>

Which is translated, by the receiver, into:

To: Maria <[email-masked]>

> If I send a message with the following header:
> To: MARIA <[email-masked]>
> Then the mail will arrive to recipients in the following
> format:
> To: MARIA <[email-masked]>
> i.e. The upper case Ü in the original message has through
> translation to IDMA format, and back again, been converted
> to a lower case ü.

Similar in this case; you invoke IDNA on the IDN and then put the
output in the header.

MÜNCHEN.NET => xn--mnchen-3ya.NET

Thus, you will send:

To: MARIA <[email-masked]>

Which is translated, by the receiver, into:

To: Maria <[email-masked]>

Regards,
Simon

From: "Al Costanzo" <[email-masked]>
To: "James M Galvin" <[email-masked]>, <[email-masked]>
Subject: Re: making mail traceable
Date: Fri, 16 Jan 2004 19:42:09 -0500
Sender: [email-masked]

I may be missing the point here but about a year ago I wrote a draft for a
DNS RR record to keep track of the physical location of the A or MX record
as a "physical postal address" just for this purpose, with the intention of
being able to track back the location of SPAM.

With new state laws I thought it was the way to go,  and then use MTA use
this information.

Would this not help?  If so I have a copy of the draft I am currently
revising.

Al Costanzo
----- Original Message ----- 
From: "James M Galvin" <[email-masked]>
To: "Dave Crocker" <[email-masked]>
Cc: "ietf-822" <[email-masked]>
Sent: Friday, January 16, 2004 5:02 PM
Subject: Re: making mail traceable

>
>
> On Fri, 16 Jan 2004, Dave Crocker wrote:
>
>     >     >  The "Received" header is woefully inadequate for spam tracing
>     >
>     >     True.  Then again, so is the rest of the message format and mail
>     >     transport.
>     >
>     > I don't agree it's "woefully inadequate."
>
>     When we have some agreement on the information that is needed to
>     facilitate spam tracing, then we can decide whether it is better to
>     add it to Received or create a new header.
>
> Absolutely.
>
> And let us not forget some means to validate that the information we do
> have or get is accurate and correct.  Or is that what Nathaniel and
> Keith meant by "traceable?"
>
> Jim
>
>

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: making mail traceable
Date: Fri, 16 Jan 2004 17:23:12 -0500
To: James M Galvin <[email-masked]>
Sender: [email-masked]

>     If we really want to make mail traceable, we need to do a bit more 
> than
>     fix Received.  As I see it, we need:
>
>     - A message hash function that is invariant across the various 
> kinds of
>     munging that happens in mail transport, but still good enough for
>     non-repudiation (though it probably won't be good enough to serve 
> as a
>     general-purpose signature)
>
>     - A new header field which associates the message hash, 
> originator-id,
>     timestamp, and originating ISP or organization, which is signed by 
> that
>     originating ISP or organization, and which is easily verifiable by
>     recipients or MTAs
>
> Doesn't RFC1847 (security multiparts) already provide both of these, or
> at least the framework sans the actual algorithm?

Only if everyone's user agents treated a security multipart containing 
a message semantically the same as they would the original message.  
With current user agents, having submission servers push messages down 
into security multiparts would change the way that messages were 
interpreted, processed, and presented by a significant number of 
intermediaries and MUAs.

>     - An originator-id separate from From, MAIL FROM/Return-Path, 
> Reply-To
>     or Sender that uniquely distinguishes the originator of the message
>     from other message originators.  it doesn't have to actually 
> expose the
>     originator's name, email address, account name, etc. - it could be 
> a
>     nonce as long as the originating ISP or organization could trace 
> it to
>     the actual originator within a reasonable time.
>
> So you want something functionally like the ident protocol built into
> the Received header?

There's at least a vague similarity between the ident protocol and one 
piece of the package I have in mind, but it might be confusing to make 
too many associations between the two.

>     - A way to ensure that messages get tagged with originator-id when 
> they
>     are injected into the mail systems (e.g. ISPs blocking port 25 
> and/or
>     MTAs refusing to accept incoming mail without originator-ids or 
> with
>     unverifiable originator-ids)
>
> I agree in principle, I think.  But couldn't a Received already provide
> this information if everyone would just do it, i.e., it's optional now?

The two are fairly different.  You want to capture the originator-id at 
submission time, not at every hop.  And you need protocol elements that 
Received (not being extensible) cannot convey.  To me it makes more 
sense to start from whole cloth.

>     - If you really wanted to, you could augment Received or add a new
>     trace field that recomputed the hash at each hop (to show if and 
> where
>     the message was corrupted in transport)
>
> Why would you forward a message that was discovered to be corrupted?

Probably because the message might have been corrupted in a way that 
makes the hash invalid without actually changing the content of the 
message.  Something tells me it will be difficult to write a 
canonicalization function that accommodates all of the various kinds of 
message and header munging that is out there.

>     This would give you a way to associate each message with an 
> identifier
>     for the originator, issued by the originator's ISP or organization.
>     Then you'd need some way to ask that ISP or organization "is the 
> guy
>     who sent this message trustworthy?"  And they could say "as far as 
> we
>     know, he doesn't have many abuse reports and he's been with us for
>     years" or "he just signed up yesterday" or "this is a trial 
> account, we
>     have no billing information for him" or "we've had several hundred
>     abuse reports in the last 3 hours".
>
> So you're looking for an extension to the ident protocol based on the
> presence of a "string" inserted in a message by its original point of
> submission?

Again, I think an analogy to ident would be confusing.  You would not 
be asking this ISP to say "who is on the other end of this connection"  
- you'd be asking the ISP to tell you some information about the guy 
who sent the message with originator-id field "XXXXX".  (Though it is 
conceivable that a first-hop MTA might want to use something like ident 
as a means to obtain originator-ids for messages that don't have them, 
that's not what I'm proposing now.)

>     But do we really want traceability?  Or to put it another way, do 
> we
>     really want to put hooks in the mail system that make mass
>     surveillance (by governments, or perhaps even by large companies or
>     unscrupulous ISPs) that much easier?
>
> I'm sorry but I just don't see what you have in mind that is worse than
> it is today,

Given the apparent aspirations of the current occupant of the US White 
House, things could get far, far worse than they are now.  I'd far 
rather have spammers than Big Brother George any day.

> Can you please elaborate on the "mass surveillance" you fear?

Have you read the Patriot Act lately?

Okay, I'll be more specific.    If every message has an originator-id 
tag that can be traced to the origin, it becomes fairly simple for the 
US Government to insist that all US ISPs (and perhaps, all foreign ISPs 
peering with US ISPs) give them a list of mappings from tags to more 
recognizable identifiers (such as credit card #s), or that the ISPs 
generate tags in such a way that the USG can simply decrypt them to 
obtain those identifiers.  Given the kind of stuff that is already 
authorized by existing laws, (and if not authorized, obtained by 
coercion of various kinds) this isn't much of a stretch.  And everybody 
knows that terrorists use email...

Anyway, the reason my proposal allows originator-ids to be ephemeral is 
to make it hard for ordinary people to track messages - I believe 
anonymous speech is important - and also out of recognition that 
spammers can probably get ephemeral accounts anyway.  The solution to 
ephemeral accounts is to provide a way for recipients to distinguish 
these from ones where the ISP really does know who sent the message.  
But I really don't know how to make messages more traceable on one hand 
and not enable more surveillance on the other.

Keith

Date: Fri, 16 Jan 2004 17:02:36 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Dave Crocker <[email-masked]>
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

On Fri, 16 Jan 2004, Dave Crocker wrote:

    >     >  The "Received" header is woefully inadequate for spam tracing
    >
    >     True.  Then again, so is the rest of the message format and mail
    >     transport.
    >
    > I don't agree it's "woefully inadequate."

    When we have some agreement on the information that is needed to
    facilitate spam tracing, then we can decide whether it is better to
    add it to Received or create a new header.

Absolutely.

And let us not forget some means to validate that the information we do
have or get is accurate and correct.  Or is that what Nathaniel and
Keith meant by "traceable?"

Jim

From: Dave Crocker <[email-masked]>
To: ietf-822 <[email-masked]>
Reply-To: [email-masked]
Date: Fri, 16 Jan 2004 16:46:24 -0500
Subject: some message bullet-proofing
Sender: [email-masked]

>     If we really want to make mail traceable, we need to do a bit more than
>     fix Received.  As I see it, we need:
>
>     - A message hash function that is invariant across the various kinds of
>     munging that happens in mail transport, but still good enough for
>     non-repudiation (though it probably won't be good enough to serve as a
>     general-purpose signature)
>
>     - A new header field which associates the message hash, originator-id,
>     timestamp, and originating ISP or organization, which is signed by that
>     originating ISP or organization, and which is easily verifiable by
>     recipients or MTAs

This has some general utility, and is also needed for anything along the lines 
of the "domain keys" transit validation mechanism.

d/

From: Dave Crocker <[email-masked]>
To: <[email-masked]>, Keith Moore <[email-masked]>
CC: ietf-822 <[email-masked]>
Reply-To: [email-masked]
Date: Fri, 16 Jan 2004 16:43:35 -0500
Subject: Re: making mail traceable
Sender: [email-masked]

>     >  The "Received" header is woefully inadequate for spam tracing
>
>     True.  Then again, so is the rest of the message format and mail
>     transport.
>
> I don't agree it's "woefully inadequate."  

In the realm of anti-spam, the typical approach to discussion seems to be 
first to declare that some portion of the system is woefully inadequate, and 
then to propose a new and wonderful solution.  The 'then' clause is often 
omitted. 

So...

When we have some agreement on the information that is needed to facilitate 
spam tracing, then we can decide whether it is better to add it to Received or 
create a new header.

d/

Date: Fri, 16 Jan 2004 16:30:56 -0500 (EST)
From: James M Galvin <[email-masked]>
Subject: Re: making mail traceable
To: Keith Moore <[email-masked]>
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

On Fri, 16 Jan 2004, Keith Moore wrote:

    >  The "Received" header is woefully inadequate for spam tracing

    True.  Then again, so is the rest of the message format and mail
    transport.

I don't agree it's "woefully inadequate."  I agree that many do not put
all the information they could in a Received header (because it is
optional), but that does not make the Received header itself inadequate.

The syntax could be stricter to make automatic processing easier, but
that does not make it inadequate, either.

What makes spam tracing hard is the fact that spammers lie in the
Received headers, or at least cheat so the wrong information is there.
How would a new syntax fix that?

Or did you have something else in mind?

    If we really want to make mail traceable, we need to do a bit more than
    fix Received.  As I see it, we need:

    - A message hash function that is invariant across the various kinds of
    munging that happens in mail transport, but still good enough for
    non-repudiation (though it probably won't be good enough to serve as a
    general-purpose signature)

    - A new header field which associates the message hash, originator-id,
    timestamp, and originating ISP or organization, which is signed by that
    originating ISP or organization, and which is easily verifiable by
    recipients or MTAs

Doesn't RFC1847 (security multiparts) already provide both of these, or
at least the framework sans the actual algorithm?

    - An originator-id separate from From, MAIL FROM/Return-Path, Reply-To
    or Sender that uniquely distinguishes the originator of the message
    from other message originators.  it doesn't have to actually expose the
    originator's name, email address, account name, etc. - it could be a
    nonce as long as the originating ISP or organization could trace it to
    the actual originator within a reasonable time.

So you want something functionally like the ident protocol built into
the Received header?

    - A way to ensure that messages get tagged with originator-id when they
    are injected into the mail systems (e.g. ISPs blocking port 25 and/or
    MTAs refusing to accept incoming mail without originator-ids or with
    unverifiable originator-ids)

I agree in principle, I think.  But couldn't a Received already provide
this information if everyone would just do it, i.e., it's optional now?

    - If you really wanted to, you could augment Received or add a new
    trace field that recomputed the hash at each hop (to show if and where
    the message was corrupted in transport)

Why would you forward a message that was discovered to be corrupted?

    This would give you a way to associate each message with an identifier
    for the originator, issued by the originator's ISP or organization.
    Then you'd need some way to ask that ISP or organization "is the guy
    who sent this message trustworthy?"  And they could say "as far as we
    know, he doesn't have many abuse reports and he's been with us for
    years" or "he just signed up yesterday" or "this is a trial account, we
    have no billing information for him" or "we've had several hundred
    abuse reports in the last 3 hours".

So you're looking for an extension to the ident protocol based on the
presence of a "string" inserted in a message by its original point of
submission?

    But do we really want traceability?  Or to put it another way, do we
    really want to put hooks in the mail system that make mass
    surveillance (by governments, or perhaps even by large companies or
    unscrupulous ISPs) that much easier?

I'm sorry but I just don't see what you have in mind that is worse than
it is today, nor do I see us needing anything that we don't already have
(except perhaps to operationally require it).

Can you please elaborate on the "mass surveillance" you fear?

Jim

Date: Fri, 16 Jan 2004 12:05:22 +0100
To: IETF mailing list on MIME and e-mail  <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Will case of non-ascii character be preserved by IDNA?
Sender: [email-masked]

Will case of non-ascii character be preserved by IDNA?

My understanding of the current standards is as follows:

If I send a message with the following header:
To: Maria <[email-masked]>
Then the mail will arrive to all recipients with this header
exacly as when I sent it as shown above, i.e. in upper case.

If I send a message with the following header:
To: Maria <[email-masked]>
Then the mail will arrive to the same recipients with this header
exacly as when I sent it as shown above, i.e. in lower case.

If I send a message with the following header:
To: Maria <[email-masked]>
Then the mail will arrive to all recipients with this header
exacly as when I sent it as shown above, i.e. in lower case.
"münchen" will be translated to IDNA format, and then back
to user-friendly format at receipt again.

If I send a message with the following header:
To: MARIA <[email-masked]>
Then the mail will arrive to recipients in the following
format:
To: MARIA <[email-masked]>
i.e. The upper case Ü in the original message has through
translation to IDMA format, and back again, been converted
to a lower case ü.

Is this right, or have I not correctly understood the IDMA
standard? Will, in fact, upper case Ü come to the recipients
as upper case Ü, which of course would be a little neater,
although this is surely not a very important issue?
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

From: Arnt Gulbrandsen <[email-masked]>
To: Charles Lindsey <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Cc: [email-masked]
Date: Fri, 16 Jan 2004 19:16:16 +0100
Sender: [email-masked]

Charles Lindsey writes:
> No, that was always allowed, because a dot-atom can always be preceded 
> by CFWS. But it is a good example of how hard it can be to check some 
> of these cases in the present syntax.

Yes.

>> 3. The date syntax is reordered, and four-digit years seem to have 
>> disappeared from the obsolete syntax. This looks like a good idea:
>
> I don't think 4DIGIT was ever in obs-year. Nor even in RFC 822?

Sorry, Pete is right, 2*DIGIT was on the page and my brain was seeing 
2DIGIT. 2*DIGIT is perfectly okay.

>> 7. There may be a few cases where FWS in 2822 is replced by CWFS in 
>> Bruce's grammar. It's a little hard to tell. For example, in Bruce's 
>> grammar there always is CFWS following field-name ":", I'm not sure 
>> the same holds for 2822. It may.
>
> Not so. Look at orig-date, for example, where it is explicitly [FWS] 
> that follows the ":".

Anyway, I think Bruce's stuff is better, because it's easier to read. In 
2822 I'd need a long time to be sure where CFWS is permitted (as 
opposed to only FWS), in Bruce's grammar it's much easier.

Arnt

Cc: Keith Moore <[email-masked]>, ietf-822 <[email-masked]>, Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: making mail traceable (was Re: Received header Considered Pathetic)
Date: Fri, 16 Jan 2004 11:47:43 -0500
To: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

>  The "Received" header is woefully inadequate for spam tracing

True.  Then again, so is the rest of the message format and mail 
transport.

If we really want to make mail traceable, we need to do a bit more than 
fix Received.  As I see it, we need:

- A message hash function that is invariant across the various kinds of 
munging that happens in mail transport, but still good enough for 
non-repudiation (though it probably won't be good enough to serve as a 
general-purpose signature)
- An originator-id separate from From, MAIL FROM/Return-Path, Reply-To 
or Sender that uniquely distinguishes the originator of the message 
from other message originators.  it doesn't have to actually expose the 
originator's name, email address, account name, etc. - it could be a 
nonce as long as the originating ISP or organization could trace it to 
the actual originator within a reasonable time.
- A new header field which associates the message hash, originator-id, 
timestamp, and originating ISP or organization, which is signed by that 
originating ISP or organization, and which is easily verifiable by 
recipients or MTAs
- A way to ensure that messages get tagged with originator-id when they 
are injected into the mail systems (e.g. ISPs blocking port 25 and/or 
MTAs refusing to accept incoming mail without originator-ids or with 
unverifiable originator-ids)
- If you really wanted to, you could augment Received or add a new 
trace field that recomputed the hash at each hop (to show if and where 
the message was corrupted in transport)

This would give you a way to associate each message with an identifier 
for the originator, issued by the originator's ISP or organization.  
Then you'd need some way to ask that ISP or organization "is the guy 
who sent this message trustworthy?"  And they could say "as far as we 
know, he doesn't have many abuse reports and he's been with us for 
years" or "he just signed up yesterday" or "this is a trial account, we 
have no billing information for him" or "we've had several hundred 
abuse reports in the last 3 hours".  And the receiving MTA or recipient 
could delay, filter, or bounce the message accordingly.  Of course, 
some ISPs would lie, and some would not support the verification 
protocol.  But they'd get reputations for those decisions, and they'd 
be marginalized.

But do we really want traceability?  Or to put it another way, do we 
really want to put hooks in the mail system that make mass surveillance 
(by governments, or perhaps even by large companies or unscrupulous 
ISPs) that much easier?

Keith

Date: Fri, 16 Jan 2004 10:32:46 -0600
To: ietf-822 <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (quoted-pair)
Cc: Arnt Gulbrandsen <[email-masked]>, Charles Lindsey <[email-masked]>, Bruce Lilly <[email-masked]>
Sender: [email-masked]

On 1/16/04 at 2:43 AM -0500, Bruce Lilly wrote:

>Pete Resnick wrote:
>
>>
>>On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:
>>
>>>1. Bruce elimites obs-qp, even though it can match a few pairs 
>>>quoted-pair cannot, such as "\" DEL.
>>
>>
>>Yup, I think that's Bruce's mistake.

Of course, as Bruce points out, DEL is in there, so this is no problem.

>>On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:
>>In <[email-masked]> [email-masked] writes:
>>
>>>quoted-pair     =       ("\" text)
>>>[N.B. had redundant obs-qp alternative]
>>
>>
>>I think not. The obs version allows \NUL, \CR and \LF, which the 
>>regular version does not.
>>Right.
>
>
>RFC 2822 gives quoted-pair as:
>
>quoted-pair     =       ("\" text) / obs-qp
>
>and text as:
>
>text            =       %d1-9 /         ; Characters excluding CR and LF
>                        %d11 /
>                        %d12 /
>                        %d14-127 /
>                        obs-text
>
>and obs-text (N.B. included in text) as:
>
>obs-text        =       *LF *CR *(obs-char *LF *CR)
>
>with obs-char defined as:
>
>obs-char        =       %d0-9 / %d11 /          ; %d0-127 except CR and
>                        %d12 / %d14-127         ;  LF
>
>DEL is %d127, which is explicitly included in text and obs-char. 
>Indeed, text explicitly includes every US-ASCII character by value 
>except NUL, CR, and LF, and includes those as well via obs-text 
>(explicitly permitting a single CR or LF) and  obs-char (which 
>includes NUL).  I don't see  any single US-ASCII character  which 
>quoted-pair doesn't permit  after the backslash,  without having to 
>resort to obs-qp.
>
>Maybe that wasn't intended by RFC 2822, but there it is.

Yup, you're right, but that wasn't intended by 2822. Or more to the 
point, it's not obvious to me that it was intended that bare CR or 
bare LF should appear in the obs- version of body, which is the 
result.

>As an aside, note that obs-text may consist of multiple octets, so 
>"\foo" could be considered a
>quoted-pair ("foo" matches obs-text via *(obs-char *LF *CR)).  I 
>believe that's a problem with 2822.

I agree that it is a bug.

>On the other hand, 822 specifically mentioned the multi-character 
>\CRLF and gave its semantics, but 2822 doesn't seem to permit that.

\CRLF is a quoted pair with CR followed by a bare LF. The current 
syntax of 2822 permits that, leaving aside the question of whether or 
not it should.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Fri, 16 Jan 2004 09:57:19 -0500
Subject: Received header Considered Pathetic (was Re: Revisiting RFC 2822 grammar (scratching the surface of Received issues))
Cc: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
To: ietf-822 <[email-masked]>
From: Nathaniel Borenstein <[email-masked]>
Sender: [email-masked]

Bruce has raised an issue I've been thinking about a lot recently.  The 
"Received" header is woefully inadequate for spam tracing, and has a 
history of syntactic cruft that will be hard to simplify.  I would like 
to see us design a replacement that is specifically designed to support 
rapid automated spam tracing.  I think that a combination of an 
easily-parsed Received header with a well designed Mail Source Tracing 
Protocol (by which you could automate certain queries to the various 
nodes a message had passed through) would be an extremely useful tool 
-- not a silver bullet, of course -- in the fight against spam, and in 
particular in efforts to enforce antispam laws.  Is anyone interested 
in working on a spec for this?  -- Nathaniel

On Friday, January 16, 2004, at 06:30  AM, Bruce Lilly wrote:

>
> Pete Resnick wrote:
>
>>
>> On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:
>>
>>> 5. obs-received merits discussion on its own. RFC 2822 says
>>>
>>>   obs-received = "Received" *WSP ":" name-val-list CRLF
>>>
>>> which Bruce changes to
>>>
>>>   obs-received = "Received" *WSP ":" [CFWS] name-val-list [ ";" 
>>> [CFWS] obs-date-time ] CRLF
>>>
>>> An incompatible change, but perhaps correct.
>>
>>
>> Yes, I think Bruce fixes a bug in 2822 here.
>
> Received has issues, both w.r.t. 2822 and 2821.
>
> First an historical overview:  Received was first specified in RFC 
> 821, also known as a
> "time stamp line".  There have been, and still are, discrepancies 
> between the 821/822 and
> 2821/2822 definitions of the field body.  Received is one of the trace 
> fields, and as noted
> in RFC 1123, was initially primarily examined by hand.  1123 permitted 
> adding information
> actually useful for tracing (i.e. the peer IP address, as opposed to 
> the HELO string which is
> far too easily forged), however gave no syntax for doing so [common 
> practice has been to
> include it in a comment].
>
> Nowadays, tracing Received fields by hand is far too labor-intensive.  
> Unfortunately, due
> to the exponential growth of spam, it is necessary.  We now have the 
> unfortunate situation
> where the required information (at least in 2821) includes the 
> easily-forged HELO/EHLO
> string, and the useful not-so-easily-forged connection information is 
> relegated to an optional
> construct.  Worse, in 2821 that optional construct is specified as 
> some sort of structured
> comment, indeed it is indistinguishable from a comment.
>
> What is really desirable is something that has reliable trace 
> information in machine-readable
> form, perhaps with the easily-forged information relegated to 
> comments.  But that's another
> discussion...
>
> Prior to 2822, a time stamp line (a.k.a. Received field) always 
> required a time stamp. 2822
> permits a time stamp line w/o a time stamp, which is an oxymoron.  I 
> don't personally like
> that (IMO the time stamp should be mandatory), but that's what 2822 
> says, and that's part
> of what is included in the syntax above.
>
> Another issue, not included above, is an incompatibility which has 
> crept in. 821/822 required
> SP before the semicolon which delimits the start of the date-time.  
> However, most MTAs
> incorrectly omit that space [in some, such as sendmail, that can be 
> easily rectified via a run-
> time configuration patch; others, such as qmail, have that error 
> hard-coded].  2821 requires
> CFWS immediately before the semicolon, but 2822 makes it optional.  
> Given 821/822/2821's
> requirements, I'd be inclined to revise 2822 to make at least CFWS 
> mandatory before the
> semicolon when generating a message, and given past (clearly wrong, 
> but quite widespread)
> practice, I'd require being able to parse a Received field w/o CFWS 
> before the semicolon
> (via obs- syntax).
>
> One more remaining incompatibility between 2821 and 2822 lies in the 
> permitted constructs;
> 2821 permits a quoted string as an item value (via 2821's "String") 
> whereas 2822 has no
> such provision.  That shows up in the "id" component, which has a long 
> history of conflicts
> between 821/822 (1123 tried to rectify the conflict, but only added 
> more confusion).
>
> Yet another incompatibility is that 2821 permits a mix of angle-addrs 
> (a.k.a. Paths) and
> addr-specs (a.k.a. Mailboxes) in a "for" component, whereas 2822 
> permits a single addr-spec
> or multiple angle-addrs (and no mixture).  It turns out that for 
> rather complicated reasons,
> the 2821 provision for multiple addr-specs is rather difficult to 
> parse.  Perhaps 2821's
> successor should address that issue; in any event, let's at least 
> remove the remaining conflicts
> between the 2821 and 2822 definitions one way or another.
>
> I note also that there exist broken implementations which generate 
> cruft that cannot be parsed
> even with 2822's exceptionally liberal rules.  Here are some 
> real-world examples:
>
> Received: from web197.nyc01.cbsig.net ([63.240.56.197])
> 	by mx08.mrf.mail.rcn.net with smtp (Exim 3.35 #7)
> 	id 1Af3Xn-0005vx-00
> 	for [email-masked]; Fri, 09 Jan 2004 15:47:47 -0500
> Received: (qmail 28572 invoked from network); 9 Jan 2004 20:47:13 -0000
> Received: from nychubg02.cbs.com (170.20.9.151)
>  by web197 with SMTP; 9 Jan 2004 20:47:13 -0000
> Received: by nychubg02.cbs.com with Internet Mail Service (5.5.2656.59)
> 	id <ZC7JJYV9>; Fri, 9 Jan 2004 15:41:17 -0500
>
> That's one recent example; among the problems:
> as noted above, the SP/CFWS-before-semicolon issue
> id's other than properly-constructed msg-ids
> RFC 821 does not permit day-of-week in the time stamp
> missing from and/or by components in some cases
> illegal (non-RFC 1700 cruft) in "with" components
> there is no defined "Mail" item-name
>
> Received: from panic.noceast.dws.disney.com (panic.corp.disney.com 
> [153.6.248.200])
> 	by mail.disney.com (Switch-3.1.2/Switch-3.1.0) with ESMTP id 
> h9NCwuN4022589
> 	for <[email-masked]>; Thu, 23 Oct 2003 05:58:57 -0700 (PDT)
> Received: from sm-flor-xc03.wdw.disney.com 
> (sm-flor-xc03.wdw.disney.com [172.16.177.30]) by 
> panic.noceast.dws.disney.com with ESMTP; Thu, 23 Oct 2003 08:55:03 
> -0400
> Received: from sm-flor-xc01.wdw.disney.com ([172.16.177.21]) by 
> sm-flor-xc03.wdw.disney.com with Microsoft SMTPSVC(5.0.2195.5329);
> 	 Thu, 23 Oct 2003 08:59:43 -0400
> Received: from SM-NYNY-XC01.nena.wdpr.disney.com ([167.13.137.76]) by 
> sm-flor-xc01.wdw.disney.com with Microsoft SMTPSVC(5.0.2195.5329);
> 	 Thu, 23 Oct 2003 08:59:42 -0400
> Received: from sm-nyny-xm05.nena.wdpr.disney.com ([167.13.137.80]) by 
> SM-NYNY-XC01.nena.wdpr.disney.com with Microsoft 
> SMTPSVC(5.0.2195.6713);
> 	 Thu, 23 Oct 2003 08:59:41 -0400
>
>
> That's even worse; additional problem is that parsing fails after the 
> (illegal) with
> component on encountering a lone "SMTPSVC" (and Microsoft was informed 
> about that bug
> in Windows 2000 well befor SP1; 3 service packs and as many years 
> later and the bug
> still hasn't been fixed (it shouldn't take more than 10 seconds for a 
> competent
> programmer to modify the source to a) use a legal value (ESMtp or 
> SMTP) in the with
> component, or b) elide the optional with component, or c) put the 
> marketing BS in a
> comment)...
>
> For the record, I am NOT in favor of extending the syntax to accept 
> such cruft -- I wish
> that certain purveyors of brokenware would clean up their acts.
>
>
>
> #################################################################
> #################################################################
> #################################################################
> #####
> #####
> #####
> #################################################################
> #################################################################
> #################################################################
>
>

Date: Fri, 16 Jan 2004 08:30:32 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Revisiting RFC 2822 grammar (obs-utext)
Sender: [email-masked]

Pete Resnick wrote:

>
> On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:
>
>>> obs-utext       =       *LF *CR *(obs-char *LF *CR)
>>> [N.B. was obs-text]
>>
>>
>> No, that does not work because it allows CRLF not followed by WSP in 
>> the middle of an 'unstructured'.
>
>
> Yup, because you can have "obs-utext obs-utext" which could be (abc 
> CR)(LF def).
>
>> I think the only way out of that is to rewrite the rule for 
>> 'unstructured':
>>
>> unstructured    =    *(utext [FWS]) obs-ltext
>> obs-utext    =    (1*LF *CR / 1*CR) obs-char / NUL
>> obs-ltext    =    *LF *CR
>
>
> Blech. I'll take a look and see what I can figure out without 
> resorting to that.

I believe that the only place that comes up is in 2822's "unstructured":

unstructured    =       *([FWS] utext) [FWS]

and could be rewritten to prevent the problem noted. Or a note could be 
added stating
that any CRLF which is part of obs-utext must be followed by WS.

Further note that this is obs-syntax, which is not for generation of new 
messages.

Also note that ASCII NUL is included in obs-char.

To: [email-masked]
Xref: clerew local.mime:2887
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Date: Fri, 16 Jan 2004 10:56:46 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>1. Bruce elimites obs-qp, even though it can match a few pairs 
>quoted-pair cannot, such as "\" DEL.

No, "\" DEL is in fact permitted by the regular syntax. The missed ones
are "\" (NUL / LF / CR)

>2. I think I found a case in which Bruce permits CFWS where it wasn't 
>before, although I cannot decipher my very cramped margin notes. It 
>was something like "To: blah: (cfws) [email-masked];".

No, that was always allowed, because a dot-atom can always be preceded by
CFWS. But it is a good example of how hard it can be to check some of
these cases in the present syntax.

>3. The date syntax is reordered, and four-digit years seem to have 
>disappeared from the obsolete syntax. This looks like a good idea:

I don't think 4DIGIT was ever in obs-year. Nor even in RFC 822?

>7. There may be a few cases where FWS in 2822 is replced by CWFS in 
>Bruce's grammar. It's a little hard to tell. For example, in Bruce's 
>grammar there always is CFWS following field-name ":", I'm not sure 
>the same holds for 2822. It may.

Not so. Look at orig-date, for example, where it is explicitly [FWS] that
follows the ":".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 16 Jan 2004 06:30:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (scratching the surface of Received issues)
Sender: [email-masked]

Pete Resnick wrote:

>
> On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:
>
>> 5. obs-received merits discussion on its own. RFC 2822 says
>>
>>   obs-received = "Received" *WSP ":" name-val-list CRLF
>>
>> which Bruce changes to
>>
>>   obs-received = "Received" *WSP ":" [CFWS] name-val-list [ ";" 
>> [CFWS] obs-date-time ] CRLF
>>
>> An incompatible change, but perhaps correct.
>
>
> Yes, I think Bruce fixes a bug in 2822 here.

Received has issues, both w.r.t. 2822 and 2821.

First an historical overview:  Received was first specified in RFC 821, 
also known as a
"time stamp line".  There have been, and still are, discrepancies 
between the 821/822 and
2821/2822 definitions of the field body.  Received is one of the trace 
fields, and as noted
in RFC 1123, was initially primarily examined by hand.  1123 permitted 
adding information
actually useful for tracing (i.e. the peer IP address, as opposed to the 
HELO string which is
far too easily forged), however gave no syntax for doing so [common 
practice has been to
include it in a comment].

Nowadays, tracing Received fields by hand is far too labor-intensive.  
Unfortunately, due
to the exponential growth of spam, it is necessary.  We now have the 
unfortunate situation
where the required information (at least in 2821) includes the 
easily-forged HELO/EHLO
string, and the useful not-so-easily-forged connection information is 
relegated to an optional
construct.  Worse, in 2821 that optional construct is specified as some 
sort of structured
comment, indeed it is indistinguishable from a comment.

What is really desirable is something that has reliable trace 
information in machine-readable
form, perhaps with the easily-forged information relegated to comments.  
But that's another
discussion...

Prior to 2822, a time stamp line (a.k.a. Received field) always required 
a time stamp. 2822
permits a time stamp line w/o a time stamp, which is an oxymoron.  I 
don't personally like
that (IMO the time stamp should be mandatory), but that's what 2822 
says, and that's part
of what is included in the syntax above.

Another issue, not included above, is an incompatibility which has crept 
in. 821/822 required
SP before the semicolon which delimits the start of the date-time.  
However, most MTAs
incorrectly omit that space [in some, such as sendmail, that can be 
easily rectified via a run-
time configuration patch; others, such as qmail, have that error 
hard-coded].  2821 requires
CFWS immediately before the semicolon, but 2822 makes it optional.  
Given 821/822/2821's
requirements, I'd be inclined to revise 2822 to make at least CFWS 
mandatory before the
semicolon when generating a message, and given past (clearly wrong, but 
quite widespread)
practice, I'd require being able to parse a Received field w/o CFWS 
before the semicolon
(via obs- syntax).

One more remaining incompatibility between 2821 and 2822 lies in the 
permitted constructs;
2821 permits a quoted string as an item value (via 2821's "String") 
whereas 2822 has no
such provision.  That shows up in the "id" component, which has a long 
history of conflicts
between 821/822 (1123 tried to rectify the conflict, but only added more 
confusion).

Yet another incompatibility is that 2821 permits a mix of angle-addrs 
(a.k.a. Paths) and
addr-specs (a.k.a. Mailboxes) in a "for" component, whereas 2822 permits 
a single addr-spec
or multiple angle-addrs (and no mixture).  It turns out that for rather 
complicated reasons,
the 2821 provision for multiple addr-specs is rather difficult to 
parse.  Perhaps 2821's
successor should address that issue; in any event, let's at least remove 
the remaining conflicts
between the 2821 and 2822 definitions one way or another.

I note also that there exist broken implementations which generate cruft 
that cannot be parsed
even with 2822's exceptionally liberal rules.  Here are some real-world 
examples:

That's one recent example; among the problems:
as noted above, the SP/CFWS-before-semicolon issue
id's other than properly-constructed msg-ids
RFC 821 does not permit day-of-week in the time stamp
missing from and/or by components in some cases
illegal (non-RFC 1700 cruft) in "with" components
there is no defined "Mail" item-name

That's even worse; additional problem is that parsing fails after the (illegal) with
component on encountering a lone "SMTPSVC" (and Microsoft was informed about that bug
in Windows 2000 well befor SP1; 3 service packs and as many years later and the bug
still hasn't been fixed (it shouldn't take more than 10 seconds for a competent
programmer to modify the source to a) use a legal value (ESMtp or SMTP) in the with
component, or b) elide the optional with component, or c) put the marketing BS in a
comment)...

For the record, I am NOT in favor of extending the syntax to accept such cruft -- I wish
that certain purveyors of brokenware would clean up their acts. 

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

From: Arnt Gulbrandsen <[email-masked]>
To: Pete Resnick <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Cc: [email-masked]
Date: Fri, 16 Jan 2004 10:24:36 +0100
Sender: [email-masked]

Pete Resnick writes:
>> 2. I think I found a case in which Bruce permits CFWS where it wasn't 
>>  before, although I cannot decipher my very cramped margin notes. It 
>> was something like "To: blah: (cfws) [email-masked];".
>
> 2822 allows CFWS there.

Yes, it was something like that. I remember I thought it was a really 
odd case. I'll see if I can find it again when the jetlag's gone.

Anyway, now that you've cleared up the things I misunderstood or didn't 
understand, I like Bruce's changes even better. (Thanks, Bruce.)

Arnt

Date: Fri, 16 Jan 2004 03:11:11 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (obs-time-of-day)
Sender: [email-masked]

Pete Resnick wrote:

>
> On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:
>
>> 4. obs-time-of-day looks wrong.
>>
>>   obs-time-of-day = hour [CFWS] ":" [CFWS] minute [CFWS] ":" [[CFWS]
>> second]
>>
>> Better:
>>
>>   obs-time-of-day = hour [CFWS] ":" [CFWS] minute [CFWS] [":" [CFWS] 
>> second]
>
>
> Yup, that looks better.

Agreed.

Date: Fri, 16 Jan 2004 02:54:39 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Revisiting RFC 2822 grammar (ASCII NUL)
Sender: [email-masked]

Pete Resnick wrote:

>
> On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:
>
>> In <[email-masked]> [email-masked] writes:
>
>>> [N.B. RFC 822 ASCII NUL not permitted, even with obs- rules]
>>
>>
>> Bruce gives many examples of differences from RFC 822. I will leave 
>> it to others to comment on the rights and wrongs, but some of them 
>> certainly look like bugs in RFC 2822 to me.
>
>
> I think Bruce is wrong on this one. I see ASCII NUL in 822's CHAR, and 
> that appears in ctext (minus a few characters).

But CHAR doesn't appear in 2822.  ASCII NUL is excluded from NO_WS_CTL, 
and so
cannot be in (2822's) ctext.  The point is that some legal 822 messages 
cannot be parsed
even with 2822's obs- rules.  The specific instance of (unescaped) ASCII 
NUL in a comment
is an example of that.

Date: Fri, 16 Jan 2004 02:43:45 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
CC: Pete Resnick <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, Charles Lindsey <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar (quoted-pair)
Sender: [email-masked]

Pete Resnick wrote:

>
> On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:
>
>> 1. Bruce elimites obs-qp, even though it can match a few pairs 
>> quoted-pair cannot, such as "\" DEL.
>
>
> Yup, I think that's Bruce's mistake.

and

> On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:
> In <[email-masked]> [email-masked] writes:
>
>> quoted-pair     =       ("\" text)
>> [N.B. had redundant obs-qp alternative]
>
>
> I think not. The obs version allows \NUL, \CR and \LF, which the 
> regular version does not.
> Right.

RFC 2822 gives quoted-pair as:

quoted-pair     =       ("\" text) / obs-qp

and text as:

text            =       %d1-9 /         ; Characters excluding CR and LF
                        %d11 /
                        %d12 /
                        %d14-127 /
                        obs-text

and obs-text (N.B. included in text) as:

obs-text        =       *LF *CR *(obs-char *LF *CR)

with obs-char defined as:

obs-char        =       %d0-9 / %d11 /          ; %d0-127 except CR and
                        %d12 / %d14-127         ;  LF

DEL is %d127, which is explicitly included in text and obs-char.  
Indeed, text explicitly includes
every US-ASCII character by value except NUL, CR, and LF, and includes 
those as well via obs-text
(explicitly permitting a single CR or LF) and  obs-char (which includes 
NUL).  I don't see  any single
US-ASCII character  which quoted-pair doesn't permit  after the 
backslash,  without having to resort
to obs-qp.

Maybe that wasn't intended by RFC 2822, but there it is.

As an aside, note that obs-text may consist of multiple octets, so 
"\foo" could be considered a
quoted-pair ("foo" matches obs-text via *(obs-char *LF *CR)).  I believe 
that's a problem with 2822.
On the other hand, 822 specifically mentioned the multi-character \CRLF 
and gave its semantics,
but 2822 doesn't seem to permit that.

Date: Thu, 15 Jan 2004 23:07:28 -0600
To: "Charles Lindsey" <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Cc: [email-masked]
Sender: [email-masked]

On 1/15/04 at 9:59 PM +0000, Charles Lindsey wrote:

>In <[email-masked]> [email-masked] writes:
>
>>quoted-pair     =       ("\" text)
>>[N.B. had redundant obs-qp alternative]
>
>I think not. The obs version allows \NUL, \CR and \LF, which the 
>regular version does not.

Right.

>>[N.B. RFC 822 ASCII NUL not permitted, even with obs- rules]
>
>Bruce gives many examples of differences from RFC 822. I will leave 
>it to others to comment on the rights and wrongs, but some of them 
>certainly look like bugs in RFC 2822 to me.

I think Bruce is wrong on this one. I see ASCII NUL in 822's CHAR, 
and that appears in ctext (minus a few characters).

>>zone            =       ( "+" / "-" ) 4DIGIT
>>[N.B. no CFWS between +- and 4DIGIT]
>
>Indeed. Are you saying that such was allowed in RFC 822?

It's not entirely clear, but it probably is.

>>message         =       (fields / obs-fields) [CRLF body]
>
>But this rule leads to horrendous ambiguities, with no prospect of 
>avoiding them in less than 50 pages of syntax :-( . But I shall 
>defer discussion of that till later, because there are other issues 
>with it.

This is identical to 2822.

>>subject         =       "Subject:" [FWS] [("cmsg" / "Re: ") [FWS]] 
>>unstructured CRLF
>>[ RFC 1036 sect. 2.2.6 "cmsg" Subject hack, sect. 2.1.4 "Re: " ]
>
>Please, no "Re: " or "cmsg" in the syntax.

Agreed.

>>path            =       ("<" [CFWS] [addr-spec] ">" [CFWS]) / obs-path
>
>I think it would be better to say
>
>path		= angle-addr / "<" [CFWS] ">" [CFWS]
>
>That way you avoid the need for obs-path (obs-angle-address takes care of it)

Seems OK to me.

>Now we come to the obs- syntax, where there are still many 
>ambiguities. As things stand, sometimes the obs- syntax allows 
>something that is already in the regular syntax (that is ambiguous). 
>OTOH, sometimes it does not, and sometimes it allows only a part of 
>what is in the regular syntax, all of which can be very confusing to 
>the reader who tries to work out exactly how the obs- syntax differs 
>from the regular.

I disagree completely. I think it's much easier for the reader to 
have some complete pieces in the obs- syntax even if there is 
redundancy. For example, I think the horrible hoop you have to jump 
through below for obs-local-part:

>obs-local-part	=	*(word ".") word "." CFWS word *("." [CFWS] word) /
>			*(atom ".") atom "." quoted-string *("." word) /
>			*(quoted-string ".") quoted-string "." atom 
>*("." word) /
>			1*(quoted-string ".") quoted-string

is just nuts. I don't think this is the correct approach, I think it 
makes the syntax completely unusable to a reader, and I would 
strongly object to anything like it.

I'm going to skip all of your examples of this.

>>obs-utext       =       *LF *CR *(obs-char *LF *CR)
>>[N.B. was obs-text]
>
>No, that does not work because it allows CRLF not followed by WSP in 
>the middle of an 'unstructured'.

Yup, because you can have "obs-utext obs-utext" which could be (abc 
CR)(LF def).

>I think the only way out of that is to rewrite the rule for 'unstructured':
>
>unstructured	=	*(utext [FWS]) obs-ltext
>obs-utext	=	(1*LF *CR / 1*CR) obs-char / NUL
>obs-ltext	=	*LF *CR

Blech. I'll take a look and see what I can figure out without 
resorting to that.

>>obs-phrase      =       word *(word / ("." [CFWS]))
>
>OK, but that is not "obsolete". It is intended as an extension to be 
>allowed sometime in the future on a "MUST accept, SHOULD NOT 
>generate yet" basis. So please can we rename it as 'extended-phrase' 
>(which is what I have currently put in Usefor).

I am not convinced this is worth it. It's explained perfectly well in the text.

>Currently, RFC2822 requires:
>
>1. Return-Path
>2. 1*Received
>3. *Resent-xxx
>4. Other headers

No, it doesn't. Look at the parens and the repeats. It requires:

*(*(return-path 1*(received)) *(resent-xxx))

followed by other headers.

>Yes, it is a good idea that tracing headers be added at the top, so 
>you can tell the order in which the message passed through various 
>agents, but there are some useful cases which have been excluded, 
>for example:
>
>Received: from D by E
>Received: from C by D
>Resent-To: bar@E
>Resent-From: foo@C
>Received: from B by C
>Received: from A by B

That's legal in 2822.

>Here is another example (a real one this time, which some readers of 
>uk.net.news.management may recognize):
>
>Received:  from lon-mail-1.gradwell.net (localhost [127.0.0.1])
>  by clerew.man.ac.uk (8.11.7+Sun/8.11.7) with ESMTP id i05HCjF01021
>  for <[email-masked]>; Mon, 5 Jan 2004 17:12:45 GMT
>Delivered-To:	postmaster@A
>Received:  (qmail 81124 invoked by uid 800); 5 Jan 2004 12:54:22 -0000
[...]

You're right, that's not allowed, and I think that is a bug that 
needs to be fixed.

>Now there are all sorts of perfectly genuine "tracing headers" in 
>there, all added in transit, and all useful.

So, likely we need optional-field to appear in trace. I think that's 
the logical answer.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Thu, 15 Jan 2004 22:26:46 -0600
To: Arnt Gulbrandsen <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Cc: ietf-822 <[email-masked]>
Sender: [email-masked]

On 1/15/04 at 7:09 PM +0100, Arnt Gulbrandsen wrote:

>1. Bruce elimites obs-qp, even though it can match a few pairs 
>quoted-pair cannot, such as "\" DEL.

Yup, I think that's Bruce's mistake.

>2. I think I found a case in which Bruce permits CFWS where it 
>wasn't  before, although I cannot decipher my very cramped margin 
>notes. It was something like "To: blah: (cfws) [email-masked];".

2822 allows CFWS there.

>3. The date syntax is reordered, and four-digit years seem to have 
>disappeared from the obsolete syntax.

Nope. 2*DIGIT means 2 or more digits, which is correct.

>4. obs-time-of-day looks wrong.
>
>   obs-time-of-day = hour [CFWS] ":" [CFWS] minute [CFWS] ":" [[CFWS]
>second]
>
>Better:
>
>   obs-time-of-day = hour [CFWS] ":" [CFWS] minute [CFWS] [":" [CFWS] second]

Yup, that looks better.

>5. obs-received merits discussion on its own. RFC 2822 says
>
>   obs-received = "Received" *WSP ":" name-val-list CRLF
>
>which Bruce changes to
>
>   obs-received = "Received" *WSP ":" [CFWS] name-val-list [ ";" 
>[CFWS] obs-date-time ] CRLF
>
>An incompatible change, but perhaps correct.

Yes, I think Bruce fixes a bug in 2822 here.

>6. I don't like Bruce's changes to subject. They mix in RFC 1036 
>syntax, which IMO does not belong in 2822.

I agree. These should be removed.

>7. There may be a few cases where FWS in 2822 is replced by CWFS in 
>Bruce's grammar. It's a little hard to tell. For example, in Bruce's 
>grammar there always is CFWS following field-name ":", I'm not sure 
>the same holds for 2822. It may.

I believe the only place where this is an issue is the Date: field, 
and he got that one correct. I see no problems on this one.

>8. obs-domain-list is changed in a way I don't understand, from
>
>   obs-domain-list = "@" domain *(*(CFWS / "," ) [CFWS] "@" domain)
>
>to
>
>   obs-domain-list = "@" [CFWS] domain *(1*("," [CFWS]) "@" [CFWS] domain)

This was a bug in 2822. It allowed "@[email-masked]", 
where there is a required comma before the second "@".

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

To: [email-masked]
Xref: clerew local.mime:2885
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Date: Thu, 15 Jan 2004 21:59:25 GMT
Lines: 334
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>Pete Resnick wrote:

>> I notice that the ABNF in there has a few things that are non-2822 
>> such as encoded words. Do you have a copy of the ABNF which is purely 
>> the 2822 replacement that we can post to the ietf-2822 list?

>I have an older version which does not have the encoded-word grammar 
>(full text below).

>Following is the text of the modified grammar w/o encoded-word grammar, 
>interspersed
>with some notes:
>
>rfc2822grammar_simplified.txt version 0.13 2001/08/08 16:02:35  
>excerpted from RFC 2822 and modified by Bruce Lilly
>
>
>quoted-pair     =       ("\" text)
>[N.B. had redundant obs-qp alternative]

I think not. The obs version allows \NUL, \CR and \LF, which the regular
version does not.

>[N.B. RFC 822 ASCII NUL not permitted, even with obs- rules]

Bruce gives many examples of differences from RFC 822. I will leave it to
others to comment on the rights and wrongs, but some of them certainly
look like bugs in RFC 2822 to me.

>atom            =       1*atext [CFWS]

And here starts the changes to remove ambiguities from the grammar.
Essentially, we want to get rid of cases where the grammar can produce two
CFWSs side by side (because it makes it harder for the human reader to
puzzle it out, and it makes it harder to derive automatic parsers, whether
LR(1) or otherwise).

When I first encountered this syntax, I convinced myself that it would be
more trouble than it was worth, and hugely increase the size of the
grammar, to remove these cases. Now Bruce has unconvinced me.

Essentially, he arranges that all the relevant constructs can be followed
by a CFWS (usually optional) but never preceded by one. So he has removed
around a dozen occurrences of [CFWS] from the syntax, and inserted around
30 new occurrences. AFAICS his revisions all work, and are worthwhile.

I have a few niggles, and there are still some other ambiguities which he has
not touched, which I will come to later.

>date-time       =       ([ day-name "," [FWS]] date FWS time [CFWS]) / 
>obs-date-time

All the obs- versions of month, day, etc are gone, and are replaced by a
single rule for obs-date-time. On the face of it, this is a good move, but
it leads to problems later on, as we shall see.

>zone            =       ( "+" / "-" ) 4DIGIT
>[N.B. no CFWS between +- and 4DIGIT]

Indeed. Are you saying that such was allowed in RFC 822?

>message         =       (fields / obs-fields) [CRLF body]

But this rule leads to horrendous ambiguities, with no prospect of
avoiding them in less than 50 pages of syntax :-( . But I shall defer
discussion of that till later, because there are other issues with it.

>subject         =       "Subject:" [FWS] [("cmsg" / "Re: ") [FWS]] 
>unstructured CRLF
>[ RFC 1036 sect. 2.2.6 "cmsg" Subject hack, sect. 2.1.4 "Re: " ]

AAARRRRRGGGGGGGGGHHHHHHHHHHH!

Please, no "Re: " or "cmsg" in the syntax. Quite apart from the introduced
ambiguity (the 'unstructured' could begin with those things anyway), the
"Re: " convention is better described by verbiage in the semantics. We
have removed it from the syntax of Usefor (and Bruce was one of the people
who urged that). It remains as a semantic convention with wording with
much the same effect as the wording currently in RFC 2822 (and even that
is not cast in concrete yet). As for "cmsg", that convention is no longer
implemented anywhere AFAIK; Usefor says it MUST NOT be used for any
semantic effect, though it still recommends putting it there for old
time's sake when a Control header is present (though I am now dubious
about even that).

>path            =       ("<" [CFWS] [addr-spec] ">" [CFWS]) / obs-path

I think it would be better to say

path		= angle-addr / "<" [CFWS] ">" [CFWS]

That way you avoid the need for obs-path (obs-angle-address takes care of
it)

Now we come to the obs- syntax, where there are still many ambiguities. As
things stand, sometimes the obs- syntax allows something that is already
in the regular syntax (that is ambiguous). OTOH, sometimes it does not,
and sometimes it allows only a part of what is in the regular syntax, all
of which can be very confusing to the reader who tries to work out exactly
how the obs- syntax differs from the regular.

It would be much better for each bit of obs- syntax to produce only the
extra bits which are not already in the regular stuff, and as I shall show
this is quite easily done.

>obs-qp          =       "\" (%d0-127)
>[N.B. unnecessary]

But if you do keep it, all it needs is:
obs-qp		= "\" ( NUL / LF / CR )

>obs-text        =       %d0-127

should be:
obs-text	= NUL / LF / CR

>obs-char        =       %d0-9 / %d11 /          ; %d0-127 except CR and
>                        %d12 / %d14-127         ;  LF
>

It would be clearer to say:
obs-char	=	utext / NUL / WSP

>obs-utext       =       *LF *CR *(obs-char *LF *CR)
>[N.B. was obs-text]

No, that does not work because it allows CRLF not followed by WSP in the
middle of an 'unstructured'. I think the only way out of that is to
rewrite the rule for 'unstructured':

unstructured	=	*(utext [FWS]) obs-ltext
obs-utext	=	(1*LF *CR / 1*CR) obs-char / NUL
obs-ltext	=	*LF *CR

>obs-phrase      =       word *(word / ("." [CFWS]))

OK, but that is not "obsolete". It is intended as an extension to be
allowed sometime in the future on a "MUST accept, SHOULD NOT generate yet"
basis. So please can we rename it as 'extended-phrase' (which is what I
have currently put in Usefor).

>obs-phrase-list =       phrase / (1*([phrase] "," [CFWS]) [phrase])

To be truly "obs-", that needs to contain at least one occurrence of two
"."s with no phrase between them. A syntax to achieve this would be:

obs-phrase-list	=	phrase *("," [CFWS] phrase)
			  2*("," [CFWS])
			  *(1*("," [CFWS]) phrase)

>obs-FWS         =       1*WSP *(CRLF 1*WSP)

should be:
obs-FWS		=	2*(*WSP CRLF 1*WSP)
>
>obs-date-time   =       [ day-name [CFWS] "," [CFWS]] obs-date [CFWS] 
>FWS [CFWS] obs-time [CFWS]

Essentially, if all the CFWS are, in fact, FWS, then it is regular;
otherwise, it is obs-. But it would be a pain to write it all as one rule
that way. However, if you were to restore the separate obs-month, obs-day,
etc as in RFC 2822, I think it could be done quite easily.

>obs-angle-addr  =       "<" [CFWS] [obs-route] addr-spec ">" [CFWS]

should be:
obs-angle-addr	=	"<" [CFWS] obs-route addr-spec ">" [CFWS]

>obs-local-part  =       word *("." [CFWS] word)

That is a tricky one. Essentially, a phrase can consist of a collection of
atoms and quoted-strings separated by ("." [CFWS]). To be considered
"obs", it has to contain, somewhere, either a genuine CFWS, or an
(atom "." quoted-string), or a (quoted-string "." atom), or a
(quoted-string "." quoted-string). If it does not have one of those
somewhere, it is regular. Here is the syntax to do it:

obs-local-part	=	*(word ".") word "." CFWS word *("." [CFWS] word) /
			*(atom ".") atom "." quoted-string *("." word) /
			*(quoted-string ".") quoted-string "." atom *("." word) /
			1*(quoted-string ".") quoted-string

>obs-domain      =       atom *("." [CFWS] atom)

should be:
obs-domain	=	dot-atom 1*("." CFWS dot-atom)

>obs-mbox-list   =       1*([mailbox] "," [CFWS]) [mailbox]

Needs same treatment as obs-phrase-list.

>obs-addr-list   =       1*([address] "," [CFWS]) [address]

Needs same treatment as obs-phrase-list.

>obs-fields      =       *(obs-return / obs-received / obs-orig-date / 
>obs-from / obs-sender / obs-reply-to / obs-to / obs-cc / obs-bcc / 
>obs-message-id / obs-in-reply-to / obs-references / obs-subject / 
>obs-comments / obs-keywords / obs-resent-date / obs-resent-from / 
>obs-resent-send / obs-resent-rply / obs-resent-to / obs-resent-cc / 
>obs-resent-bcc / obs-resent-mid / obs-optional)
>
>obs-orig-date   =       "Date" *WSP ":" [CFWS] date-time CRLF

If you retain the ambiguous syntax for 'message' then this rule, and the
following ones like it, are fine. But if you do away with that syntax (see
below), then they would need to be changed to

obs-orig-date	=	"Date" 1*WSP ":" [CFWS] date-time CRLF

in order to disambiguate them from their regular counterparts. Also, some
of them which include explicit obs- syntax would need some attention.

>obs-subject     =       "Subject" *WSP ":" [FWS] [("cmsg" / "Re:") 
>[FWS]] unstructured CRLF
>[ RFC 1036 sect. 2.2.6 "cmsg" hack, 2.1.4 "Re:" (w/ or w/o space) ]

But, please no, "Re: " or "cmsg".

>obs-path        =       obs-angle-addr

Not needed if my suggestion above for 'path' is adopted.

Not back to the message grammar:

>message         =       (fields / obs-fields) [CRLF body]

That is trying to kill two birds with one stone:
  1) to force some order into the regular headers;
  2) to allow WSP before the ":", and also a few extra obs- features

The problem is that everything that turns up in fields also turns up in
obs-fields. Hence it is ambiguous (and I doubt it is LR(1) either). And it
would be totally unfixable by writing further syntax (except as a
theoretical possibility. So the alternatives are
  a) Leave it ambiguous, and admit that it is so, or
  b) Enforce the ordering of the headers by verbiage, rather than
     syntactic means.

However, before doing either of those, please reconsider whether the
ordering you are trying to enforce is too rigid. Currently, RFC2822
requires:

1. Return-Path
2. 1*Received
3. *Resent-xxx
4. Other headers

Yes, it is a good idea that tracing headers be added at the top, so you
can tell the order in which the message passed through various agents, but
there are some useful cases which have been excluded, for example:

> Received: from D by E
Resent-To: bar@E
Resent-From: foo@C

IOW, why forbid keeping a record of how it travelled from its origin to
the place where it was resent?

Here is another example (a real one this time, which some readers of
uk.net.news.management may recognize):

Mailing-List: 	contact [email-masked]; run by ezmlm	
Reply-To: 	[email-masked]	
.....

Now there are all sorts of perfectly genuine "tracing headers" in there,
all added in transit, and all useful. Some of them are X-headers (so you
need the concept of an "X-tracing-header"). Some, like "Delivered-To"
probably should have been X-headers. Some of them, like the "List.*" ones
are properly defined by an RFC, just not by RFC 2822. And that "Reply-To"
in the middle was added by the mailing list expander, as its position
indicates.

It is non-conformant with RFC 2822 but, IMO, it ought not to be.

So what RFC 2822bis really needs is some careful discussion of what
tracing headers are and how they are to be added, and not a rigid syntax.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 15 Jan 2004 19:09:03 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

Hi,

I've now gone through Bruce Lilly's suggested changes to the RFC 2822 
grammar.

Generally, changes mostly seem to be removal of FWS and CFWS to the 
left of other tokens. For example, in "From: A b <[email-masked]>" 2822 
permits the space after b to be matched by either the trailing [CFWS] 
in the Atom rule or by leading one in angle-addr. Bruce changes the 
grammar such that there aren't leading CFWS/FWS invocations, and adds 
other CFWS/FWS invocations in other places to compensate.

In general I like his changes. (I didn't expect to. 2822 is not some 
draft-blah-blah-00 whose syntax may be changed lightly.) There are 
some details, though. As always.

1. Bruce elimites obs-qp, even though it can match a few pairs 
quoted-pair cannot, such as "\" DEL.

2. I think I found a case in which Bruce permits CFWS where it wasn't 
before, although I cannot decipher my very cramped margin notes. It 
was something like "To: blah: (cfws) [email-masked];".

3. The date syntax is reordered, and four-digit years seem to have 
disappeared from the obsolete syntax. This looks like a good idea:

  obs-year = 2DIGIT / 4DIGIT

4. obs-time-of-day looks wrong.

  obs-time-of-day = hour [CFWS] ":" [CFWS] minute [CFWS] ":" [[CFWS]
second]

Better:

  obs-time-of-day = hour [CFWS] ":" [CFWS] minute [CFWS] [":" [CFWS] second]

5. obs-received merits discussion on its own. RFC 2822 says

  obs-received = "Received" *WSP ":" name-val-list CRLF

which Bruce changes to

  obs-received = "Received" *WSP ":" [CFWS] name-val-list [ ";" [CFWS] obs-date-time ] CRLF

An incompatible change, but perhaps correct.

6. I don't like Bruce's changes to subject. They mix in RFC 1036 
syntax, which IMO does not belong in 2822.

7. There may be a few cases where FWS in 2822 is replced by CWFS in 
Bruce's grammar. It's a little hard to tell. For example, in Bruce's 
grammar there always is CFWS following field-name ":", I'm not sure 
the same holds for 2822. It may.

8. obs-domain-list is changed in a way I don't understand, from

  obs-domain-list = "@" domain *(*(CFWS / "," ) [CFWS] "@" domain)

to

  obs-domain-list = "@" [CFWS] domain *(1*("," [CFWS]) "@" [CFWS] domain)

I may have missed something; I did this on the airplane and finished 
typing it down after arriving 20 hours late. I do not like the Mumbai 
airport.

Arnt

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Getting 2822 to Draft
Date: Mon, 12 Jan 2004 13:47:52 +0100
Sender: [email-masked]

Charles Lindsey writes:
>>> why not post the revised ABNF as an I-D so that others can go over 
>>> it with a fine tooth comb?
>>
>> Anyone done so yet? Keith? Charles? Anyone?
>
> Sorry. Not enough Round Tuits yet.

FWIW, I plan on doing it on Wednesday.

Arnt

To: [email-masked]
Xref: clerew local.mime:2883
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Mon, 12 Jan 2004 10:15:38 GMT
Lines: 21
Sender: [email-masked]

In <p0610073bbc2636eb4ba8@[216.43.25.67]> Pete Resnick <[email-masked]> writes:

>On 1/2/04 at 10:02 AM -0500, Keith Moore wrote:

>>why not post the revised ABNF as an I-D so that others can go over 
>>it with a fine tooth comb?

>Anyone done so yet? Keith? Charles? Anyone?

Sorry. Not enough Round Tuits yet.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Sat, 10 Jan 2004 17:12:34 -0600
To: Keith Moore <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Getting 2822 to Draft
Cc: Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

On 1/2/04 at 10:02 AM -0500, Keith Moore wrote:

>why not post the revised ABNF as an I-D so that others can go over 
>it with a fine tooth comb?

Anyone done so yet? Keith? Charles? Anyone?

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

To: [email-masked]
Subject: Re: X-* header fields
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Wed, 07 Jan 2004 20:41:39 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> And also in NNTP where there are widely deployed commands such as XOVER
> and XHDR. Nevertheless, the new NNTP draft if defining new commands OVER
> and HDR even though their syntax/semantics is 99.9% identical.

In both cases, fixing some long-standing problems in the process.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Wed, 07 Jan 2004 10:00:36 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields
To: Keith Moore <[email-masked]>
Cc: Russ Allbery <[email-masked]>, Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

> >> Now IMHO, the use of X- in X-Face shouldn't prevent us from
> >> documenting
> >> X-Face in an RFC, nor even from making it a standard - but only if we
> >> really had consensus to choose that name and format.
> >
> > And there's actually a much nicer and more modern Face header
> > specification that would be more suitable for standardization.  I think
> > X-Face is really more of an informational RFC at this point, not a
> > direction we'd want to move towards going forward.

> I guess I'm still looking for that canonical example of a read-write X-
> field that really, really should be standardized as-is.  Or maybe I've
> forgotten it.

X-Accept-Language comes very close IMO.

				Ned

To: [email-masked]
Xref: clerew local.mime:2879
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: X-* header fields
Date: Wed, 7 Jan 2004 10:58:39 GMT
Lines: 54
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Here's another take on the subject (I hope it doesn't open up several cans
>of worms, but here goes...)

>X- is used as an indicator for experimental or private-use tags in many 
>places
>in Internet message protocols other than field names. For example:

>charsets (RFC 2978, sect. 3.1, also RFC 2046, sect. 4.1.2)
>MIME media types (RFC 2045, sections 5 & 5.1, also RFC 2046, section 6) 
>and subtypes
>language tags (RFC 3066, section 2.2)
>disposition-notification options (RFC2298 section 2.2 )
>transfer encodings (RFC 2045, sect. 6.3)
>MTA-name-types (RFC 3464 sect. 2.2)
>access-types (RFC 2046 sect 5.2.3.1)
>address-types (RFC 2298 sect. 3.1.2, also RFC 3464 sect. 2.2)
>dispositions (RFC 2183 sect 2.8)
>diagnostic types (RFC 3464 sect. 2.2)
>disposition modifiers (RFC 2298 sect 3.2.6.3)

And generally speaking for tokens to introduce new parameters in
Content-Type and other headers that use the same parameter syntax.

And also in NNTP where there are widely deployed commands such as XOVER
and XHDR. Nevertheless, the new NNTP draft if defining new commands OVER
and HDR even though their syntax/semantics is 99.9% identical.

>And is mentioned in Keith's auto-response draft for auto-submitted types 
>-- and I don't recall
>any discussion about that here when that draft was recently discussed.

>And of course there are MDN X- fields (RFC 2298 sect. 3.3) and DSN X- 
>fields (RFC 3464
>sect. 2.4).

>So what's so abhorrent about top-level X- fields (RFC 822) that doesn't 
>also apply to
>charsets, media types, language tags, etc.?

I agree. There is a widespread understanding that these X-things don't get
standardized as such.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: X-* header fields
Date: Tue, 6 Jan 2004 18:13:01 -0500
To: Russ Allbery <[email-masked]>
Sender: [email-masked]

>> Now IMHO, the use of X- in X-Face shouldn't prevent us from 
>> documenting
>> X-Face in an RFC, nor even from making it a standard - but only if we
>> really had consensus to choose that name and format.
>
> And there's actually a much nicer and more modern Face header
> specification that would be more suitable for standardization.  I think
> X-Face is really more of an informational RFC at this point, not a
> direction we'd want to move towards going forward.

I guess I'm still looking for that canonical example of a read-write X- 
field that really, really should be standardized as-is.  Or maybe I've 
forgotten it.

To: [email-masked]
Subject: Re: X-* header fields
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 06 Jan 2004 14:39:16 -0800
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> Now IMHO, the use of X- in X-Face shouldn't prevent us from documenting
> X-Face in an RFC, nor even from making it a standard - but only if we
> really had consensus to choose that name and format.

And there's actually a much nicer and more modern Face header
specification that would be more suitable for standardization.  I think
X-Face is really more of an informational RFC at this point, not a
direction we'd want to move towards going forward.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 06 Jan 2004 11:57:47 -0500
From: John C Klensin <[email-masked]>
To: [email-masked]
cc: Pete Resnick <[email-masked]>
Subject: Re: Getting 2822 to Draft
Sender: [email-masked]

--On Tuesday, 06 January, 2004 09:07 -0500 Bruce Lilly 
<[email-masked]> wrote:

> Pete Resnick wrote:
>
>>
>> So, for the new year I started thinking about getting 2822 to
>> Draft.  (Also, 3822 is coming up in the RFC numbers :-) ).
>> I've gotten started  on getting a new draft together. There
>> are a small bunch of nits to  fix; that I can handle. There's
>> also an implementation report to  write. On that I'd like to
>> get some help.
>>
>> The one possible big thing has to do with the ABNF in 2822.
>> Out of  either altruism or insanity, some time ago Bruce
>> Lilly had written up  changes to the ABNF in 2822 to do some
>> cool things. On the plus side,  it seems to get rid of all of
>> the [C]FWS shift-reduce conflicts, and  it is already done.
>> On the minus side, I don't know anyone (myself  included) who
>> has gone over it with a fine tooth comb, it is a  significant
>> number of ABNF changes, and it therefore might recycle us  at
>> Proposed. I am open to suggestions on this.
>>
>> Comments?
>>
>> pr
>
> As
> a) 2821 and 2822 are closely related
> and
> b) there remain a few inconsistencies between 2821 and 2822
> is there a draft of a 2821 successor that can be reviewed at
> the same time
> as the draft of the 2822 successor?

The editor of 2821 also made a new year's resolution, and 
actually opened the document last week to incorporate more of 
the changes that have been suggested.   The bad news is that the 
working copy is in Word XP. I'm not a Word fan -- all of my 
other I-Ds are in XML or directly edited in ASCII-- but because, 
after the experiences with DRUMS, I wanted to be sure that I 
could identify, in the working text, the source and 
justification for all of the requested changes if any 
controversy arose.   The difficulty is that converting a Word 
document with as many comments, change markup, internal cross 
references, etc., as that one now has is far beyond the scope of 
RFC 3285 (the attempt was what produced the "RFC 3285 considered 
harmful" thread on the IETF list a year or so ago).  So, I'm 
reluctant to go through the rather painful conversion until I 
have a complete draft.   If you would like to look directly at 
the Word form, or a PDF display of it (our experience has been 
that Word XP documents with extensive comments and markup don't 
get along with Mac Word either), I think that could be arranged 
within the next week or two.

    john

Date: Tue, 06 Jan 2004 09:07:07 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: John C Klensin <[email-masked]>
CC: Pete Resnick <[email-masked]>, [email-masked]
Subject: Re: Getting 2822 to Draft
Sender: [email-masked]

Pete Resnick wrote:

>
> So, for the new year I started thinking about getting 2822 to Draft. 
> (Also, 3822 is coming up in the RFC numbers :-) ). I've gotten started 
> on getting a new draft together. There are a small bunch of nits to 
> fix; that I can handle. There's also an implementation report to 
> write. On that I'd like to get some help.
>
> The one possible big thing has to do with the ABNF in 2822. Out of 
> either altruism or insanity, some time ago Bruce Lilly had written up 
> changes to the ABNF in 2822 to do some cool things. On the plus side, 
> it seems to get rid of all of the [C]FWS shift-reduce conflicts, and 
> it is already done. On the minus side, I don't know anyone (myself 
> included) who has gone over it with a fine tooth comb, it is a 
> significant number of ABNF changes, and it therefore might recycle us 
> at Proposed. I am open to suggestions on this.
>
> Comments?
>
> pr

As
a) 2821 and 2822 are closely related
and
b) there remain a few inconsistencies between 2821 and 2822
is there a draft of a 2821 successor that can be reviewed at the same time
as the draft of the 2822 successor?

#################################################################
#################################################################
#################################################################
#####
#####
#####
#################################################################
#################################################################
#################################################################

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: X-* header fields
Date: Tue, 6 Jan 2004 08:14:01 -0500
To: [email-masked] (Kai Henningsen)
Sender: [email-masked]

>>> But *why* would an implementation care about the publication
>>> guidelines for the field? That's the *only* thing "X-" tells you.
>>
>> because any field that doesn't have a published definition probably
>> isn't something that should be implemented on a widespread basis -
>> since without a definition you have no reason to expect it to work
>> predictably from one implementation to another.
>>
>> admittedly, this inference might be a bit subtle.  I see no harm in
>> making it more explicit.
>
> It's not so much subtle as it is false.
>
> Counterexamples:
>
> X-Face: - there's every expectation of this interoperating just fine.

Note that I said "probably".  The fact that there are a small number of 
counterexamples doesn't make it false.  Yes, it can be made to 
interoperate on any platform, but if we were to standardize a field for 
the author's face, we might or might not choose this particular field 
name, image format, and encoding.

Now IMHO, the use of X- in X-Face shouldn't prevent us from documenting 
X-Face in an RFC, nor even from making it a standard - but only if we 
really had consensus to choose that name and format.

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Cc: [email-masked], Pete Resnick <[email-masked]>, [email-masked]
Date: Tue, 6 Jan 2004 13:38:31 +0100
Sender: [email-masked]

Keith Moore writes:
> Sigh. I'd hoped we could avoid this discussion, but it seems like 
> people can't resist focusing on it to the detriment of attention to 
> more substantive issues. (Or maybe this really is the most 
> substantive issue in 2822, and we should regard this debate as a sign 
> that we don't have anything worse to worry about?)

I don't think said substantive issues will be discussed until there is a 
draft, or at least an suggested list of changes. (I saw Bruce's 
grammar, and will go through it next Thursday.)

--Arnt

Date: Tue, 06 Jan 2004 09:21:46 +0000
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Sender: [email-masked]

FWIW (NAL..)

My view on X- header fields is that they're a good thing if used properly - 
but there should probably be a registry for optional fields in the long term.

RFC 2822 section 3.6.8 leaves things wide open IMHO - I could legitimately, 
according to RFC 2822, create my own header field called 'Content-Type' and 
put anything I want into it - because the only restriction in RFC 2822 is 
"The field names of any optional-field MUST NOT be identical to any field 
name specified elsewhere in *this* standard." Content-Type isn't defined in 
RFC 2822, so I can use it..

If I want to create my own header for my own use (which happens) and I 
don't have an exhaustive knowledge of all the RFCs, it's quite possible I 
could inadvertently re-use a field name which already exists in a standard.

Also, I may use a custom field name which doesn't already exist, but then 
someone else comes along and makes a new standard using the same name but 
in a different way.

At the moment, there's no definitive way to avoid this, other than to use 
an X- header (Yes, I could use '<MTA Name>-header' - which I have no 
problem with - *if* this was 'standardised' as a way of defining custom 
headers)

If there was a 'registry' of headers where I could register either field 
names which I think would be useful to the community in general, or an MTA 
name which I could then append my own custom field names to, then I'd be happy.

But, until that happens, I think there needs to be a way you can GUARANTEE 
that a field name someone invents does not clash with a standard name. The 
X- headers achieve that aim IMHO.

(Also, a minor point, but possibly significant none-the-less. Moderately 
technical people who aren't email experts can generally accept that 'X-' 
headers are custom headers. Without this distinction they can assume 
they're standard. So, if an email parser can't parse, for instance, 
'Status:' or 'Delivered-To:' headers, there can be user support issues 
because 'they're not custom headers, why can't your parser understand 
them?', whereas if they were X- headers this misunderstanding would be less 
likely, or if they are standardised properly, the parser could understand them)

JM2P

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: 05 Jan 2004 22:12:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: X-* header fields
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Keith Moore)  wrote on 05.01.04 in <[email-masked]>:

> > But *why* would an implementation care about the publication
> > guidelines for the field? That's the *only* thing "X-" tells you.
>
> because any field that doesn't have a published definition probably
> isn't something that should be implemented on a widespread basis -
> since without a definition you have no reason to expect it to work
> predictably from one implementation to another.
>
> admittedly, this inference might be a bit subtle.  I see no harm in
> making it more explicit.

It's not so much subtle as it is false.

Counterexamples:

X-Face: - there's every expectation of this interoperating just fine.
HowDoYouDo: - no such expectation whatsoever

"X-" doesn't actually tell you what you claim it does.

MfG Kai

Date: Tue, 06 Jan 2004 03:16:22 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

Here's another take on the subject (I hope it doesn't open up several cans
of worms, but here goes...)

X- is used as an indicator for experimental or private-use tags in many 
places
in Internet message protocols other than field names. For example:

charsets (RFC 2978, sect. 3.1, also RFC 2046, sect. 4.1.2)
MIME media types (RFC 2045, sections 5 & 5.1, also RFC 2046, section 6) 
and subtypes
language tags (RFC 3066, section 2.2)
disposition-notification options (RFC2298 section 2.2 )
transfer encodings (RFC 2045, sect. 6.3)
MTA-name-types (RFC 3464 sect. 2.2)
access-types (RFC 2046 sect 5.2.3.1)
address-types (RFC 2298 sect. 3.1.2, also RFC 3464 sect. 2.2)
dispositions (RFC 2183 sect 2.8)
diagnostic types (RFC 3464 sect. 2.2)
disposition modifiers (RFC 2298 sect 3.2.6.3)

And is mentioned in Keith's auto-response draft for auto-submitted types 
-- and I don't recall
any discussion about that here when that draft was recently discussed.

And of course there are MDN X- fields (RFC 2298 sect. 3.3) and DSN X- 
fields (RFC 3464
sect. 2.4).

So what's so abhorrent about top-level X- fields (RFC 822) that doesn't 
also apply to
charsets, media types, language tags, etc.?

And yet another feature of X- fields:
RFC 2047, section 5 notes that encoded-words may appear in X- fields.  
In all
other fields, it is necessary to know the syntax of the field in order 
to know
whether and where encoded-words are permitted.  I.e. when displaying fields,
one may decode an encoded word in an appropriate place in a field body 
if the
field syntax is known, and one may decode an encoded-word in any X- field
(presumed to be unstructured), but one may not decode anything which has
the form of an encoded-word in an unrecognized field (other than an X- 
field).

Date: Tue, 06 Jan 2004 02:18:25 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
CC: [email-masked]
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Sender: [email-masked]

Pete Resnick wrote:

>
> On 1/4/04 at 2:54 AM -0500, Bruce Lilly wrote:
>
>> Pete Resnick wrote:
>>
>>> The only difference then between 822 and 2822 in this respect is 
>>> that 822 gives publication guidelines for extensions where 2822 does 
>>> not. This has absolutely *no* effect on implementations of the 
>>> protocol.
>>
>>
>> There is a minor effect. A user-defined field (per 822 definition) 
>> can be recognized as such by examining the first two octets of the 
>> field name, which is quite efficient.
>
>
> But *why* would an implementation care about the publication 
> guidelines for the field? That's the *only* thing "X-" tells you.

An implementation *might* be written so as to hand off parsing of X- 
fields to a separate
procedure, or ignore them entirely.  And the ability to efficiently 
bypass checking against
the 100+ standardized field names *might* be considered important.

[my implementation does hand off most X- fields (one notable exception 
being X-Accept-Language)
and unrecognized (in this case anything not standardized in an RFC or an 
Internet Draft that
seems likely to become an RFC) to optional user-supplied functions. 
Separate functions may
be specified for unrecognized X- (user-defined) fields and for 
unrecognized extension fields
(or the application can point to a single function).  Were it not for 
handling X-Accept-Language,
I might have elected to bypass lookup for standard field names (via 
gperf) in the case of user-
defined fields.]

>> "X-" serves to differentiate user-defined fields from non-defined 
>> fields (i.e. those field names for which there is no IETF published 
>> definition or which the implementation does not recognize).
>
>
> Again, what use is there in making that distinction? Do you think the 
> fact that "X-Priority" and "X-Face" start with "X-" means that you 
> shouldn't support them in your implementation or they don't have 
> well-defined syntax? What about "X-Sender"?

I don't support them because I haven't found a specification for them in 
any RFC.

> Is "List-ID" a non-defined field if your implementation doesn't 
> recognize it? What useful purpose is there in differentiating fields 
> which start with "X-"?

I do recognize and support List-ID (RFC 2919).  Flexibility is one 
reason to differentiate;
because RFC 822 made the distinction, it is conceivable that some 
message-processing
application may wish to treat user-defined and non-standard extension 
fields differently.
By providing for separate function pointers for the two cases, an 
application using the mparse
library can either treat them differently (using distinct function 
pointers) or treat them
identically (by providing two pointers to a single function).

>> In order to be at all useful to implementors, registration would have 
>> to be contingent upon the existence of a stable, formal, public 
>> definition of the proposed field's syntax (with ABNF) and semantics.
>
>
> Nonsense. It is useful to an implementation (especially an 
> implementation that generates fields) to know that a field exists for 
> a particular purpose even if its syntax and semantics have not 
> undergone extensive review and comment or are still under development. 
> It is by implementation that fields get stable and then they can be 
> documented as standards.

Example:
somebody registers a "Foo" field, but provides no public syntax or 
semantics. As an implementor,
what am I supposed to do about "Foo"?

>> It's difficult to see how such problems are either unique to X- or 
>> are more of a hindrance to interoperability than for other fields. As 
>> an example, consider "Status" which was in "private" use (I believe) 
>> by BSD "mailx" decades ago, and which is currently in use by several 
>> other MUAs, and which *does* leak out; there is also a formal 
>> definition of a "Status" field -- incompatible with the private usage 
>> -- defined as one of the delivery status notification fields (RFC 
>> 3464). So neither leakage nor incompatibility seem to be unique to X- 
>> fields. Note that if BSD mailx' author(s) had used X-Status for 
>> private use, there would be no conflict with the formal DSN Status 
>> field.
>
>
> First of all, the "Status" field of DSN is not defined to appear in a 
> top-level header of a [2]822 message, so there is no "incompatibility" 
> between the two. But let's talk about the leakage: Yes, the "Status" 
> field leaks. Now, what can be done about that? Well, it could be 
> documented in an RFC so that everyone could know what it means. And if 
> turned out useful, it could be a standardized top-level header field. 
> But what if mailx's author had instead used "X-Status"? In that case, 
> it's DOA, because by definition it could not be documented in a 
> standard way. So there is no way for an implementor to figure out what 
> "X-Status" means other than by word of mouth. That invites 
> incompatibility. So what exactly would that "X-" have gained you?

My implementation -- and I believe that it is not unique in this respect 
-- parses a field with
a given name depending on that name and not specifically on its 
context.  The incompatibility
is that the DSN Status field has a specific syntax for the field body 
(viz. three dot-separated
numbers) which is not matched by the BSD et al usage of Status (a string 
of alphabetic characters).
A single field name with two different syntax definitions (and different 
semantics) would IMO
be a bad thing, even if theoretically they could be differentiated by 
context; header fields do
occasionally end up in message bodies due to some software inserting an 
empty line, and that
empty line may cause top-level message headers moved into the body to 
appear to be MIME-part
fields (i.e. if the message header still contains an appropriate 
Content-Type field).  Mailx uses
Status to store state metadata about the mail store, which is a private 
use. X-Status could be
documented (e.g. via an Informative RFC), but not registered with IANA 
as an extension field,
though *as* private use, there would be no reason to document it (given 
that X- guarantees no
collision with any future registered extension field).  The only reason 
for an implementor to care
about a hypothetical X-Status field is if said implementor had reason to 
interact with a message
store on a system that also used mailx with that message store -- that's 
a storage issue, not a message
format or transmission issue and therefore an issue outside of the scope 
of IETF.  So X-Status would
not "invite incompatibility" in any sense in which it matters to IETF.

>> There is one very good reason to use X- for private or experimental 
>> use, viz. interoperability. Use of X- as a field name prefix for 
>> private or experimental use guarantees that there will be no conflict 
>> with a formal field name. Use of other names can lead to conflicts
>
>
> No it doesn't. Some "X-" field names are now just as formal as some 
> non-"X-" field names.[...]

"[J]ust as formal"? Really? Which X- field names are defined in 
Standards-track RFCs? For that
matter which X- field names are defined in *any* RFC?  Or by "non-'X-' 
field names" do you
mean something other than registered extension field names?

>> An issue not mentioned above is migration of X- fields to a formal 
>> definition following successful experimental use. That obviously 
>> entails a name change. However name changes are not uncommon, e.g. 
>> refer to the MIXER RFCs, which define a number of fields whose names 
>> have changed. That merely means that parsers need to recognize one 
>> name as a synonym for another.
>
>
> But what about generators? Because there will be parsers out there 
> that will only interpret the "X-" form of the field, the generators 
> must continue to send the "X-" form. Furthermore, updating some 
> parsers is non-trivial.

By the same argument, generators will have to generate multiple versions 
with a single non-X- name
as syntax changes during experimentation.  Presumably those 
participating in an experiment have a
vested interest in moving forward with an official implementation at the 
end of the period of experimentation.

Whether or not some parsers are difficult to update is irrelevant to the 
issue; if a parser can be updated to
handle Obsoletes/Supersedes, Expiry-Date/Expires, 
Content-Identifier/X400-Content-Identifier, etc., then
it can be updated to handle X-Accept-Language/Accept-Language.  
Conversely if it cannot be updated
to handle X-Accept-Language/Accept-Language, then presumably it cannot 
handle the other (standard)
fields whose field names have been changed.

>
> pr

Date: Tue, 06 Jan 2004 00:34:15 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Kai Henningsen <[email-masked]>
CC: [email-masked]
Subject: Re: X-* header fields
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------030203070503070001010801
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Kai Henningsen wrote:

>[email-masked] (Bruce Lilly)  wrote on 04.01.04 in <[email-masked]>:
>
>  
>
>>Pete Resnick wrote:
>>
>>    
>>
>>>This has absolutely *no* effect on implementations of
>>>the protocol.
>>>      
>>>
>>There is a minor effect. A user-defined field (per 822 definition) can
>>be recognized as such by examining the
>>first two octets of the field name, which is quite efficient.  While
>>there exist efficient methods of identifying
>>fields (e.g. http://www.gnu.org/directory/gperf.html), they are not
>>quite *as* efficient as a two-octet (case-
>>insensitive) comparison.
>>    
>>
>
>This seems an incredibly silly reason to be for or against the concept.
>  
>
Is *isn't* "a reason to be for or against the concept." It is an example 
of an
effect on implementation to counter the claim of "absolutely *no* effect 
on implementations".

>>An issue not mentioned above is migration of X- fields to a formal
>>definition following
>>successful experimental use.  That obviously entails a name change.
>>However name changes
>>are not uncommon, e.g. refer to the MIXER RFCs, which define a number of
>>fields whose
>>names have changed.  That merely means that parsers need to recognize
>>one name as a
>>synonym for another.
>>    
>>
>
>Or in other words, it means that interoperation with the installed base is  
>broken. Presumably that installed base is nontrivial, or else there would  
>have been no need to standardize. ("merely"?!)
>  
>
Yes, "merely".  As noted, parsers already need to be able to cope with 
name changes:
Obsoletes (RFC 1327) -> Supersedes (RFC 2156)
Content-Identifier (RFC 1327) -> X400-Content-Identifier (RFC 2156)
Expiry-Date (RFC 1327) -> Expires (RFC 2156)
etc., and
X-Accept-Language (MUA usage) -> Accept-Language (RFC 3066)

--------------030203070503070001010801
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Kai Henningsen wrote:
<blockquote cite="[email-masked]" type="cite">
  <pre wrap=""><a class="moz-txt-link-abbreviated" href="mailto:[email-masked]">[email-masked]</a> (Bruce Lilly)  wrote on 04.01.04 in <a class="moz-txt-link-rfc2396E" href="mailto:[email-masked]">&lt;[email-masked]&gt;</a>:

  </pre>
  <blockquote type="cite">
    <pre wrap="">Pete Resnick wrote:

    </pre>
    <blockquote type="cite">
      <pre wrap="">This has absolutely *no* effect on implementations of
the protocol.
      </pre>
    </blockquote>
    <pre wrap="">There is a minor effect. A user-defined field (per 822 definition) can
be recognized as such by examining the
first two octets of the field name, which is quite efficient.  While
there exist efficient methods of identifying
fields (e.g. <a class="moz-txt-link-freetext" href="http://www.gnu.org/directory/gperf.html">http://www.gnu.org/directory/gperf.html</a>), they are not
quite *as* efficient as a two-octet (case-
insensitive) comparison.
    </pre>
  </blockquote>
  <pre wrap=""><!---->
This seems an incredibly silly reason to be for or against the concept.
  </pre>
</blockquote>
Is *isn't* "a reason to be for or against the concept." It is an
example of an<br>
effect on implementation to counter the claim of "absolutely *no*
effect on implementations".<br>
<br>
<blockquote cite="[email-masked]" type="cite">
  <blockquote type="cite">
    <pre wrap="">An issue not mentioned above is migration of X- fields to a formal
definition following
successful experimental use.  That obviously entails a name change.
However name changes
are not uncommon, e.g. refer to the MIXER RFCs, which define a number of
fields whose
names have changed.  That merely means that parsers need to recognize
one name as a
synonym for another.
    </pre>
  </blockquote>
  <pre wrap=""><!---->
Or in other words, it means that interoperation with the installed base is  
broken. Presumably that installed base is nontrivial, or else there would  
have been no need to standardize. ("merely"?!)
  </pre>
</blockquote>
Yes, "merely".&nbsp; As noted, parsers already need to be able to cope with
name changes:<br>
Obsoletes (RFC 1327) -&gt; Supersedes (RFC 2156)<br>
Content-Identifier (RFC 1327) -&gt; X400-Content-Identifier (RFC 2156)<br>
Expiry-Date (RFC 1327) -&gt; Expires (RFC 2156)<br>
etc., and<br>
X-Accept-Language (MUA usage) -&gt; Accept-Language (RFC 3066)<br>
<br>
</body>
</html>

--------------030203070503070001010801--

Date: Mon, 5 Jan 2004 17:02:01 -0500
From: Keith Moore <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Sender: [email-masked]

Sigh.  I'd hoped we could avoid this discussion, but it seems like
people can't resist focusing on it to the detriment of attention to more
substantive issues.  (Or maybe this really is the most substantive issue
in 2822, and we should regard this debate as a sign that we don't have
anything worse to worry about?)

> The idea behind X-* fields (what 822 called user-defined fields) was 
> so that folks could use new field names without worrying that they 
> would interfere with the standard fields. That way, you didn't have 
> to go through the trouble of publishing an RFC just to use a field 
> that was only going to be internal to your particular system. The 
> downside was that someone else could come along and potentially use 
> the same field name in a completely incompatible way, but that was 
> the risk that you took.

I won't claim that I can make statements about the original idea behind
X- fields, since I wasn't in on the discussions at the time.  However
there are other useful properties of X- fields that derive from the 822
specification, specifically that the use of X- serves as a notice that
the field is nonstandard  (and thus it hasn't received the review and
scrutiny that a standard would have received), that it never will be a
standard, that IF it somehow becomes widely used there is a good chance 
it will not be consistently used, and that you should therefore avoid
assuming that an X- field means what you think it means.

These properties *are* useful for flagging experimental or local
features. And it's useful to have such a flag precisely because such
fields will leak outside of their experimental or local environment. 
The fact that these fields leak should not cause a problem as long as
(a) the extra fields don't change the meaning of other fields and (b)
the software that creates the extra fields doesn't assume that either
recipient's UAs or intermediares will understand those fields.  (These
rules should generally be followed by all header fields, not just
extension fields that begin with X-.)  But IF you follow the 822
specification, the fact that a field-name begins with X- is an
indication that you shouldn't expect it to work on a widespread basis,
and that's a useful indication.

But this does mean that X- fields shouldn't be used by widely-deployed
software, except perhaps for "write only" fields like X-User-Agent
that mostly serve as comments or indications to human recipients.
 
> But since then, two things have made it clear that such user-defined 
> fields are unnecessary and problematic:
> 
> 1. Unnecessary: We now have IANA registration services. We could very 
> simply create an IANA registry of e-mail field names. Then, if you 
> wanted to create a new field, you just fill out the registration form 
> and you're not only guaranteed that you won't interfere with 
> published standard fields, you're also guaranteed that nobody else 
> would use the field you chose in an incompatible way. And you 
> wouldn't have to have a name that started with "X-".

Even when we have header registration, this won't make X- fields
unnecessary.  Sometimes it's just too much trouble to get a field name
registered, especially if it's just for some experiment or local
feature.  

That, and it's too easy to just code new field names into software
without registering them.  If you're writing code and you find that 
you need a new field, are you really going to take the page fault
to get that field name registered, or are you just going to type
printf ("new-field-name: whatever\r\n"); and be done with it?
If we give implementors a convention that let's them say "this field
isn't intended for widespread use" then at least some of those
implementors will be responsible and use that convention, which is far
better than not having a convention at all and encouraging implementors
to allocate field names on a whim.  (URI prefix space is a huge mess
because implementors have done precisely that.)

And if we really registered every field name for every
local or experimental extension, we'd soon find that the field names we
wanted to use for extensions that are worthy of standardization, were
already allocated by half-baked ideas not worthy of standardization. 
Yet the half-baked ideas would have the appearance of legitimacy, since
they'd have the "better" or "more obvious" names.  We'd also lose the
useful ability to mark extensions as "local" or "experimental".
(though this could be accomodated by a different naming convention)

> 2. Problematic: Some folks thought it was nice to have "X-*" fields 
> for completely private use. But history teaches us that inevitably 
> "X-*" fields *will* leak out onto the rest of the Internet. As 
> several people in this thread have pointed out, we now have some 
> "X-*" fields that are in widespread use. Sometimes, people use them 
> very interoperably and they have served a fabulous purpose. 
> Sometimes, they don't get used interoperably. It would be great if we 
> could publish some standards to make sure that they do get used 
> interoperably. But guess what: We can't!  It is forbidden by RFC 822 
> to publish any standard for a field name that beings with "X-". No 
> matter how widespread the use, no matter how useful the field is, it 
> can not be standardized *by definition*. That decreases 
> interoperability on the Internet.

The cases where you really want to standardize a field whose name begins
with X- are rare, because it's rare that someone manages to define a new
field in such a way that it's suitable for widespread use without
benefit of a published spec and wide review before deployment of that 
feature.  But I don't read RFC 822 as preventing publication of
information about X- fields. Indeed RFC 2076 does mention some X-
fields, so clearly it's possible to publish information about X- fields
with Informational status.

For most cases where the X- field turns out to really be what we want to
standardize, what we should do is publish a new standard with a new
field name, document the previous usage in a non-normative section of
the standard, and describe how senders and receivers should cope with
the two fields.

But if there were a case where there was a compelling reason to
standardize the use of an existing X- field, then I wouldn't object to
that specific field being treated as an exceptional case.   We should do
the right thing for interoperability and functionality even if it wasn't
what 822 anticipated. But we should make it clear that this is an
exceptional case and not the general rule.  If we make X- fields the
same as any other fields then we are optimizing for the rare case, and
at the same time doing harm to our ability to extend the mail protocol.

Keith

Date: Mon, 05 Jan 2004 10:03:09 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> [email-masked] writes:

> >> On 1/3/04 at 11:00 AM -0500, Al Costanzo wrote:

> >> 1. Unnecessary: We now have IANA registration services. We could very
> >> simply create an IANA registry of e-mail field names.

> >This is already underway and the document specifying it -
> >draft-klyne-msghdr-registry-06.txt - has been discussed on this list. I note in
> >passing that it doesn't distinguish X- fields as being in any way special
> >either.

> Yes, I am surprised that people have been going over issues which the
> Klyne draft has already resolved. I think it will be best to get that
> draft adopted as a BCP and the IANA register created before starting
> serious work on 2822-bis. But I hope that will have happened within
> months, if not sooner.

I agree 100%. The current status of the registry draft is that it has
completed last call and has been discussed by the IESG. There were
two discuss votes on the specification and the authors have posted
as revised version that hopefully addresses the issues that needed to 
be addressed. However, there were a couple of concerns it isn't clear
how to address so the document remains under discussion.

> > An interesting question is whether or not 2822bis should reference this
> > registry, assuming the specification for it is approved in the near future
> > (which I hope it will be).

> Yes, I think it should. There is a corresponding place in the Usefor draft
> where it discusses X-headers (and when not to use them), and the wording
> there would be much simpler if it could just point to the Klyne BCP.

OK.

> >> Then, if you
> >> wanted to create a new field, you just fill out the registration form
> >> and you're not only guaranteed that you won't interfere with
> >> published standard fields, you're also guaranteed that nobody else
> >> would use the field you chose in an incompatible way. And you
> >> wouldn't have to have a name that started with "X-".

> >> 2. Problematic: Some folks thought it was nice to have "X-*" fields
> >> for completely private use. But history teaches us that inevitably
> >> "X-*" fields *will* leak out onto the rest of the Internet. As
> >> several people in this thread have pointed out, we now have some
> >> "X-*" fields that are in widespread use. Sometimes, people use them
> >> very interoperably and they have served a fabulous purpose.
> >> Sometimes, they don't get used interoperably. It would be great if we
> >> could publish some standards to make sure that they do get used
> >> interoperably. But guess what: We can't! It is forbidden by RFC 822
> >> to publish any standard for a field name that beings with "X-". No
> >> matter how widespread the use, no matter how useful the field is, it
> >> can not be standardized *by definition*. That decreases
> >> interoperability on the Internet.

> >I agree with this as well.

> Bur I don't. What you say is that X-headers are for human readers only -
> essentially they give the possibility for creating lots of extra
> Comment-headers. But you also say that they may also be used for protocol
> purposes ONLY where there is prior agreement between sender and recipient.

I see nothing in the previous discussion having anything to do with whether
the fields are used by humans or programs. The issue of whether or not
something is standardizable has nothing to do with who the eventual
consumer is.

> That is not so improbable as it sounds. If my ISP is kind enough to put
> all my mail through Spamassassin and record the result in the form:

>     X-Spamassassin-Score: sssss

> then I can use that header as a filter to send high scoring messages to a
> special box, or even to /dev/null. Essentially, there is an agreement
> (unwritten) between the ISP and myself that the header will appear and
> that I can rely on that format.

Sure, there are loads of examples of both sorts of X- fields (and both
sorts of non-X- fields for that matter).

				Ned

To: [email-masked]
Xref: clerew local.mime:2863
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Date: Mon, 5 Jan 2004 10:18:54 GMT
Lines: 74
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> On 1/3/04 at 11:00 AM -0500, Al Costanzo wrote:

>> 1. Unnecessary: We now have IANA registration services. We could very
>> simply create an IANA registry of e-mail field names.

>This is already underway and the document specifying it -
>draft-klyne-msghdr-registry-06.txt - has been discussed on this list. I note in
>passing that it doesn't distinguish X- fields as being in any way special
>either.

Yes, I am surprised that people have been going over issues which the
Klyne draft has already resolved. I think it will be best to get that
draft adopted as a BCP and the IANA register created before starting
serious work on 2822-bis. But I hope that will have happened within
months, if not sooner.

>An interesting question is whether or not 2822bis should reference this
>registry, assuming the specification for it is approved in the near future
>(which I hope it will be).

Yes, I think it should. There is a corresponding place in the Usefor draft
where it discusses X-headers (and when not to use them), and the wording
there would be much simpler if it could just point to the Klyne BCP.

>> Then, if you
>> wanted to create a new field, you just fill out the registration form
>> and you're not only guaranteed that you won't interfere with
>> published standard fields, you're also guaranteed that nobody else
>> would use the field you chose in an incompatible way. And you
>> wouldn't have to have a name that started with "X-".

>> 2. Problematic: Some folks thought it was nice to have "X-*" fields
>> for completely private use. But history teaches us that inevitably
>> "X-*" fields *will* leak out onto the rest of the Internet. As
>> several people in this thread have pointed out, we now have some
>> "X-*" fields that are in widespread use. Sometimes, people use them
>> very interoperably and they have served a fabulous purpose.
>> Sometimes, they don't get used interoperably. It would be great if we
>> could publish some standards to make sure that they do get used
>> interoperably. But guess what: We can't! It is forbidden by RFC 822
>> to publish any standard for a field name that beings with "X-". No
>> matter how widespread the use, no matter how useful the field is, it
>> can not be standardized *by definition*. That decreases
>> interoperability on the Internet.

>I agree with this as well.

Bur I don't. What you say is that X-headers are for human readers only -
essentially they give the possibility for creating lots of extra
Comment-headers. But you also say that they may also be used for protocol
purposes ONLY where there is prior agreement between sender and recipient.

That is not so improbable as it sounds. If my ISP is kind enough to put
all my mail through Spamassassin and record the result in the form:

    X-Spamassassin-Score: sssss

then I can use that header as a filter to send high scoring messages to a
special box, or even to /dev/null. Essentially, there is an agreement
(unwritten) between the ISP and myself that the header will appear and
that I can rely on that format.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2862
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Mon, 5 Jan 2004 10:05:24 GMT
Lines: 44
Sender: [email-masked]

In <p06100723bc1aa0f50ab0@[10.0.2.4]> Pete Resnick <[email-masked]> writes:

>So, for the new year I started thinking about getting 2822 to Draft. 
>(Also, 3822 is coming up in the RFC numbers :-) ). I've gotten 
>started on getting a new draft together. There are a small bunch of 
>nits to fix; that I can handle. There's also an implementation report 
>to write. On that I'd like to get some help.

Well if we are collecting small things that need to be changed, there is
the matter of compatibility with Netnews. Usefor has tried to bring the
syntax of Netnews headers into line with RFC 2822, and has mostly
succeeded, but there are one to two nits which remain different, and where
it is plain impractical for News to change. Therefore, there are a few
features currently valid in RFC 2822 that I would like to move into the
obsolete syntax. None of them is ever encountered in current email
practice that I have seen, so no one will notice their loss.

So, in due course, I would like to start a separate thread to discuss
these.

>The one possible big thing has to do with the ABNF in 2822. Out of 
>either altruism or insanity, some time ago Bruce Lilly had written up 
>changes to the ABNF in 2822 to do some cool things. On the plus side, 
>it seems to get rid of all of the [C]FWS shift-reduce conflicts, and 
>it is already done. 

One of the problems with this is likely to be that, although it is known
to be theoretically possible to construct a BNF grammar for any
context-free language, in practice it leads to humanly-unreadable grammars
in many cases, and I suspect this may be one such.

However, I see that Bruce has now posted his grammar, so I shall study it
carefully.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Date: Mon, 5 Jan 2004 00:22:03 -0500
To: Pete Resnick <[email-masked]>
Sender: [email-masked]

> But *why* would an implementation care about the publication 
> guidelines for the field? That's the *only* thing "X-" tells you.

because any field that doesn't have a published definition probably 
isn't something that should be implemented on a widespread basis - 
since without a definition you have no reason to expect it to work 
predictably from one implementation to another.

admittedly, this inference might be a bit subtle.  I see no harm in 
making it more explicit.

Keith

Date: Mon, 5 Jan 2004 01:36:40 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Pete Resnick <[email-masked]> wrote:

> In 2822bis, I think it would be fine to clarify that and say
> "User-defined header fields (those starting with 'X-') no longer
> specifically discussed",

I think that might be a little too terse.  I think it could be a lot
clearer with slightly more explanation:

    There is no longer a distinction made between User-defined header
    fields (which start with "X-") and Extension header fields (which
    don't).  Both are now treated uniformly as Optional header fields.

AMC

Date: 04 Jan 2004 14:12:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: X-* header fields
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Bruce Lilly)  wrote on 04.01.04 in <[email-masked]>:

> Pete Resnick wrote:
>
> > X-* fields are perfectly legal in 2822 (they fall under
> > "optional-field"); they simply are not given the special treatment of
> > 822, that they won't ever be official published extensions to the
> > standard. The only difference then between 822 and 2822 in this
> > respect is that 822 gives publication guidelines for extensions where
> > 2822 does not. This has absolutely *no* effect on implementations of
> > the protocol.
>
> There is a minor effect. A user-defined field (per 822 definition) can
> be recognized as such by examining the
> first two octets of the field name, which is quite efficient.  While
> there exist efficient methods of identifying
> fields (e.g. http://www.gnu.org/directory/gperf.html), they are not
> quite *as* efficient as a two-octet (case-
> insensitive) comparison.

This seems an incredibly silly reason to be for or against the concept.

>  Moreover, "X-" serves to differentiate
> user-defined fields from non-defined
> fields (i.e. those field names for which there is no IETF published
> definition or which the implementation
> does not recognize).

And this looks like a distinction in search of a difference.

> It's difficult to see how such problems are either unique to X- or are
> more of a hindrance to
> interoperability than for other fields. As an example, consider "Status"
> which was in "private" use
> (I believe) by BSD "mailx" decades ago, and which is currently in use by
> several other MUAs,
> and which *does* leak out; there is also a formal definition of a
> "Status" field -- incompatible
> with the private usage -- defined as one of the delivery status
> notification fields (RFC 3464).
> So neither leakage nor incompatibility seem to be unique to X- fields.
> Note that if BSD mailx'
> author(s) had used X-Status for private use, there would be no conflict
> with the formal DSN
> Status field.

I would have thought the proper solution to this were obvious. I've  
certainly see it used by various *other* MUAs.

That is, you use fieldnames of "<yourmuaname>-<tag>:" (or similar).

In this case, mailx could easily have used "Mailx-Status:" and been safe  
from this particular trap. (ISTR Pegasus calls this "PM-Flags:" or  
something like that. XP (the DOS MUA I'm using right now) has various  
"XP<something>:" headers.

Note that this actually works *better* than X-headers, in that someone  
else with the same idea won't reuse the same field name with a slightly  
different semantic.

> There is one very good reason to use X- for private or experimental use,
> viz. interoperability.
> Use of X- as a field name prefix for private or experimental use
> guarantees that there will be
> no conflict with a formal field name.   Use of other names can lead to
> conflicts, as in the case
> of "Status", the damage in that case fortunately being limited by the
> fact that in that case, DSN
> fields are significant only in one part of a specific type of MIME
> multipart message and the
> private use occurs only in the top-level message header.

Note that the above convention does this even better.

> An issue not mentioned above is migration of X- fields to a formal
> definition following
> successful experimental use.  That obviously entails a name change.
> However name changes
> are not uncommon, e.g. refer to the MIXER RFCs, which define a number of
> fields whose
> names have changed.  That merely means that parsers need to recognize
> one name as a
> synonym for another.

Or in other words, it means that interoperation with the installed base is  
broken. Presumably that installed base is nontrivial, or else there would  
have been no need to standardize. ("merely"?!)

MfG Kai

Date: Sun, 4 Jan 2004 12:12:07 -0600
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Sender: [email-masked]

On 1/4/04 at 2:54 AM -0500, Bruce Lilly wrote:

>Pete Resnick wrote:
>
>>The only difference then between 822 and 2822 in this respect is 
>>that 822 gives publication guidelines for extensions where 2822 
>>does not. This has absolutely *no* effect on implementations of the 
>>protocol.
>
>There is a minor effect. A user-defined field (per 822 definition) 
>can be recognized as such by examining the first two octets of the 
>field name, which is quite efficient.

But *why* would an implementation care about the publication 
guidelines for the field? That's the *only* thing "X-" tells you.

>"X-" serves to differentiate user-defined fields from non-defined 
>fields (i.e. those field names for which there is no IETF published 
>definition or which the implementation does not recognize).

Again, what use is there in making that distinction? Do you think the 
fact that "X-Priority" and "X-Face" start with "X-" means that you 
shouldn't support them in your implementation or they don't have 
well-defined syntax? What about "X-Sender"? Is "List-ID" a 
non-defined field if your implementation doesn't recognize it? What 
useful purpose is there in differentiating fields which start with 
"X-"?

>In order to be at all useful to implementors, registration would 
>have to be contingent upon the existence of a stable, formal, public 
>definition of the proposed field's syntax (with ABNF) and semantics.

Nonsense. It is useful to an implementation (especially an 
implementation that generates fields) to know that a field exists for 
a particular purpose even if its syntax and semantics have not 
undergone extensive review and comment or are still under 
development. It is by implementation that fields get stable and then 
they can be documented as standards.

>It's difficult to see how such problems are either unique to X- or 
>are more of a hindrance to interoperability than for other fields. 
>As an example, consider "Status" which was in "private" use (I 
>believe) by BSD "mailx" decades ago, and which is currently in use 
>by several other MUAs, and which *does* leak out; there is also a 
>formal definition of a "Status" field -- incompatible with the 
>private usage -- defined as one of the delivery status notification 
>fields (RFC 3464). So neither leakage nor incompatibility seem to be 
>unique to X- fields. Note that if BSD mailx' author(s) had used 
>X-Status for private use, there would be no conflict with the formal 
>DSN Status field.

First of all, the "Status" field of DSN is not defined to appear in a 
top-level header of a [2]822 message, so there is no 
"incompatibility" between the two. But let's talk about the leakage: 
Yes, the "Status" field leaks. Now, what can be done about that? 
Well, it could be documented in an RFC so that everyone could know 
what it means. And if turned out useful, it could be a standardized 
top-level header field. But what if mailx's author had instead used 
"X-Status"? In that case, it's DOA, because by definition it could 
not be documented in a standard way. So there is no way for an 
implementor to figure out what "X-Status" means other than by word of 
mouth. That invites incompatibility. So what exactly would that "X-" 
have gained you?

>There is one very good reason to use X- for private or experimental 
>use, viz. interoperability. Use of X- as a field name prefix for 
>private or experimental use guarantees that there will be no 
>conflict with a formal field name. Use of other names can lead to 
>conflicts

No it doesn't. Some "X-" field names are now just as formal as some 
non-"X-" field names. And I can come up with a list of non-"X-" field 
names for which it would be *much* safer to have two incompatible 
implemenations than some "X-" field names. The only thing the "X-" 
guarantees you is long, ongoing conflicts. Using a non-"X-" field 
name means that you can document one use as *the* standard way to do 
something.

>An issue not mentioned above is migration of X- fields to a formal 
>definition following successful experimental use. That obviously 
>entails a name change. However name changes are not uncommon, e.g. 
>refer to the MIXER RFCs, which define a number of fields whose names 
>have changed. That merely means that parsers need to recognize one 
>name as a synonym for another.

But what about generators? Because there will be parsers out there 
that will only interpret the "X-" form of the field, the generators 
must continue to send the "X-" form. Furthermore, updating some 
parsers is non-trivial.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sun, 4 Jan 2004 11:10:44 -0600
To: ietf-822 <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

On 1/4/04 at 12:34 AM -0500, Bruce Lilly wrote:

>I have an older version which does not have the encoded-word grammar 
>(full text below).

Thanks for sending it.

>From an implementor's perspective, I'd like to see all of the 
>relevant base grammar (i.e.
>base field and supporting grammar) in a single document; indeed, one 
>of the benefits of
>2822 is that it consolidated most of the "... amends RFC 822" 
>piecemeal details into a
>single document (obviously, the 2047/2231 amendments somehow didn't 
>make it into
>2822).

One of the rules we lived under during DRUMS (the WG that produced 
2822) was that we would not include anything from MIME so that this 
document could (if it needed to) make it to full Standard before MIME 
did. (You can't make normative reference to standards lower on the 
standards track.) Once both the base mail format and 2047/2231 make 
it to full Standard, I think it would be wise to combine them into a 
single document.

>I don't believe there is any harm in including the encoded-word grammar as
>encoded-words appear in the higher-level constructs as alternatives 
>to ccontent, word,
>and utext.

Except that there would have to be explanation of these terms 
referring to 2047/2231. And this would surely be more than a simple 
syntactic change to 2822. I think this is out of the question.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Sun, 04 Jan 2004 04:36:10 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: IETF RFC-822 list <[email-masked]>
Organization: Bruce Lilly
To: Arnt Gulbrandsen <[email-masked]>
Cc: IETF RFC-822 list <[email-masked]>
Subject: Re: Getting RFC 2047 encoding right
Sender: [email-masked]

Arnt Gulbrandsen wrote:

>
> Hi,
>
> I have a tiny little problem that you chaps may be able to help me with.
>
> Suppose that a mail client receives a message whose subject is encoded 
> according to RFC 2047. Suppose further that the message is decoded and 
> stored in a database somewhere - in RAM, on disk, on a server 
> somewhere. Some time later, the same or a different program sends a 
> reply.
>
> Now, to be kind and courteous that program should use the same subject 
> field, perhaps prefaced by "Re: " (or "Auto: "), such that if the 
> recipient threads based on subject, everything work, no matter whether 
> the recipient supports RFC 2047 or not.
>
> That implies using the same character set, q/b encoding etc. as the 
> original.
>
> To be kind and courteous, the program should use a widely supported 
> character set when 2047-encoding (whether it's composing original 
> messages or replies), and use as few 2047-encoded words as possible.
>
> To be kind and courteous, the program should observe all other rules 
> set out in RFC 2047 and 2822 (including ones which were broken by an 
> earlier message).
>
> To be reliable and bug-free, the algorithm that does all this should 
> be simple and straightforward.
>
> There's a conflict here. How do you all address this?

In reverse order:

The nature of the problem precludes a truly simple algorithm; it's a 
complex issue.

Some errors can be repaired; others cannot. Attempts to repair errors 
might be
more-or-less successful. Successful repair is contingent on being able to
unambiguously determine what was intended.

As far as practicable, original content should be preserved. E.g. in a 
reply, the address
given in the original message's Reply-To (or From) field should be used 
verbatim
(same case, same display name if present, including the same 
encoded-words if used)
in the To field of the reply.

I would extend that to the Subject field, and go so far as to say that 
"Re:", "Auto:",
etc. are best avoided. Incidentally, collating (colloquially "sorting") 
by subject is
not the same as "threading"; the latter entails use of References and/or 
In-Reply-To
fields with Message-ID fields to follow a related "thread" of messages 
(Consider a
collection of 10 messages with "Subject: Help" and 50 with "Subject: Re: 
Help" --
collating by Subject (with or w/o stripping "Re: ") doesn't group 
responses with
originals).

Non-transient storage of a message is best done in RFC 2822/MIME format,
possibly with lossless compression if the tradeoff between space and 
compression/
decompression effort warrants it, and possibly with encryption where 
necessary or
desirable. That does not preclude some additional metadata regarding the 
message,
but the original message ought to be 100% recoverable for use in 
replies, etc. Even
conversion of CRLF to local line endings can be troublesome (consider a 
multipart
MIME message with a binary part containing the octets 0x0D 0x0A).

In <vyDCOksjW0T5JTUD+zj/[email-masked]>:

> If origmsg->subject is "=?latin_1?q?=80?=" and the user doesn't change 
> the subject, newmsg->subject is "Re: =?latin_1?q?=80?=". If the 
> decoder knows whether the string could have been generated by a 
> reasonably conservative generator, that case can be avoided. A very 
> tricky decision.

Observing RFC 2047 rules, "=?latin_1?q?=80?=" should remain unchanged -- 
it's
NOT an encoded-word. RFC 2047 section 3 requires that the charset name 
be one that
is allowed in a MIME charset parameter for media type text/plain or that 
it be registered
for use with text/plain. The rules for text/plain are in RFC 2046 
section 4.1.2 which
states (in part):

" No character set name other than those defined above may be used in
Internet mail without the publication of a formal specification and
its registration with IANA, or by private agreement, in which case
the character set name must begin with "X-".
"

The "defined above" text refers to the us-ascii and iso-8859-X charsets. 
"latin_1"
is not registered nor is it in the initial set of MIME-compatible 
charsets in RFC 2046
(all of which are now registered), and it obviously does not begin with 
"X-",
therefore the RFC 822 atom containing "latin_1" is NOT an encoded-word. It
should be displayed verbatim and should remain unchanged.

In <[email-masked]>:

> Suppose the original message had "Subject: =?latin_1?q?The price is 
> =80216". The MUA fuzzily matches latin_1 to the IANA-defined alias 
> latin1, knows about the Microsoft breakage, and presents the user with 
> "Subject: The price is €216".

That's where things go wrong. There's no encoded-word -- 
"=?latin_1q?The" and
"=80216" should be displayed verbatim. Even if the spaces in the subject 
were replaced
with underscores, and the subject ended with "?=" -- as would be the 
case in a real
encoded-word -- the subject would have to be displayed verbatim as it 
still would not
contain a valid encoded-word. And as Keith Moore has pointed out, if it 
so happened
that "latin_1" were valid, but not recognized by the MUA in question, it 
should still
be displayed verbatim (because that MUA has no way to know what to 
display). In
this case the "fuzzily matches" is wrong, and two wrongs don't make a right.

> The reply is a message from the user to the recipient(s), and should 
> faithfully encode whatever the user saw and typed. The MUA SHOULD NOT 
> substitute some other text of unknown meaning for its user's text.

And that's exactly *why* "fuzzily matches" is wrong -- it involves 
substituting text
of unknown meaning for the original text.

In <[email-masked]>:

> Something like this, usually:
>
> a = new QLineEdit(...); // http://doc.trolltech.com/3.0/qlineedit.html
> if ( reply )
> a->setText( "Re: " + orig->subject() );
> else if ( forwarding )
> a->setText( "Fwd: " + orig->subject() );
> a->show();

Why the special case for "Fwd: "? Where is that standardized? Why not 
"FW: "?

> If the editor were internal the MUA could do 2047-decoding for display 
> purposes and keep the raw data as its basic storage. But since the 
> editor is external, the MUA must do 2047-decoding and hand the result 
> to the editor. Later, when the editor hands it back, the "obvious" way 
> is to 2047-encode the editor's result use it. Then there's only one 
> encoder to write and test, and it's used for original messages, for 
> forwarding and for replies. Less to write, less to test, fewer bugs.

Editing of the subject field is contrary to "to be kind and courteous 
that program should use
the same subject field". That's a design decision for an MUA author. 
Clearly, eliminating
editing means "[l]ess to write, less to test, fewer bugs". If editing is 
desired, that need not
mean that the entire field is decoded, then re-encoded; real-world 
editing usually means that
some portion(s) of the text are added, deleted, or changed, while much 
is unchanged.

"Re: " is an interesting case. To be kind, courteous, and RFC 2277 
conforming, one should
indicate the language. So that should probably be "=?us-ascii*la?q?Re:?= 
" (or the B
encoded equivalent, and/or using the ISO 3-letter code for Latin, "lat", 
and/or using any other
MIME-compatible charset, and/or another capitalization variant...). And 
the reply should
have appropriate References and In-Reply-To fields, assuming that the 
original had a
Message-ID.

Subject is supposed to be an unstructured field, but things like "Re: " 
impose unnecessary
and useless structure. Consider
Subject: FW: Sv: Fwd: Re^2: =?us-ascii*en-us?q?Auto:_?= 
=?iso-8859-1*lat?q?Re:?= RE: Auto: cmsg sendsys
Do you really want to have to be able to recognize and handle every type 
of hack to the subject
field, in every possible combination of capitalization, in encoded or 
unencoded form? What
do any of them indicate that isn't already evident via MIME packaging, 
Resent- fields, References,
In-Reply-To, and Message-ID fields, and/or Auto-Submitted fields?

Date: Sun, 04 Jan 2004 02:54:29 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Pete Resnick <[email-masked]>
Cc: [email-masked]
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Sender: [email-masked]

Pete Resnick wrote:

> X-* fields are perfectly legal in 2822 (they fall under 
> "optional-field"); they simply are not given the special treatment of 
> 822, that they won't ever be official published extensions to the 
> standard. The only difference then between 822 and 2822 in this 
> respect is that 822 gives publication guidelines for extensions where 
> 2822 does not. This has absolutely *no* effect on implementations of 
> the protocol.

There is a minor effect. A user-defined field (per 822 definition) can 
be recognized as such by examining the
first two octets of the field name, which is quite efficient.  While 
there exist efficient methods of identifying
fields (e.g. http://www.gnu.org/directory/gperf.html), they are not 
quite *as* efficient as a two-octet (case-
insensitive) comparison.  Moreover, "X-" serves to differentiate 
user-defined fields from non-defined
fields (i.e. those field names for which there is no IETF published 
definition or which the implementation
does not recognize).

> Now, let me use this opportunity to explain *why* I don't think X-* 
> fields should be in the standard in the first place:
>
> The idea behind X-* fields (what 822 called user-defined fields) was 
> so that folks could use new field names without worrying that they 
> would interfere with the standard fields. That way, you didn't have to 
> go through the trouble of publishing an RFC just to use a field that 
> was only going to be internal to your particular system. The downside 
> was that someone else could come along and potentially use the same 
> field name in a completely incompatible way, but that was the risk 
> that you took.
>
> But since then, two things have made it clear that such user-defined 
> fields are unnecessary and problematic:
>
> 1. Unnecessary: We now have IANA registration services. We could very 
> simply create an IANA registry of e-mail field names. Then, if you 
> wanted to create a new field, you just fill out the registration form 
> and you're not only guaranteed that you won't interfere with published 
> standard fields, you're also guaranteed that nobody else would use the 
> field you chose in an incompatible way. And you wouldn't have to have 
> a name that started with "X-".

Yes, but we don't yet have such a field name registry (and obviously we 
didn't have one in 2001 when
2822 was published).  In order to be at all useful to implementors, 
registration would have to be contingent
upon the existence of a stable, formal, public definition of the 
proposed field's syntax (with ABNF) and
semantics.  That differs from publication via RFC regarding review and 
comment; and I'm not convinced
that foregoing such review and comment would be a good thing.

> 2. Problematic: Some folks thought it was nice to have "X-*" fields 
> for completely private use. But history teaches us that inevitably 
> "X-*" fields *will* leak out onto the rest of the Internet. As several 
> people in this thread have pointed out, we now have some "X-*" fields 
> that are in widespread use. Sometimes, people use them very 
> interoperably and they have served a fabulous purpose. Sometimes, they 
> don't get used interoperably. It would be great if we could publish 
> some standards to make sure that they do get used interoperably. But 
> guess what: We can't! It is forbidden by RFC 822 to publish any 
> standard for a field name that beings with "X-". No matter how 
> widespread the use, no matter how useful the field is, it can not be 
> standardized *by definition*. That decreases interoperability on the 
> Internet.

It's difficult to see how such problems are either unique to X- or are 
more of a hindrance to
interoperability than for other fields. As an example, consider "Status" 
which was in "private" use
(I believe) by BSD "mailx" decades ago, and which is currently in use by 
several other MUAs,
and which *does* leak out; there is also a formal definition of a 
"Status" field -- incompatible
with the private usage -- defined as one of the delivery status 
notification fields (RFC 3464).
So neither leakage nor incompatibility seem to be unique to X- fields.  
Note that if BSD mailx'
author(s) had used X-Status for private use, there would be no conflict 
with the formal DSN
Status field.

> Now, personally, I would love the document to say, "There is no good 
> reason to use fields that begin with 'X-' as defined by RFC 822; any 
> name that does not interfere with a currently registered or published 
> field name is fine, and using an 'X-' field is discouraged because 
> they can't ever be standardized." But there was widespread 
> non-consensus in the DRUMS Working Group when this was discussed. 2822 
> does say "Extension header fields no longer specifically called out." 
> In 2822bis, I think it would be fine to clarify that and say 
> "User-defined header fields (those starting with 'X-') no longer 
> specifically discussed", and even have an informational reference to a 
> document (or 2) on why it was pulled. But I see absolutely no reason 
> that this should stop 2822bis from moving to full Standard and 
> obsoleting 822.

There is one very good reason to use X- for private or experimental use, 
viz. interoperability.
Use of X- as a field name prefix for private or experimental use 
guarantees that there will be
no conflict with a formal field name.   Use of other names can lead to 
conflicts, as in the case
of "Status", the damage in that case fortunately being limited by the 
fact that in that case, DSN
fields are significant only in one part of a specific type of MIME 
multipart message and the
private use occurs only in the top-level message header.

An issue not mentioned above is migration of X- fields to a formal 
definition following
successful experimental use.  That obviously entails a name change. 
However name changes
are not uncommon, e.g. refer to the MIXER RFCs, which define a number of 
fields whose
names have changed.  That merely means that parsers need to recognize 
one name as a
synonym for another.

I don't see the X- issue as any reason to keep 2822 from progressing; 
from a practical point of
view (as noted by others), parsers will need to be able to handle the X- 
fields already in use,
reading "old" messages still requires parsers to be able to handle RFC 
822 syntax -- and that
will continue to be the case even after 822 is eventually obsoleted 
regardless of what 2822
et succ. have to say about the matter, just as it is now necessary to 
support at least some
RFC 733 and 724 constructs to be able to read really "old" messages even 
though those RFCs
have been formally obsolete for decades.

Date: Sun, 04 Jan 2004 01:46:41 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
Cc: Pete Resnick <[email-masked]>, [email-masked]
Subject: Re: Getting 2822 to Draft
Sender: [email-masked]

Keith Moore wrote:

>
>> The one possible big thing has to do with the ABNF in 2822. Out of 
>> either altruism or insanity, some time ago Bruce Lilly had written up 
>> changes to the ABNF in 2822 to do some cool things. On the plus side, 
>> it seems to get rid of all of the [C]FWS shift-reduce conflicts, and 
>> it is already done. On the minus side, I don't know anyone (myself 
>> included) who has gone over it with a fine tooth comb, it is a 
>> significant number of ABNF changes, and it therefore might recycle us 
>> at Proposed. I am open to suggestions on this.
>
I can't comment on insanity (catch-22).  I decided to build a 
grammar-based parser because
I was tired of writing (and rewriting) application-specific 822 
parsers.  I started a couple
of months after 2822 was published, which was a big help primarily 
because of the formal
and explicit use of FWS and CFWS vs. 822's rather vague text.

> if the new grammar accepted the same language as the old one, I'd 
> consider it nothing more than a clarification of the spec, and a reset 
> to Proposed would be unnecessary.
>
> or if the only differences could be seen as minor bug fixes to the old 
> grammar, I'd consider it a minor bug fix to the spec.

The intent is to accept the same message text as the existing 2822 
grammar, modulo some minor
glitches in 2822.

> IMHO simplifying the grammar and getting rid of parsing conflicts 
> would both be highly desirable improvements.  if it then becomes 
> possible to feed the ABNF to a parser generator and generate a 
> verifier for 2822 messages, that's a huge win.

Whether it's "simpler" is perhaps debatable.  Shift/reduce conflicts are 
eliminated.  Generating
a verifier for 2822 [822/733/724/561/MIME/etc.] isn't quite as easy as 
feeding the ABNF to
a parser generator, but it is possible to feed something quite close to 
the ABNF to a parser
generator (http://users.erols.com/blilly/mparse). And that parser 
generator can be used to build
a message verifier.  The complete grammar (2822/822/MIME/various 
extensions/most of 733,
724, 561) has no unresolved shift/reduce conflicts and a single 
reduce/reduce conflict (due to
changing grammar of the MIXER Supersedes (formerly Obsoletes) field, 
resolved in favor of
the current (RFC 2156) definition).

> why not post the revised ABNF as an I-D so that others can go over it 
> with a fine tooth comb?  or has this already been done, and I missed it?

An earlier version of the revised ABNF has been posted to ietf-822 and 
the full version
(with encoded-word grammar) is available at the above URI.  Let the 
combing begin.

Date: Sun, 04 Jan 2004 01:42:34 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

[email-masked] wrote:

> a) 822 (as amended by RFC 2047 section 5 and as further amended by RFC 
> 2231) had it,
>    though spread out over three documents

Correction:
2047 and 2231 are also amended by the errata page, so that's a total of 
4 documents.

Date: Sun, 04 Jan 2004 00:34:52 -0500
From: [email-masked]
Reply-To: ietf-822 <[email-masked]>
To: Pete Resnick <[email-masked]>
Cc: ietf-822 <[email-masked]>
Subject: Re: Revisiting RFC 2822 grammar
Sender: [email-masked]

Pete Resnick wrote:

> Hi Bruce,
>
> As I said on the ietf-822 list, I was getting started on updates to 
> RFC 2822 to move it along to draft and was looking at:
>
> <http://users.erols.com/blilly/mparse/rfc2822grammar_simplified.txt>
>
> I notice that the ABNF in there has a few things that are non-2822 
> such as encoded words. Do you have a copy of the ABNF which is purely 
> the 2822 replacement that we can post to the ietf-2822 list?
>
> pr

I have an older version which does not have the encoded-word grammar 
(full text below).

The rationale for adding encoded-word grammar is:
a) 822 (as amended by RFC 2047 section 5 and as further amended by RFC 
2231) had it,
    though spread out over three documents
b) it is necessary for MIME-conforming implementations
c) the encoded-word rules are rather complex -- I believe that the 
grammar in the current
    document (URI above) covers everything except the rule prohibiting 
encoded-words in
    Received fields.  In particular, the rules regarding adjacent linear 
whitespace are quite
    complex.

 From an implementor's perspective, I'd like to see all of the relevant 
base grammar (i.e.
base field and supporting grammar) in a single document; indeed, one of 
the benefits of
2822 is that it consolidated most of the "... amends RFC 822" piecemeal 
details into a
single document (obviously, the 2047/2231 amendments somehow didn't make 
it into
2822).  I don't believe there is any harm in including the encoded-word 
grammar as
encoded-words appear in the higher-level constructs as alternatives to 
ccontent, word,
and utext.

Following is the text of the modified grammar w/o encoded-word grammar, 
interspersed
with some notes:

rfc2822grammar_simplified.txt version 0.13 2001/08/08 16:02:35  
excerpted from RFC 2822 and modified by Bruce Lilly

NO-WS-CTL       =       %d1-8 /         ; US-ASCII control characters
                        %d11 /          ;  that do not include the
                        %d12 /          ;  carriage return, line feed,
                        %d14-31 /       ;  and white space characters
                        %d127

text            =       %d1-9 /         ; Characters excluding CR and LF
                        %d11 / %d12 / %d14-127 / obs-text

specials        =       "(" / ")" /     ; Special characters used in
                        "<" / ">" /     ;  other parts of the syntax
                        "[" / "]" / ":" / ";" / "@" / "\" / "," / "." / 
DQUOTE

quoted-pair     =       ("\" text)
[N.B. had redundant obs-qp alternative]

FWS             =       ([*WSP CRLF] 1*WSP) /   ; Folding white space
                        obs-FWS

ctext           =       NO-WS-CTL /     ; Non white space controls
                        %d33-39 /       ; The rest of the US-ASCII
                        %d42-91 /       ;  characters not including "(",
                        %d93-126        ;  ")", or "\"
[N.B. RFC 822 ASCII NUL not permitted, even with obs- rules]

ccontent        =       ctext / quoted-pair / comment

comment         =       "(" *([FWS] ccontent) [FWS] ")"

CFWS            =       *([FWS] comment) (([FWS] comment) / FWS)

atext           =       ALPHA / DIGIT / ; Any character except controls,
                        "!" / "#" /     ;  SP, and specials.
                        "$" / "%" /     ;  Used for atoms
                        "&" / "'" / "*" / "+" / "-" / "/" / "=" / "?" / 
"^" / "_" / "`" / "{" / "|" / "}" / "~"

atom            =       1*atext [CFWS]

dot-atom        =       dot-atom-text [CFWS]

dot-atom-text   =       1*atext *("." 1*atext)

qtext           =       NO-WS-CTL /     ; Non white space controls
                        %d33 /          ; The rest of the US-ASCII
                        %d35-91 /       ;  characters not including "\"
                        %d93-126        ;  or the quote character
[N.B. RFC 822 ASCII NUL not permitted, even with obs- rules]

qcontent        =       qtext / quoted-pair

quoted-string   =       DQUOTE [FWS] *(qcontent [FWS]) DQUOTE [CFWS]

word            =       atom / quoted-string

phrase          =       1*word / obs-phrase

utext           =       NO-WS-CTL /     ; Non white space controls
                        %d33-126 /      ; The rest of US-ASCII
                        obs-utext

unstructured    =       *(utext [FWS])

date-time       =       ([ day-name "," [FWS]] date FWS time [CFWS]) / 
obs-date-time

day-name        =       "Mon" / "Tue" / "Wed" / "Thu" / "Fri" / "Sat" / 
"Sun"

date            =       day FWS month-name FWS year

year            =       4*DIGIT

month-name      =       "Jan" / "Feb" / "Mar" / "Apr" / "May" / "Jun" / 
"Jul" / "Aug" / "Sep" / "Oct" / "Nov" / "Dec"

day             =       1*2DIGIT

time            =       time-of-day FWS zone

time-of-day     =       hour ":" minute [ ":" second ]

hour            =       2DIGIT

minute          =       2DIGIT

second          =       2DIGIT

zone            =       ( "+" / "-" ) 4DIGIT
[N.B. no CFWS between +- and 4DIGIT]

address         =       mailbox / group

mailbox         =       name-addr / addr-spec

name-addr       =       [display-name] angle-addr

angle-addr      =       ("<" [CFWS] addr-spec ">" [CFWS]) / obs-angle-addr

group           =       display-name ":" [CFWS] [mailbox-list] ";" [CFWS]

display-name    =       phrase

mailbox-list    =       (mailbox *("," [CFWS] mailbox)) / obs-mbox-list

address-list    =       (address *("," [CFWS] address)) / obs-addr-list

addr-spec       =       local-part "@" [CFWS] domain

local-part      =       dot-atom / quoted-string / obs-local-part

domain          =       dot-atom / domain-literal / obs-domain

domain-literal  =       "[" [FWS] *(dcontent [FWS]) "]" [CFWS]

dcontent        =       dtext / quoted-pair

dtext           =       NO-WS-CTL /     ; Non white space controls
                        %d33-90 /       ; The rest of the US-ASCII
                        %d94-126        ;  characters not including "[",
                                        ;  "]", or "\"
[N.B. RFC 822 ASCII NUL not permitted, even with obs- rules]

message         =       (fields / obs-fields) [CRLF body]

body            =       *(*998text CRLF) *998text

fields          =       *(trace *(resent-date / resent-from / 
resent-sender / resent-to / resent-cc / resent-bcc / resent-msg-id))
                        *(orig-date / from / sender / reply-to / to / cc 
/ bcc / message-id / in-reply-to / references / subject / comments / 
keywords / optional-field)

orig-date       =       "Date:" [FWS] date-time CRLF

from            =       "From:" [CFWS] mailbox-list CRLF

sender          =       "Sender:" [CFWS] mailbox CRLF

reply-to        =       "Reply-To:" [CFWS] address-list CRLF

to              =       "To:" [CFWS] address-list CRLF

cc              =       "Cc:" [CFWS] address-list CRLF

bcc             =       "Bcc:" [CFWS] [address-list] CRLF

message-id      =       "Message-ID:" [CFWS] msg-id CRLF

in-reply-to     =       "In-Reply-To:" [CFWS] 1*msg-id CRLF

references      =       "References:" [CFWS] 1*msg-id CRLF

msg-id          =       ( "<" id-left "@" id-right ">" [CFWS]) / obs-msg-id

id-left         =       dot-atom-text / no-fold-quote

id-right        =       dot-atom-text / no-fold-literal

no-fold-quote   =       DQUOTE *(qtext / quoted-pair) DQUOTE

no-fold-literal =       "[" *(dtext / quoted-pair) "]"

subject         =       "Subject:" [FWS] [("cmsg" / "Re: ") [FWS]] 
unstructured CRLF
[ RFC 1036 sect. 2.2.6 "cmsg" Subject hack, sect. 2.1.4 "Re: " ]

comments        =       "Comments:" [FWS] unstructured CRLF

keywords        =       "Keywords:" [CFWS] phrase *("," [CFWS] phrase) CRLF

resent-date     =       "Resent-Date:" [FWS] date-time CRLF

resent-from     =       "Resent-From:" [CFWS] mailbox-list CRLF

resent-sender   =       "Resent-Sender:" [CFWS] mailbox CRLF

resent-to       =       "Resent-To:" [CFWS] address-list CRLF

resent-cc       =       "Resent-Cc:" [CFWS] address-list CRLF

resent-bcc      =       "Resent-Bcc:" [CFWS] [address-list] CRLF

resent-msg-id   =       "Resent-Message-ID:" [CFWS] msg-id CRLF

trace           =       [return] 1*received

return          =       "Return-Path:" [CFWS] path CRLF

path            =       ("<" [CFWS] [addr-spec] ">" [CFWS]) / obs-path

received        =       "Received:" [CFWS] name-val-list ";" [FWS] 
date-time CRLF

name-val-list   =       [*(name-val-pair CFWS) name-val-pair]
[N.B. 2822 specification does not provide for mandatory CFWS at end of 
list (as opposed to RFC 821 (required <SP>) and 2821)
    [name-val-pair CFWS *(name-val-pair CFWS)]
]

name-val-pair   =       item-name CFWS item-value

item-name       =       ALPHA *(["-"] (ALPHA / DIGIT))

item-value      =       1*angle-addr / addr-spec / atom / domain / msg-id

optional-field  =       field-name ":" [FWS] unstructured CRLF

field-name      =       1*ftext

ftext           =       %d33-57 /               ; Any character except
                        %d59-126                ;  controls, SP, and
                                                ;  ":".

obs-qp          =       "\" (%d0-127)
[N.B. unnecessary]

obs-text        =       %d0-127
[N.B. original 2822 specification was as obs-utext in this file, which 
permitted multiple characters]

obs-char        =       %d0-9 / %d11 /          ; %d0-127 except CR and
                        %d12 / %d14-127         ;  LF

obs-utext       =       *LF *CR *(obs-char *LF *CR)
[N.B. was obs-text]

obs-phrase      =       word *(word / ("." [CFWS]))

obs-phrase-list =       phrase / (1*([phrase] "," [CFWS]) [phrase])

obs-FWS         =       1*WSP *(CRLF 1*WSP)

obs-date-time   =       [ day-name [CFWS] "," [CFWS]] obs-date [CFWS] 
FWS [CFWS] obs-time [CFWS]
[N.B. obs- rule does not provide for adjacent date and time permitted by 
RFC 822]

obs-date        =       day CFWS month-name CFWS obs-year
[N.B. obs- rule does not permit (e.g.) 1Jan2001 which was permissible 
under RFC 822]

obs-year        =       2*DIGIT

obs-time        =       obs-time-of-day CFWS (zone / obs-zone)
[N.B. obs- rule does not permit adjacent time and zone, which was 
permissible under RFC 822]

obs-time-of-day =       hour [CFWS] ":" [CFWS] minute [CFWS] ":"  
[[CFWS] second]

obs-zone        =       "UT" / "GMT" /          ; Universal Time
                                                ; North American UT
                                                ; offsets
                        "EST" / "EDT" /         ; Eastern:  - 5/ - 4
                        "CST" / "CDT" /         ; Central:  - 6/ - 5
                        "MST" / "MDT" /         ; Mountain: - 7/ - 6
                        "PST" / "PDT" /         ; Pacific:  - 8/ - 7
                        %d65-73 /               ; Military zones - "A"
                        %d75-90 /               ; through "I" and "K"
                        %d97-105 /              ; through "Z", both
                        %d107-122               ; upper and lower case

obs-angle-addr  =       "<" [CFWS] [obs-route] addr-spec ">" [CFWS]

obs-route       =       obs-domain-list ":" [CFWS]

obs-domain-list =       "@" [CFWS] domain *(1*("," [CFWS]) "@" [CFWS] 
domain)

obs-local-part  =       word *("." [CFWS] word)

obs-domain      =       atom *("." [CFWS] atom)

obs-mbox-list   =       1*([mailbox] "," [CFWS]) [mailbox]

obs-addr-list   =       1*([address] "," [CFWS]) [address]

obs-fields      =       *(obs-return / obs-received / obs-orig-date / 
obs-from / obs-sender / obs-reply-to / obs-to / obs-cc / obs-bcc / 
obs-message-id / obs-in-reply-to / obs-references / obs-subject / 
obs-comments / obs-keywords / obs-resent-date / obs-resent-from / 
obs-resent-send / obs-resent-rply / obs-resent-to / obs-resent-cc / 
obs-resent-bcc / obs-resent-mid / obs-optional)

obs-orig-date   =       "Date" *WSP ":" [CFWS] date-time CRLF

obs-from        =       "From" *WSP ":" [CFWS] mailbox-list CRLF

obs-sender      =       "Sender" *WSP ":" [CFWS] mailbox CRLF

obs-reply-to    =       "Reply-To" *WSP ":" [CFWS] address-list CRLF

obs-to          =       "To" *WSP ":" [CFWS] address-list CRLF

obs-cc          =       "Cc" *WSP ":" [CFWS] address-list CRLF

obs-bcc         =       "Bcc" *WSP ":" [CFWS] [address-list] CRLF

obs-message-id  =       "Message-ID" *WSP ":" [CFWS] msg-id CRLF

obs-in-reply-to =       "In-Reply-To" *WSP ":" [CFWS] *(phrase / msg-id) 
CRLF

obs-references  =       "References" *WSP ":" [CFWS] *(phrase / msg-id) CRLF

obs-msg-id      =       "<" [CFWS] addr-spec ">" [CFWS]

obs-subject     =       "Subject" *WSP ":" [FWS] [("cmsg" / "Re:") 
[FWS]] unstructured CRLF
[ RFC 1036 sect. 2.2.6 "cmsg" hack, 2.1.4 "Re:" (w/ or w/o space) ]

obs-comments    =       "Comments" *WSP ":" [FWS] unstructured CRLF

obs-keywords    =       "Keywords" *WSP ":" [CFWS] obs-phrase-list CRLF

obs-resent-from =       "Resent-From" *WSP ":" [CFWS] mailbox-list CRLF

obs-resent-send =       "Resent-Sender" *WSP ":" [CFWS] mailbox CRLF

obs-resent-date =       "Resent-Date" *WSP ":" [CFWS] date-time CRLF

obs-resent-to   =       "Resent-To" *WSP ":" [CFWS] address-list CRLF

obs-resent-cc   =       "Resent-Cc" *WSP ":" [CFWS] address-list CRLF

obs-resent-bcc  =       "Resent-Bcc" *WSP ":" [CFWS] [address-list] CRLF

obs-resent-mid  =       "Resent-Message-ID" *WSP ":" [CFWS] msg-id CRLF

obs-resent-rply =       "Resent-Reply-To" *WSP ":" [CFWS] address-list CRLF

obs-return      =       "Return-Path" *WSP ":" [CFWS] path CRLF

obs-received    =       "Received" *WSP ":" [CFWS] name-val-list [ ";" 
[CFWS] obs-date-time ] CRLF
[N.B. RFC 822 required date-time stamp]
[N.B. reference online version of 2822 specification does not permit WSP 
before colon if date-time stamp is used; RFC 822 permitted (nay, required)
            "Received" *WSP ":" [CFWS] name-val-list ";" [CFWS] 
obs-date-time CRLF
]

obs-path        =       obs-angle-addr

obs-optional    =       field-name *WSP ":" [FWS] unstructured CRLF

--------------------------------------------------------------------------------
Notes not part of modified grammar:

For LR(1) parser compatibility, lexical tokens are grouped such that 
trailing
WS, FWS, or CFWS is associated with its preceding lexical token.  Therefore,
no lexical token handled by the higher-level parser grammar rules has any
ambiguity associated with optional WS, FWS, or CFWS. So, where this revised
grammar has:

    obs-mbox-list   =       1*([mailbox] "," [CFWS]) [mailbox]

that is handled by the implementation as:

    obs-mbox-list   =       1*([mailbox] ("," [CFWS])) [mailbox]

Additional rules such as:

    start           =       (":" [FWS]) / obs-start
    obs-start       =       *WSP ":" [FWS]
    cstart          =       (":" [CFWS]) / obs-cstart
    obs-cstart      =       *WSP ":" [CFWS]
    dstart          =       start / obs-cstart

can be used to reduce the number of rules, e.g.:

    orig-date       =    "Date" dstart date-time CRLF
(eliminating obs-orig-date (also applies to resent-date))
    subject         =       "Subject" start ["cmsg" [FWS]] unstructured CRLF
(eliminating obs-subject (start also applies to comments and 
optional-field))
    from            =       "From" cstart mailbox-list CRLF
(eliminating obs-from (cstart applies to remaining header fields))
etc., allowing all of the obs- header fields to be eliminated, and 
obs-fields to
be simplified.

And adding:

    resent          =       "Resent-"

allows:

    resent-from     =       resent from

etc., allowing the resent- fields to be simplified and ensuring that the
definitions remain in sync between base and resent- versions.

Date: Sat, 03 Jan 2004 18:33:23 -0800 (PST)
From: [email-masked]
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
To: [email-masked]
Sender: [email-masked]

> On 1/3/04 at 11:00 AM -0500, Al Costanzo wrote:

> >>I don't think anyone would have a problem with a small note in the
> >>"Changes to previous standards" section which says, "Some people
> >>think X-* fields are bad, some think they are good, they were
> >>removed from this document." For a fuller discussion, I think Keith
> >>is right that we might consider a separate informational document.
> >
> >If they are not part of this document it cannot replace/obsolete 822.

> I'm sorry, but I think this is rubbish. X-* fields are perfectly
> legal in 2822 (they fall under "optional-field"); they simply are not
> given the special treatment of 822, that they won't ever be official
> published extensions to the standard. The only difference then
> between 822 and 2822 in this respect is that 822 gives publication
> guidelines for extensions where 2822 does not. This has absolutely
> *no* effect on implementations of the protocol. I defy anyone to
> explain to me why this should have any effect on 2822bis obsoleting
> 822.

Pete, I am in complete agreement with your analysis here. Additionally, I will
point out that even had 2822 done something else with X- fields, such as
deprecating their use, even that would be no impediment to 2822 specification
eventually obsoleting 822. There is certainly no requirement that new
specifications retain all the features of old ones they obsolete. Indeed, when
advancing on the standards track, the one change we do allow along the way is
for features to be dropped.

> Now, let me use this opportunity to explain *why* I don't think X-*
> fields should be in the standard in the first place:

> The idea behind X-* fields (what 822 called user-defined fields) was
> so that folks could use new field names without worrying that they
> would interfere with the standard fields. That way, you didn't have
> to go through the trouble of publishing an RFC just to use a field
> that was only going to be internal to your particular system. The
> downside was that someone else could come along and potentially use
> the same field name in a completely incompatible way, but that was
> the risk that you took.

> But since then, two things have made it clear that such user-defined
> fields are unnecessary and problematic:

> 1. Unnecessary: We now have IANA registration services. We could very
> simply create an IANA registry of e-mail field names.

This is already underway and the document specifying it -
draft-klyne-msghdr-registry-06.txt - has been discussed on this list. I note in
passing that it doesn't distinguish X- fields as being in any way special
either.

An interesting question is whether or not 2822bis should reference this
registry, assuming the specification for it is approved in the near future
(which I hope it will be).

> Then, if you
> wanted to create a new field, you just fill out the registration form
> and you're not only guaranteed that you won't interfere with
> published standard fields, you're also guaranteed that nobody else
> would use the field you chose in an incompatible way. And you
> wouldn't have to have a name that started with "X-".

> 2. Problematic: Some folks thought it was nice to have "X-*" fields
> for completely private use. But history teaches us that inevitably
> "X-*" fields *will* leak out onto the rest of the Internet. As
> several people in this thread have pointed out, we now have some
> "X-*" fields that are in widespread use. Sometimes, people use them
> very interoperably and they have served a fabulous purpose.
> Sometimes, they don't get used interoperably. It would be great if we
> could publish some standards to make sure that they do get used
> interoperably. But guess what: We can't! It is forbidden by RFC 822
> to publish any standard for a field name that beings with "X-". No
> matter how widespread the use, no matter how useful the field is, it
> can not be standardized *by definition*. That decreases
> interoperability on the Internet.

I agree with this as well.

> Now, personally, I would love the document to say, "There is no good
> reason to use fields that begin with 'X-' as defined by RFC 822; any
> name that does not interfere with a currently registered or published
> field name is fine, and using an 'X-' field is discouraged because
> they can't ever be standardized." But there was widespread
> non-consensus in the DRUMS Working Group when this was discussed.

Yep.

> 2822 does say "Extension header fields no longer specifically called
> out." In 2822bis, I think it would be fine to clarify that and say
> "User-defined header fields (those starting with 'X-') no longer
> specifically discussed", and even have an informational reference to

I'd say "defined or discussed" rather than simply "discussed", but
other than that I think this is fine.

> a document (or 2) on why it was pulled. But I see absolutely no
> reason that this should stop 2822bis from moving to full Standard and
> obsoleting 822.

Nor do I.

				Ned

From: Arnt Gulbrandsen <[email-masked]>
To: Pete Resnick <[email-masked]>
Subject: Re: X-* header fields (Was: Getting 2822 to Draft)
Cc: Al Costanzo <[email-masked]>, [email-masked]
Date: Sat, 3 Jan 2004 20:02:18 +0100
Sender: [email-masked]

Pete Resnick writes:
> 2822 does say "Extension header fields no longer specifically called 
> out." In 2822bis, I think it would be fine to clarify that and say 
> "User-defined header fields (those starting with 'X-') no longer 
> specifically discussed", and even have an informational reference to 
> a document (or 2) on why it was pulled.

I like the suggested change. Please make sure that 2822bis includes the 
string 'X-', so that anyone searching for X- finds the right sentence.

--Arnt

Date: Sat, 3 Jan 2004 11:40:30 -0600
To: "Al Costanzo" <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: X-* header fields (Was: Getting 2822 to Draft)
Cc: <[email-masked]>
Sender: [email-masked]

On 1/3/04 at 11:00 AM -0500, Al Costanzo wrote:

>>I don't think anyone would have a problem with a small note in the 
>>"Changes to previous standards" section which says, "Some people 
>>think X-* fields are bad, some think they are good, they were 
>>removed from this document." For a fuller discussion, I think Keith 
>>is right that we might consider a separate informational document.
>
>If they are not part of this document it cannot replace/obsolete 822.

I'm sorry, but I think this is rubbish. X-* fields are perfectly 
legal in 2822 (they fall under "optional-field"); they simply are not 
given the special treatment of 822, that they won't ever be official 
published extensions to the standard. The only difference then 
between 822 and 2822 in this respect is that 822 gives publication 
guidelines for extensions where 2822 does not. This has absolutely 
*no* effect on implementations of the protocol. I defy anyone to 
explain to me why this should have any effect on 2822bis obsoleting 
822.

Now, let me use this opportunity to explain *why* I don't think X-* 
fields should be in the standard in the first place:

The idea behind X-* fields (what 822 called user-defined fields) was 
so that folks could use new field names without worrying that they 
would interfere with the standard fields. That way, you didn't have 
to go through the trouble of publishing an RFC just to use a field 
that was only going to be internal to your particular system. The 
downside was that someone else could come along and potentially use 
the same field name in a completely incompatible way, but that was 
the risk that you took.

But since then, two things have made it clear that such user-defined 
fields are unnecessary and problematic:

1. Unnecessary: We now have IANA registration services. We could very 
simply create an IANA registry of e-mail field names. Then, if you 
wanted to create a new field, you just fill out the registration form 
and you're not only guaranteed that you won't interfere with 
published standard fields, you're also guaranteed that nobody else 
would use the field you chose in an incompatible way. And you 
wouldn't have to have a name that started with "X-".

2. Problematic: Some folks thought it was nice to have "X-*" fields 
for completely private use. But history teaches us that inevitably 
"X-*" fields *will* leak out onto the rest of the Internet. As 
several people in this thread have pointed out, we now have some 
"X-*" fields that are in widespread use. Sometimes, people use them 
very interoperably and they have served a fabulous purpose. 
Sometimes, they don't get used interoperably. It would be great if we 
could publish some standards to make sure that they do get used 
interoperably. But guess what: We can't! It is forbidden by RFC 822 
to publish any standard for a field name that beings with "X-". No 
matter how widespread the use, no matter how useful the field is, it 
can not be standardized *by definition*. That decreases 
interoperability on the Internet.

Now, personally, I would love the document to say, "There is no good 
reason to use fields that begin with 'X-' as defined by RFC 822; any 
name that does not interfere with a currently registered or published 
field name is fine, and using an 'X-' field is discouraged because 
they can't ever be standardized." But there was widespread 
non-consensus in the DRUMS Working Group when this was discussed. 
2822 does say "Extension header fields no longer specifically called 
out." In 2822bis, I think it would be fine to clarify that and say 
"User-defined header fields (those starting with 'X-') no longer 
specifically discussed", and even have an informational reference to 
a document (or 2) on why it was pulled. But I see absolutely no 
reason that this should stop 2822bis from moving to full Standard and 
obsoleting 822.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

From: Dave Crocker <[email-masked]>
To: <[email-masked]>
Date: Sat, 3 Jan 2004 09:05:40 -0800
Subject: Re: Re: Getting 2822 to Draft
Sender: [email-masked]

Folks

>  2822 allows X-* fields and always has. Any future document will 
>  allow them. What 2822 doesn't do is give special meaning to them 
>  other than as optional-fields. -- Pete Resnick 
>  <http://www.qualcomm.com/~presnick/>
>  QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-
>  1102

1. If X- fields are the only technical aspect of 2822 that folks have 
any comments about, then the doc is clearly is really excellent shape.

2. In spite of originally thinking that X- fields were a really 
excellent idea, I am now a vigorous opponent to the use of X- fields. 
However I think that any significant effort to focus on their use is a 
waste of time, at best, and more likely will just confuse things.  The 
labels exist.  They are part of the usage model.  The damage they cause 
might be real but it would be difficult to say that it has really caused 
major damage to the Internet.  Adding a discussion paragraph to 2822 is 
as much as might be useful.

d/
--
Dave Crocker <dcrocker-at-brandenburg-dot-com>
Brandenburg InternetWorking <http://brandenburg.com>

Date: Sat, 3 Jan 2004 10:35:11 -0600
To: Kurt Keller <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Getting 2822 to Draft
Cc: [email-masked]
Sender: [email-masked]

On 1/3/04 at 1:24 AM +0100, Kurt Keller wrote:

>Replacing 822 and in the course silently dropping X-* would leave a lot
>of software and systems stand outside without the protective umbrella
>they had up to now; they survive but if it rains, they get wet.

2822 allows X-* fields and always has. Any future document will allow 
them. What 2822 doesn't do is give special meaning to them other than 
as optional-fields.
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Sat, 3 Jan 2004 11:20:10 -0500
To: Kurt Keller <[email-masked]>
Sender: [email-masked]

> Replacing 822 and in the course silently dropping X-* would leave a lot
> of software and systems stand outside without the protective umbrella
> they had up to now; they survive but if it rains, they get wet.

X- headers aren't going to get dropped.

> RFC-1918 is a godsend for IP addressing.

I'm going to assume this is a troll, and ignore it.

Keith

From: "Al Costanzo" <[email-masked]>
To: "Pete Resnick" <[email-masked]>, <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Sat, 3 Jan 2004 11:00:13 -0500
Sender: [email-masked]

If they are not part of this document it cannot replace/obsolete 822.

Al Costanzo
----- Original Message ----- 
From: "Pete Resnick" <[email-masked]>
To: "Keith Moore" <[email-masked]>
Cc: "Simon Josefsson" <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Friday, January 02, 2004 3:37 PM
Subject: Re: Getting 2822 to Draft

>
> On 1/2/04 at 11:14 AM -0500, Keith Moore wrote:
>
> >>>On 1/2/04 at 1:34 PM +0100, Simon Josefsson wrote:
> >>>
> >>>>Would 2822bis obsolete 822 fully?
> >
> >2822 bis will not obsolete 822 in any event - 822 is still a full
> >Standard, while 2822 bis would only be Draft Standard.  As a
> >practical matter, nothing can really obsolete 822 as long as people
> >still need to read old mail messages.
>
> Uh, no. If we can get 2822bis to full Standard (which is the desired
> intent), it should obsolete 822 in that it should contain all of the
> grammar to read old messages. That was the intent of section 4 of
> 2822.
>
> >>>>If so, I believe it should mention what happened to the X- header
> >>>>prefix that was part of 822.
>
> I don't think anyone would have a problem with a small note in the
> "Changes to previous standards" section which says, "Some people
> think X-* fields are bad, some think they are good, they were removed
> from this document." For a fuller discussion, I think Keith is right
> that we might consider a separate informational document.
>
> pr
> -- 
> Pete Resnick <http://www.qualcomm.com/~presnick/>
> QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102
>

Date: Sat, 3 Jan 2004 01:24:01 +0100
From: Kurt Keller <[email-masked]>
To: [email-masked]
Subject: Re: Getting 2822 to Draft
Mail-Followup-To: Kurt Keller <[email-masked]>, [email-masked]
Sender: [email-masked]

Replacing 822 and in the course silently dropping X-* would leave a lot
of software and systems stand outside without the protective umbrella
they had up to now; they survive but if it rains, they get wet.

Some grepping over a couple of inboxes showed that in 650 messages there
were 1232 X-* headers. Many anti-spam and anti-virus systems use such
headers, as does some mailing list software etc, even the messages from
this ietf-822 list do have a X-Authentication-Warning: header.

RFC-1918 is a godsend for IP addressing. X-* headers could be handled in
just a paragraph or two and can be a godsend in many situations as well.
Users and systems which need to use some proprietary header and can do
with an X-* won't need to bloat the standards with their specialized
extensions.

I think there definitely should be either a note in 3822, and a separate
specification, or (preferrably and simpler) a short definition directly
in  3822.

On Fri, Jan 02, 2004 at 11:55:43AM -0500, Keith Moore wrote:
> 
> > We went through this before. At the end, noone raised objections against 
> > the insertion of a sentence such as "X-* header fields are not covered 
> > in this document". Please, can we just insert that in the list of 
> > changes and be done?
> > 
> > Quarrelling about whether X-* is useful, a mistake, neither or both is 
> > not going to help 3822, so let's not do it.
> 
> well, I do sympathize with the idea that being silent on the subject is
> confusing.  but I don't want that discussion to be critical path for 3822.

Cc: Keith Moore <[email-masked]>, Simon Josefsson <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Fri, 2 Jan 2004 16:03:13 -0500
To: Pete Resnick <[email-masked]>
Sender: [email-masked]

> Start saying things like that and you'll only encourage me to write 
> the "X- considered harmful" document. ;-)

and if you do, I'll write the "X- considered wonderful" document :)

actually I think we could probably find a compromise position that we 
could agree on, as long as we don't make it part of the critical path 
for more important work.

Date: Fri, 2 Jan 2004 14:57:06 -0600
To: Keith Moore <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Getting 2822 to Draft
Cc: Keith Moore <[email-masked]>, Simon Josefsson <[email-masked]>, [email-masked]
Sender: [email-masked]

On 1/2/04 at 3:54 PM -0500, Keith Moore wrote:

>>If we can get 2822bis to full Standard (which is the desired 
>>intent), it should obsolete 822 in that it should contain all of 
>>the grammar to read old messages. That was the intent of section 4 
>>of 2822.
>
>yeah, I know.  I'm sort of a purist though - I think if you really 
>want to read 822 messages you should consult 822 as the definitive 
>spec.

It will always have the informational references to 822.

>>I don't think anyone would have a problem with a small note in the 
>>"Changes to previous standards" section which says, "Some people 
>>think X-* fields are bad, some think they are good, they were 
>>removed from this document."
>
>and for the time being 822 is still the definitive spec on X- fields.

Start saying things like that and you'll only encourage me to write 
the "X- considered harmful" document. ;-)

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Cc: Keith Moore <[email-masked]>, Simon Josefsson <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Fri, 2 Jan 2004 15:54:31 -0500
To: Pete Resnick <[email-masked]>
Sender: [email-masked]

>>>>> Would 2822bis obsolete 822 fully?
>>
>> 2822 bis will not obsolete 822 in any event - 822 is still a full 
>> Standard, while 2822 bis would only be Draft Standard.  As a 
>> practical matter, nothing can really obsolete 822 as long as people 
>> still need to read old mail messages.
>
> Uh, no. If we can get 2822bis to full Standard (which is the desired 
> intent), it should obsolete 822 in that it should contain all of the 
> grammar to read old messages. That was the intent of section 4 of 
> 2822.

yeah, I know.  I'm sort of a purist though - I think if you really want 
to read 822 messages you should consult 822 as the definitive spec.

>>>>> If so, I believe it should mention what happened to the X- header 
>>>>> prefix that was part of 822.
>
> I don't think anyone would have a problem with a small note in the 
> "Changes to previous standards" section which says, "Some people think 
> X-* fields are bad, some think they are good, they were removed from 
> this document."

and for the time being 822 is still the definitive spec on X- fields.

Date: Fri, 2 Jan 2004 14:37:40 -0600
To: Keith Moore <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Getting 2822 to Draft
Cc: Simon Josefsson <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

On 1/2/04 at 11:14 AM -0500, Keith Moore wrote:

>>>On 1/2/04 at 1:34 PM +0100, Simon Josefsson wrote:
>>>
>>>>Would 2822bis obsolete 822 fully?
>
>2822 bis will not obsolete 822 in any event - 822 is still a full 
>Standard, while 2822 bis would only be Draft Standard.  As a 
>practical matter, nothing can really obsolete 822 as long as people 
>still need to read old mail messages.

Uh, no. If we can get 2822bis to full Standard (which is the desired 
intent), it should obsolete 822 in that it should contain all of the 
grammar to read old messages. That was the intent of section 4 of 
2822.

>>>>If so, I believe it should mention what happened to the X- header 
>>>>prefix that was part of 822.

I don't think anyone would have a problem with a small note in the 
"Changes to previous standards" section which says, "Some people 
think X-* fields are bad, some think they are good, they were removed 
from this document." For a fuller discussion, I think Keith is right 
that we might consider a separate informational document.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Fri, 2 Jan 2004 14:29:09 -0600
To: Keith Moore <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Re: Getting 2822 to Draft
Cc: Keith Moore <[email-masked]>, [email-masked]
Sender: [email-masked]

On 1/2/04 at 10:02 AM -0500, Keith Moore wrote:

>why not post the revised ABNF as an I-D so that others can go over 
>it with a fine tooth comb?

I sent mail to Bruce. The only copy of the grammar he wrote that I've 
got has some MIME grammar mixed into it in a couple of places. I've 
asked him to post one that is pure 2822 grammar.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Fri, 2 Jan 2004 11:55:43 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Getting 2822 to Draft
Sender: [email-masked]

> 
> We went through this before. At the end, noone raised objections against 
> the insertion of a sentence such as "X-* header fields are not covered 
> in this document". Please, can we just insert that in the list of 
> changes and be done?
> 
> Quarrelling about whether X-* is useful, a mistake, neither or both is 
> not going to help 3822, so let's not do it.

well, I do sympathize with the idea that being silent on the subject is
confusing.  but I don't want that discussion to be critical path for 3822.

Date: Fri,  2 Jan 2004 08:52:09 -0800
From: [email-masked] (WJCarpenter)
To: [email-masked]
Subject: Re: Getting 2822 to Draft
Sender: [email-masked]

km> As a practical matter, nothing can really obsolete 822 as long as
km> people still need to read old mail messages.

I don't think this is a useful consideration, and it is perhaps even
harmful to the discussion at hand.  People (hey, I'm one of those
people) need to read old email in a variety of file formats, and that
includes some that are very like RFC-822.  It also includes many
formats which are widely different from RFC-822.  

Who cares?  

Software that used to understand RFC-822 will not suddenly stop
understanding it unless some implementer does something to make it so,
just as implementers already occasionally, silently do something which
isn't RFC-822 compliant.

In any event, people using RFC-822/2822-ish formats are an order of
magnitude less screwed than those whose archived email gets trapped
inside undocumented proprietary formats.  There are a heck of a lot
more of those people these days, though they don't tend to participate
in IETF discussions.
-- 
[email-masked] (WJCarpenter)    PGP 0x91865119
38 95 1B 69 C9 C6 3D 25    73 46 32 04 69 D6 ED F3

From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Getting 2822 to Draft
Date: Fri, 2 Jan 2004 17:22:45 +0100
Sender: [email-masked]

We went through this before. At the end, noone raised objections against 
the insertion of a sentence such as "X-* header fields are not covered 
in this document". Please, can we just insert that in the list of 
changes and be done?

Quarrelling about whether X-* is useful, a mistake, neither or both is 
not going to help 3822, so let's not do it.

--Arnt

Date: Fri, 2 Jan 2004 11:14:48 -0500
From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Getting 2822 to Draft
Sender: [email-masked]

> >> Would 2822bis obsolete 822 fully?  If so, I believe it should
> >mention> what happened to the X- header prefix that was part of 822. 
> >I don't> believe simply removing them without stating why is a good
> >idea,> people can draw many different conclusions from that, which
> >affect> real implementations.
> >
> > some people think X- headers are extremely useful, others think they
> > were a bad mistake that should be eradicated.  maybe we need a
> > separate RFC about X- fields.  I'd hate to see this argument block
> > progression of 2822 bis.
> 
> Ignoring the problem won't make it go away, though.  Consciously
> removing text is driven by some motivation.  

IIRC it was driven by the inability to get agreement on the topic within
the DRUMS WG.

2822 bis will not obsolete 822 in any event - 822 is still a full
Standard, while 2822 bis would only be Draft Standard.  As a practical
matter, nothing can really obsolete 822 as long as people still need to
read old mail messages. 

To: Keith Moore <[email-masked]>
Cc: [email-masked]
Subject: Re: Getting 2822 to Draft
From: Simon Josefsson <[email-masked]>
Date: Fri, 02 Jan 2004 16:56:53 +0100
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> Would 2822bis obsolete 822 fully?  If so, I believe it should mention
>> what happened to the X- header prefix that was part of 822.  I don't
>> believe simply removing them without stating why is a good idea,
>> people can draw many different conclusions from that, which affect
>> real implementations.
>
> some people think X- headers are extremely useful, others think they
> were a bad mistake that should be eradicated.  maybe we need a
> separate RFC about X- fields.  I'd hate to see this argument block
> progression of 2822 bis.

Ignoring the problem won't make it go away, though.  Consciously
removing text is driven by some motivation.  Why not state that
motivation in the document?  Furthermore, if 2822bis fully obsolete
822, without mentioning X- headers, then the X- headers have been
effectively obsoleted.  I don't see how acknowledging that implicit
decision by stating it makes matters worse.

Thanks,
Simon

From: "Al Costanzo" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "Simon Josefsson" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>, <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Fri, 2 Jan 2004 10:10:36 -0500
Sender: [email-masked]

I am one of those people that think that the X-Headers are useful since
software we produce uses them also one of those peopel that think MIME has
gone amuck.

I would oppose:

1. Any RFC that completely obsoletes RFC822
2. Any RFC that remove, obsoletes, etc. the X-Headers fields.

Al
----- Original Message ----- 
From: "Keith Moore" <[email-masked]>
To: "Simon Josefsson" <[email-masked]>
Cc: "Keith Moore" <[email-masked]>; <[email-masked]>
Sent: Friday, January 02, 2004 10:06 AM
Subject: Re: Getting 2822 to Draft

>
> > Would 2822bis obsolete 822 fully?  If so, I believe it should mention
> > what happened to the X- header prefix that was part of 822.  I don't
> > believe simply removing them without stating why is a good idea,
> > people can draw many different conclusions from that, which affect
> > real implementations.
>
> some people think X- headers are extremely useful, others think they
> were a bad mistake that should be eradicated.  maybe we need a separate
> RFC about X- fields.  I'd hate to see this argument block progression
> of 2822 bis.
>
>

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Fri, 2 Jan 2004 10:06:47 -0500
To: Simon Josefsson <[email-masked]>
Sender: [email-masked]

> Would 2822bis obsolete 822 fully?  If so, I believe it should mention
> what happened to the X- header prefix that was part of 822.  I don't
> believe simply removing them without stating why is a good idea,
> people can draw many different conclusions from that, which affect
> real implementations.

some people think X- headers are extremely useful, others think they 
were a bad mistake that should be eradicated.  maybe we need a separate 
RFC about X- fields.  I'd hate to see this argument block progression 
of 2822 bis.

Cc: Keith Moore <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Fri, 2 Jan 2004 10:02:49 -0500
To: Pete Resnick <[email-masked]>
Sender: [email-masked]

> The one possible big thing has to do with the ABNF in 2822. Out of 
> either altruism or insanity, some time ago Bruce Lilly had written up 
> changes to the ABNF in 2822 to do some cool things. On the plus side, 
> it seems to get rid of all of the [C]FWS shift-reduce conflicts, and 
> it is already done. On the minus side, I don't know anyone (myself 
> included) who has gone over it with a fine tooth comb, it is a 
> significant number of ABNF changes, and it therefore might recycle us 
> at Proposed. I am open to suggestions on this.

if the new grammar accepted the same language as the old one, I'd 
consider it nothing more than a clarification of the spec, and a reset 
to Proposed would be unnecessary.

or if the only differences could be seen as minor bug fixes to the old 
grammar, I'd consider it a minor bug fix to the spec.

IMHO simplifying the grammar and getting rid of parsing conflicts would 
both be highly desirable improvements.  if it then becomes possible to 
feed the ABNF to a parser generator and generate a verifier for 2822 
messages, that's a huge win.

why not post the revised ABNF as an I-D so that others can go over it 
with a fine tooth comb?  or has this already been done, and I missed 
it?

To: [email-masked]
From: Simon Josefsson <[email-masked]>
Subject: Re: Getting 2822 to Draft
Date: Fri, 02 Jan 2004 13:34:59 +0100
Lines: 12
Cancel-Lock: sha1:J0RY25sKfUoGnDCVSFY3SUK2BtE=
Sender: [email-masked]

Pete Resnick <[email-masked]> writes:

> Comments?

Would 2822bis obsolete 822 fully?  If so, I believe it should mention
what happened to the X- header prefix that was part of 822.  I don't
believe simply removing them without stating why is a good idea,
people can draw many different conclusions from that, which affect
real implementations.

Thanks,
Simon

Date: Thu, 1 Jan 2004 22:26:35 -0600
To: [email-masked]
From: Pete Resnick <[email-masked]>
Subject: Getting 2822 to Draft
Sender: [email-masked]

So, for the new year I started thinking about getting 2822 to Draft. 
(Also, 3822 is coming up in the RFC numbers :-) ). I've gotten 
started on getting a new draft together. There are a small bunch of 
nits to fix; that I can handle. There's also an implementation report 
to write. On that I'd like to get some help.

The one possible big thing has to do with the ABNF in 2822. Out of 
either altruism or insanity, some time ago Bruce Lilly had written up 
changes to the ABNF in 2822 to do some cool things. On the plus side, 
it seems to get rid of all of the [C]FWS shift-reduce conflicts, and 
it is already done. On the minus side, I don't know anyone (myself 
included) who has gone over it with a fine tooth comb, it is a 
significant number of ABNF changes, and it therefore might recycle us 
at Proposed. I am open to suggestions on this.

Comments?

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102