Subject: Updated Format = Flowed Draft
From: Randall Gellens <[email-masked]>
To: [email-masked]
Date: Fri, 26 Feb 1999 19:27:05 -0800

The updated Format = Flowed draft is on the I-D sites, such as 
<ftp://ftp.ietf.org/internet-drafts/draft-gellens-format-03.txt>

Here are the changes:

	-  The quote indicator is now ">"
	-  Lines now space-stuffed
	-  Flowed lines now end in one or more spaces
	-  Reconciled text, examples, and ABNF for paragraph definition
	-  Clarified Quoted-Printable instructions
	-  Added "Interpreting Format=Flowed" section
	-  Clarified section on Quoting
	-  Added section on space-stuffing
	-  Added examples to section on "Embarrassing Line Wrap"
	-  Modified "Generating Format=Flowed" and ABNF in light of other
	   changes
	-  Added IANA and Internationalization sections
	-  Updated Acknowledgments and Internet Draft boiler plate

To: Jacob Palme <[email-masked]>
Cc: [email-masked]
Subject: Re: IETF Draft: Good Mailing List Behaviour 
From: [email-masked]
Date: Fri, 26 Feb 1999 10:19:53 -0500
Sender: [email-masked]

--==_Exmh_-172333286P
Content-Type: text/plain; charset=us-ascii

On Fri, 26 Feb 1999 15:56:58 +0100, you said:
> At 13.09 +0100 99-02-26, Charles Lindsey wrote:
> > Now suppose there is a topic that should be discussed on both lists (and
> > actually, this is a real situation, because I am about to start such a
> > topic in a few days). One way would be to create a new list (call it
> > usenet-plus-mail) whose only subscribers are ietf-822 and usenet-format.
> > So if you want to post to both lists, you post instead to
> > usenet-plus-mail, but you continue posting to the single lists for topics
> > that only apply to one of them (well, there may be other ways of
> > arranging things - this is just a possible example).
> 
> That is a very uusual special case. In almost all cases,
> people do not want to post to sublists except by mistake,
> when their intention was to reach the whole list.

Jacob:

I think you may have misunderstood what Charles meant.  My reading of it
is (to cite an actual local situation):

We have locally 4 lists engr-fresh, engr-soph, engr-junior, and engr-senior
(one each for the 4 years of the engineering program).  We also have
a 'super-list' engr-all which has all 4 as sublists.  Postings of
interest to *all* engineering students get sent to engr-all.

What Charles was talking about was that (for example) you still need
to be able to post to engr-fresh for those subjects of interest only
to first-year students.

As another example, if there were a list 'usenet-plus-mail', you would
still need the ietf-822 list for discussions of things that don't apply
to usenet, such as the discussion of what headers a mailing list processor
should add (as you point out, a Received: and RFC2369 headers)...

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_-172333286P
Content-Type: application/pgp-signature

--==_Exmh_-172333286P--

Date: Fri, 26 Feb 1999 15:56:58 +0100
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: IETF Draft: Good Mailing List Behaviour
Sender: [email-masked]

At 13.09 +0100 99-02-26, Charles Lindsey wrote:
> Jacob Palme wrote:
>If you have a hierarchical graph without loops, then all
> >new submissions should send to the top node, so lists below
> >the top node should only accept submission whose SMTP
> >sender is the top node!
>
> No, I don't think that is right. Let us take an example. Both you and I
> are on both of the lists ietf-822 and usenet-format, and we post to
> each of those lists independently of the other.
>
> Now suppose there is a topic that should be discussed on both lists (and
> actually, this is a real situation, because I am about to start such a
> topic in a few days). One way would be to create a new list (call it
> usenet-plus-mail) whose only subscribers are ietf-822 and usenet-format.
> So if you want to post to both lists, you post instead to
> usenet-plus-mail, but you continue posting to the single lists for topics
> that only apply to one of them (well, there may be other ways of
> arranging things - this is just a possible example).

That is a very uusual special case. In almost all cases,
people do not want to post to sublists except by mistake,
when their intention was to reach the whole list.
> >> >8.    Loop Control
> >>
> >Is this commonly used? I thought the common idea was that mailing list
> >expanders should not add Resent- headers. Am I wrong?
>
> Hmmm! I have been told by somebody else that this was not so, but it is
> not my reading of DRUMS:
>
> "3.6.6 Resent fields
>
> Resent fields SHOULD be added to any message which is reintroduced by a
> user into the transport system. A separate set of resent fields SHOULD be
> added if this occurs multiple time.... [and much else]"

A mailing list is not a user. Mailing list expansion does
not mean that a message is taken out of the transport
system and reintroduced.
>
>
> Surely, that covers mailing list expansion precisely? The message comes
> through the transport system to the list server. The list server looks at
> it, takes it apart, decides what to do with it, and then puts it back into
> the transport system with a new load of recipients, but in a form which
> still makes it appear to each ultimate recipient that it came from the
> original originator, with the original message-id, and that replies should
> go back to him (modulo any Reply-To address). The fact that the "user" is
> a bot is neither here nor there. We are used to the idea that "users" on
> the internet are often bots.

> Looking at what happens in the real world, I see that this list (expanded
> by Majordomo, I believe) does NOT add Resent fields, but the usenet-format
> list (expanded by Listserv, I believe) DOES. So the real world is clearly
> confused on this issue.

If your opinion is the agreed opinion in DRUMS, I will
write so. But very few existing mailing list expanders do
add resent-headers. Are they all wrong? Is not a standard a
description of how software does work in reality?

> And if the expander is not supposed to add Resent fields, what is it
> supposed to add? It needs something to show it has been involved (and to
> detect those loops at the same time).

It should add a Received header, and it should add the
List-headers specified in RFC 2369.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

Reply-To: "Al Costanzo" <[email-masked]>
From: "Al Costanzo" <[email-masked]>
To: <[email-masked]>, "John M. Wobus" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: IETF Draft: Good Mailing List Behaviour 
Date: Fri, 26 Feb 1999 09:40:27 -0500
Sender: [email-masked]

I would suggest the word appropriate to replace good...

-----Original Message-----
From: John M. Wobus <[email-masked]>
To: [email-masked] <[email-masked]>
Cc: [email-masked] <[email-masked]>
Date: Friday, February 26, 1999 8:40 AM
Subject: Re: IETF Draft: Good Mailing List Behaviour

>I like the document but found the title abiguous: was it going to
>describe the desirable behavior of a list member?  Or was it going to
>describe desirable behavior of list processing software and the mail
>configuration for the list?  Obviously both are worthy topics, and I
>applaud any effort to take on any part of these tasks.
>
>No good title has occurred to me, but it seems to me that the word
>"good" is unnecessary (just an extra word for the reader to scan:  by
>default, RFC's describe things considered good) and what the document
>describes is "practices", so it might be good to get that into the
>title somehow.  Even so, "Mailing List Practices" seems lacking
>somehow.
>
>Charles F. Lindsay responded:
>
>>>11.   Spam Control
>>
>>Yes, but even more importantly you need to add a section on Mailbombing,
>>pointing out the desirability for subscription requests to be returned to
>>the subscriber for confirmation, preferably with a code to be sent back
>>with the confirmation.
>
>Just to throw in more FUD, I've heard of at least one instance of
>someone who was bombed to distraction with subscription
>confirmation-requests.  I'm not sure if that was the actual intent or
>of it was actually a failed attempt at making the victim's life even
>more miserable by subscribing them to unwanted lists.  In any case, the
>arms race marches on.
>
>-John Wobus
>

To: [email-masked]
cc: [email-masked]
Subject: Re: IETF Draft: Good Mailing List Behaviour 
Date: Fri, 26 Feb 1999 08:03:28 -0500
From: "John M. Wobus" <[email-masked]>
Sender: [email-masked]

I like the document but found the title abiguous: was it going to
describe the desirable behavior of a list member?  Or was it going to
describe desirable behavior of list processing software and the mail
configuration for the list?  Obviously both are worthy topics, and I
applaud any effort to take on any part of these tasks.

No good title has occurred to me, but it seems to me that the word
"good" is unnecessary (just an extra word for the reader to scan:  by
default, RFC's describe things considered good) and what the document
describes is "practices", so it might be good to get that into the
title somehow.  Even so, "Mailing List Practices" seems lacking
somehow.

Charles F. Lindsay responded:

>>11.   Spam Control
>
>Yes, but even more importantly you need to add a section on Mailbombing,
>pointing out the desirability for subscription requests to be returned to
>the subscriber for confirmation, preferably with a code to be sent back
>with the confirmation.

Just to throw in more FUD, I've heard of at least one instance of
someone who was bombed to distraction with subscription
confirmation-requests.  I'm not sure if that was the actual intent or
of it was actually a failed attempt at making the victim's life even
more miserable by subscribing them to unwanted lists.  In any case, the
arms race marches on.

-John Wobus

To: [email-masked]
Xref: clerew local.mime:107
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: IETF Draft: Good Mailing List Behaviour
Date: Fri, 26 Feb 1999 12:09:08 GMT
Lines: 99
Sender: [email-masked]

In <v04020a00b2fb4c378c03@[130.237.161.11]> Jacob Palme <[email-masked]> writes:

>At 12.31 +0100 99-02-25, Charles Lindsey wrote:

>> What about messages arriving other than by SMTP (even uucp is still around
>> in some places)? In any case, if lists are nested, the SMTP address that
>> arrives may be from another list, whereas you may actually want to filter
>> according to the actual posters to that other list.

>If you have nested lists, you usually have a top node,
>where all new submissions should arrive. This top node
>cannot let only subscribers post, because the subscribers
>are on lists lower in the hierarchy.

>I had not thought of this problem. A solution would be if
>all the nested lists were members of each other, or if the
>links in the nesting tree were doubledirected. Then anyone
>could submit to any of the lists, and have the message sent
>to all the other lists. Each list could then be set up to
>check that only SMTP senders of its own subscribers
>(including all the other lists) can send. This structure is
>not so common, but would be a solution and should certainly
>be described.

>If you have a hierarchical graph without loops, then all
>new submissions should send to the top node, so lists below
>the top node should only accept submission whose SMTP
>sender is the top node!

No, I don't think that is right. Let us take an example. Both you and I
are on both of the lists ietf-822 and usenet-format, and we post to
each of those lists independently of the other.

Now suppose there is a topic that should be discussed on both lists (and
actually, this is a real situation, because I am about to start such a
topic in a few days). One way would be to create a new list (call it
usenet-plus-mail) whose only subscribers are ietf-822 and usenet-format.
So if you want to post to both lists, you post instead to
usenet-plus-mail, but you continue posting to the single lists for topics
that only apply to one of them (well, there may be other ways of
arranging things - this is just a possible example).

Now the new list receives posts from all sorts of people that it does not
know about. So there is no point filtering on the SMTP address or on the
>From address, or anything else, there. The lower lists only know (by SMTP)
that the posting came from usenet-plus-mail, which they should be happy
with. They may, however, look at the from line and match it against their
blacklists and whitelists (though in this case there is not much point,
because lots of the submissions may be from people who normally live on
the "other" list).

I don't think there really IS a solution to this one. I doubt the one you
suggested above would do any good.

>>
>> >8.    Loop Control
>>
>Is this commonly used? I thought the common idea was that mailing list
>expanders should not add Resent- headers. Am I wrong?

Hmmm! I have been told by somebody else that this was not so, but it is
not my reading of DRUMS:

"3.6.6 Resent fields

Resent fields SHOULD be added to any message which is reintroduced by a
user into the transport system. A separate set of resent fields SHOULD be
added if this occurs multiple time.... [and much else]"

Surely, that covers mailing list expansion precisely? The message comes
through the transport system to the list server. The list server looks at
it, takes it apart, decides what to do with it, and then puts it back into
the transport system with a new load of recipients, but in a form which
still makes it appear to each ultimate recipient that it came from the
original originator, with the original message-id, and that replies should
go back to him (modulo any Reply-To address). The fact that the "user" is
a bot is neither here nor there. We are used to the idea that "users" on
the internet are often bots.

Please have a look at the DRUMS wording, and tell me where I have
misinterpreted it. And if it doesn't mean that, then what ARE Resent
fields meant to be used for? Or is this another bug I have found in DRUMS
:-( .

Looking at what happens in the real world, I see that this list (expanded
by Majordomo, I believe) does NOT add Resent fields, but the usenet-format
list (expanded by Listserv, I believe) DOES. So the real world is clearly
confused on this issue.

And if the expander is not supposed to add Resent fields, what is it
supposed to add? It needs something to show it has been involved (and to
detect those loops at the same time).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Thu, 25 Feb 1999 20:01:21 +0100
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: IETF Draft: Good Mailing List Behaviour
Sender: [email-masked]

Thank you for useful advice. I will rewise the document, this is only
the first edition.

At 12.31 +0100 99-02-25, Charles Lindsey wrote:
> In <v04020a02b2f9946b849d@[130.237.150.138]> Jacob Palme
><[email-masked]> writes:
>
> >4.    Who may Post to a Mailing List
>
> >The checking on who may submit to a list is done on the SMTP sender of
> >the message, not on the names in From: or Sender: fields in the
> >heading.
>
> What about messages arriving other than by SMTP (even uucp is still around
> in some places)? In any case, if lists are nested, the SMTP address that
> arrives may be from another list, whereas you may actually want to filter
> according to the actual posters to that other list.

If you have nested lists, you usually have a top node,
where all new submissions should arrive. This top node
cannot let only subscribers post, because the subscribers
are on lists lower in the hierarchy.

I had not thought of this problem. A solution would be if
all the nested lists were members of each other, or if the
links in the nesting tree were doubledirected. Then anyone
could submit to any of the lists, and have the message sent
to all the other lists. Each list could then be set up to
check that only SMTP senders of its own subscribers
(including all the other lists) can send. This structure is
not so common, but would be a solution and should certainly
be described.

If you have a hierarchical graph without loops, then all
new submissions should send to the top node, so lists below
the top node should only accept submission whose SMTP
sender is the top node!
>
> >8.    Loop Control
>
> I would have thought the simplest means of detecting a loop would be to
> add a Resent-Message-ID header containing a msg-id in some easily
> recognisable form. Or else a recognizable Resent-From or Resent-Sender.

Is this commonly used? I thought the common idea was that mailing list
expanders should not add Resent- headers. Am I wrong?
>
> >10.   Header Munging
>
> >Apart from what is specifed in sections 8. Loop Control and 9. List
> >Headers, a mailing list expander should not in any way modify the
> >heading of a message. In particular, the list should not change the
> >Message-ID, not add "Resent-", "From:", "Sender:", "Auto-Submitted:"
> >or "Reply-To:". The practice to add the e-mail address of the list in
> >a "Reply-To:" header is common, but is not recommended. Instead, use
> >the "List-Post:" command from [RFC2369].
>
> I don't think you can deprecate the addition of Resent-* headers when they
> are explicitly blessed by DRUMS as the proper way to do things.

For mailing lists?
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

Date: Thu, 25 Feb 1999 20:01:35 +0100
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: IETF Draft: Good Mailing List Behaviour
Sender: [email-masked]

At 12.31 +0100 99-02-25, Charles Lindsey wrote:
> >12.   Groupware
>
> Does groupware include gateways from Netnews? I think these are common
> enough that some explicit mention is necessary.

Gatewaying to news is such a large problem, that I would
prefer someone who knows it better than me write a separate
standard or informational RFCs on how it should be done.
But I could of course mention it, and say that it is not
covered.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

To: [email-masked]
Xref: clerew local.mime:104
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: IETF Draft: Good Mailing List Behaviour
Date: Thu, 25 Feb 1999 11:31:44 GMT
Lines: 65
Sender: [email-masked]

In <v04020a02b2f9946b849d@[130.237.150.138]> Jacob Palme <[email-masked]> writes:

>3.1.1 To Subscribe

>Common commands are: sub, subscribe, join. Best is to support all of
>them.

>Sometimes the name of the requestor (not the e-mail address) is
>specified after this command, for example "Subscribe Mary Woodfence".

Surely the common case where "subscribe" is followed by the email address
of the subscriber needs to be mentioned, especially when it turns out to
be different to the From address or the SMTP address. In such cases a
human needs to look at it. Note that there are often good reasons for this
to happen, such as when a subscriber intends to gateway the list into a
local newsgroup for his own convenience.

>4.    Who may Post to a Mailing List

>The checking on who may submit to a list is done on the SMTP sender of
>the message, not on the names in From: or Sender: fields in the
>heading.

What about messages arriving other than by SMTP (even uucp is still around
in some places)? In any case, if lists are nested, the SMTP address that
arrives may be from another list, whereas you may actually want to filter
according to the actual posters to that other list.

>8.    Loop Control

I would have thought the simplest means of detecting a loop would be to
add a Resent-Message-ID header containing a msg-id in some easily
recognisable form. Or else a recognizable Resent-From or Resent-Sender.

>10.   Header Munging

>Apart from what is specifed in sections 8. Loop Control and 9. List
>Headers, a mailing list expander should not in any way modify the
>heading of a message. In particular, the list should not change the
>Message-ID, not add "Resent-", "From:", "Sender:", "Auto-Submitted:"
>or "Reply-To:". The practice to add the e-mail address of the list in
>a "Reply-To:" header is common, but is not recommended. Instead, use
>the "List-Post:" command from [RFC2369].

I don't think you can deprecate the addition of Resent-* headers when they
are explicitly blessed by DRUMS as the proper way to do things.

>11.   Spam Control

Yes, but even more importantly you need to add a section on Mailbombing,
pointing out the desirability for subscription requests to be returned to
the subscriber for confirmation, preferably with a code to be sent back
with the confirmation.

>12.   Groupware

Does groupware include gateways from Netnews? I think these are common
enough that some explicit mention is necessary.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Wed, 24 Feb 1999 12:39:56 +0100
To: IETF mailing list on MIME and e-mail  <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: IETF Draft: Good Mailing List Behaviour
Sender: [email-masked]

I have written an IETF draft on good mailing list
behaviour. It has long been a need to have such a document.
One particular area, where this is needed, is in the design
of groupware forum systems. Such systems often have a mail
gateway, which cause a forum in such a system to appear, as
seen from e-mail, as a mailing list. But the developers of
groupware forum systems often do not have so much
experience in proper mailing list behaviour, which causes
various kinds of problems when their systems become
connected to the Internet mail transport system.

This document is not intended to become a standard, but to
become an informational RFC, if accepted by the IETF. Two
reasons for this are

(a) Some of the advice is taken from other, already
    existing standards.

(b) Some of the issues are so controversial, that
    agreement on a standard is not possible.

The document is copied below, and is also available at:
  ftp://ftp.dsv.su.se/users/jpalme/draft-palme-maillist-00.txt

---------------------------------------------------------------

INTERNET-DRAFT                                      Jacob Palme
Network Working Group                  Stockholm University/KTH
draft-palme-maillist-00.txt                              Sweden
Expires August 1999                               February 1999

Good Mailing List Behaviour

Status of this Memo

This document is an Internet-Draft and is in full conformance
with all provisions of Section 10 of RFC2026.

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups.  Note that
other groups may also distribute working documents as
Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time.  It is inappropriate to use Internet-
Drafts as reference material or to cite them other than as
"work in progress."

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt

The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.

Copyright (C) The Internet Society 1998. All Rights Reserved.

Abstract

This memo summarizes common ideas on how good mailing lists should
behave. Some of this is taken from IETF standards, some is not. This
memo is not intended, itself, to become a standard, but might, if
accepted by the IETF, be published as informational RFC.

Table of contents

1.   Terminology and Scope
2.   Reserved E-mail Addresses
3.   Sending Requests to a List Expander
 3.1   Subscription Control
   3.1.1  To Subscribe
   3.1.2  To Unsubscribe
 3.2   To Get Information about a List
4.   Who may Post to a Mailing List
5.   SMTP Envelope
6.   Delivery Status Notifications
7.   Nested Lists
8.   Loop Control
9.   List Headers
10.  Header Munging
11.  Spam Control
12.  Groupware
13.  Security Considerations
14.  Copyright and Disclaimer
15.  Acknowledgments
16.  References
17.  Author's address

1.    Terminology and Scope

By a mailing list is in this specification meant an automatic agent
which has an e-mail address, and which will resend messages, sent to
this address via SMTP [RFC821], to all e-mail addresses in a list of
subscribers to the mailing list. This process of resending is
designated "expansion" of the mailing list. Note that lists which are
expanded by the sender's client before submission to the mail
transport system are not covered by this specification, even though
the word "mailing list" is sometimes used also for such lists.

This memo summarizes customary ideas on how good mailing lists should
behave. Some of this is taken from IETF standards, some is not.

2.    Reserved E-mail Addresses

Every mailing list has an e-mail address, named according to the same
conventions as for personal mailboxes, and reachable through the same
mail transport system as for personal mailboxes.

If the e-mail address of a mailing list is "[email-masked]" then
the following e-mail addresses are also reserved:

[email-masked]
[email-masked]

Messages sent to "[email-masked]" are usually handled by
an automatic process which performs common actions as requested in the
message. Sometimes all, or some, such messages are sent to a human
administrator of the list.

Messages sent to "[email-masked]" are sent to a human
administrator of the list, or cause a non-delivery notification (see
section 6. Delivery Status Notifications) in accordance with [RFC1891]
and [RFC1894], if the list administrator is not willing to handle
messages sent to this e-mail address.

3.    Sending Requests to a List Expander

Some, but not all, mailing lists accept commands sent in messages to
an automatic agent representing the list expander. The most common
address for such an agent is "[email-masked]", but other
addresses occur, such as "[email-masked]".

There is no agreed standard on the format of commands in such
messages, so it is good practice to accept a number of common variants
in either the subject or the text of the message to the agent. Such
commands are case-insensitive.

Common such commands are:

3.1   Subscription Control

The subscription control commands handle the subscription of the SMTP
sender of the request (not the name in the From: or Sender: header).
The mailing list expander may however find the name of the requestor
from the "From:" or "Sender:" field, in order to register the name.
This name is however not used in normal list expansion.

3.1.1 To Subscribe

Common commands are: sub, subscribe, join. Best is to support all of
them.

Sometimes the name of the requestor (not the e-mail address) is
specified after this command, for example "Subscribe Mary Woodfence".

3.1.2 To Unsubscribe

Common unsubscribe commands are: uns, unsubscribe, signoff, sign-off,
sign off, delete, leave, cancel, remove, rem, del. Best is to support
all of them.

3.2   To Get Information about a List

Common commands to retrieve information about a list are: help,
review, query, info, information. Best is to support all of them.

The information returned may include a textual description of the
purpose of the list and of which postings are acceptable to the list.
It may also include description of how to subscribe and unsubscribe
and where archives of the mailing list are kept. Some lists also
return a list of the subscribers of the list.

4.    Who may Post to a Mailing List

There may be different kinds of restrictions on who may submit
messages to a list. Common cases are:

- Anyone:           Anyone can submit messages to the list (warning,
                    see section 11. Spam Control below).
- Members only:     Only members are allowed to submit to the list
- Moderators only:  Only one or more designated moderators may
                    submit to the list

Other cases, such as geographical or domain name restrictions, or that
only a program, agent or filter may post, also occur.

The checking on who may submit to a list is done on the SMTP sender of
the message, not on the names in From: or Sender: fields in the
heading.

If a message is sent to a list, by someone who is not allowed to
submit to the list, this can be handled in either of two ways:

(a) Forward the message to the moderator of the list, who decides
    whether to accept or reject the message.

(b) Send a non-delivery notification to the SMTP sender of the
    rejected message, in accordance with [RFC1891] and [RFC1894].

5.    SMTP Envelope

The SMTP sender [RFC821] of a message after expansion should be the
list owner or maintainer [RFC1123], not the original sender. For
small, closed lists, the option of retaining the SMTP sender of the
original sender can also occur.

6.    Delivery Status Notifications

Delivery Status Notification [RFC1891], [RFC1894] requests are usually
not forwarded by mailing list expanders. Instead, notifications are
sent when the message arrives at the list, and the list maintainer can
request notifications when the messages are delivered to list
subscribers.

An exception to this is small, closed lists, where sometimes Delivery
Status Notification requests are forwarded through the list, and the
notifications are sent back to the original sender.

7.    Nested Lists

A subscriber of a mailing list can be another mailing list. This is
called "nested lists". Nested lists are used for efficiency reasons
and in order to distribute the management of different parts of the
subscriber space.

Nested lists can have a hierarchical structure or be looped, see
Figure 7.1:

  Figure 7.1   Examples of hierarchical and looped nesting

                  Hierarchical                  Looped

                     Top list               +---<-List A-<-+
                        |                   |         |    |
            +-----<-----+----->-----+     List B--->--+--<-+
            |           |           |       |              |
         Sublist A  Sublist B  Sublist C    +---->------List C
            |
     +--<---+---->---+
     |               |
  SubList A1  Sublist A2

With a hierarchical structure, contributions intended for all members
of the whole set of lists must be sent to the top list. Theoretically,
messages intended for only a brach of the tree might be sent to the
top of that branch, but this is usually not recommended, because users
have difficulty understanding it.

A way to stop contributions to other branches than the top list is to
designated that the sublists will only accept contributions from their
immediate superior in the nesting structure.

Looped nesting can cause loops, where the same message circles
indefinitely between the lists. How such loops can be avoided is
described in section 8. Loop Control. Another alternative is to only
use hierarhically nested lists. It is, however, sometimes desirable to
allow looped nesting, for example when one or more of the nested lists
is a groupware system which accepts local contributions using other
submission methods than e-mail (see section 12. Groupware). Looped
nesting will also avoid the problem with contributions submitted to
the wrong branch of a hierarchical structure.

8.    Loop Control

Loops can occur because lists are nested (see section 7. Nested
Lists). Even if lists are not intended to be nested, it is advisable
to employ loop control techniques, because nesting of lists can happen
by mistake.

Mailing lists commonly employ one or more of the following techniques
for avoiding loops and duplicates. It is better to employ more than
one of these techniques:

(1) Add a "Received:" header to all messages passing the list. If a
    mailing list recognizes its own "Received:" header in an incoming
    message, such a message is dropped. No non-delivery notification
    should be sent in this case (since it might cause another loop).

    Note: The content of the Received header should be different from
    what is added by the mail transport agent during ordinary routing
    of e-mail, since otherwise a message routed by this mail transport
    agent may at a later time be rejected by the mailing list, even
    though it has not actually passed the list.

(2) Store a data base of the Message-ID-s of messages which have
    passed the list, and reject incoming messages whose Message-ID
    is on this list. To achieve loop control, this list need not be
    kept for a long time, a week is enough.

(3) Store a data base of the content or checksum of messages which
    have passed this list, and use it in the same way as the
    Message-ID. The advantage with this is that it may work even when
    a message did not have any Message-ID or when some badly behaving
    list expander has removed or modified the Message-ID.

9.    List Headers

A mailing list expander should add headers to the mailing list
according to [RFC2369]. Examples:

List-Help: <mailto:[email-masked]?subject=help> (List Instructions)
List-Unsubscribe: <mailto: [email-masked]?subject=unsubscribe>
List-Subscribe: <mailto: [email-masked]?subject=subscribe>
List-Archive: <http://www.foo.net/flowers-archive>
List-Post: <mailto:[email-masked]> (Postings are Moderated)
List-Owner: <mailto:[email-masked]> (Grant Neufeld)

10.   Header Munging

Apart from what is specifed in sections 8. Loop Control and 9. List
Headers, a mailing list expander should not in any way modify the
heading of a message. In particular, the list should not change the
Message-ID, not add "Resent-", "From:", "Sender:", "Auto-Submitted:"
or "Reply-To:". The practice to add the e-mail address of the list in
a "Reply-To:" header is common, but is not recommended. Instead, use
the "List-Post:" command from [RFC2369].

11.   Spam Control

Many mailing list expanders employ various methods to counteract
spamming. Examples of such methods are:

(1) Do not allow non-subscribers to post to the list.

(2) Check all submissions by a human moderator before acceptance.

(3) Employ various filtering techniques to recognize spams, such as
    multiple occurence of the same message sent to different mailing
    lists. Since such techniques may reject legitimate messages,
    rejected messages should be passed to a human moderator for
    checking.

12.   Groupware

A groupware product may appear as a mailing list to people accessing
it via e-mail, and may at the same time appear as a forum to people
accessing it via other user interfaces, such as HTTP [RFC2068]/HTML
[RFC1866] or own protocols for this particular groupware.

Such groupware products may allow addition of e-mail addresses as
subscribers to a forum in the same way as groupware users are added as
members of the forum.

13.   Security Considerations

Allowing people to retrieve lists of members of mailing lists may be
misused by spammers and other people using these names for no-goood
purposes.

Allowing anyone to post to a list may be misused by spammers. See see
section 11. Spam Control.

Loop control may incur some risk of messages disappearing, but this
should normally not happen.

Loop control with Message-ID can be misused to stop unwanted messages,
but this would be difficult, since the offender must send the false
message with the same Message-ID before the message to be stopped.

Spam control may incur some risk of messages disappearing. A way to
reduce this risk is to forward rejected messages to a human moderator
for checking.

A well-known problem with moderated mailing lists is that if the
moderator is sick, on holiday, or otherwise occupied, the list ceases
to work.

14.   Copyright and Disclaimer

The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to pertain
to the implementation or use of the technology described in this
document or the extent to which any license under such rights might or
might not be available; neither does it represent that it has made any
effort to identify any such rights. Information on the IETF's
procedures with respect to rights in standards-track and standards-
related documentation can be found in BCP-11. Copies of claims of
rights made available for publication and any assurances of licenses
to be made available, or the result of an attempt made to obtain a
general license or permission for the use of such proprietary rights
by implementors or users of this specification can be obtained from
the IETF Secretariat."

The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF Executive
Director.

This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implmentation may be prepared, copied, published and
distributed, in whole or in part, without restriction of any kind,
provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of developing
Internet standards in which case the procedures for copyrights defined
in the Internet Standards process must be followed, or as required to
translate it into languages other than English.

The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.

15.   Acknowledgments

Many people have helped with the production of this document. Of
special value have been ..... (not completed yet)

16.   References

[RFC821]    Simple Mail Transfer Protocol. J. Postel. Aug-01-
            1982. (Format: TXT=124482 bytes) (Obsoletes
            RFC0788) (Also STD0010) (Status: STANDARD)

[RFC822]    Standard for the format of ARPA Internet text
            messages. D. Crocker. Aug-13-1982. (Format:
            TXT=109200 bytes) (Obsoletes RFC0733) (Updated by
            RFC1123, RFC1138, RFC1148, RFC1327, RFC2156) (Also
            STD0011) (Status: STANDARD)

[RFC1123]   Requirements for Internet hosts - application and
            support. R.T. Braden. Oct-01-1989. (Format:
            TXT=245503 bytes) (Updates RFC0822) (Updated by
            RFC2181) (Status: STANDARD)

[RFC1866]   Hypertext Markup Language - 2.0. T. Berners-Lee &
            D. Connolly. November 1995. (Format: TXT=146904
            bytes) (Status: PROPOSED STANDARD)

[RFC1891]   SMTP Service Extension for Delivery Status
            Notifications. K. Moore. January 1996. (Format:
            TXT=65192 bytes) (Status: PROPOSED STANDARD)

[RFC1894]   An Extensible Message Format for Delivery Status
            Notifications. K. Moore & G. Vaudreuil. January
            1996. (Format: TXT=77462 bytes) (Status: PROPOSED
            STANDARD)

[RFC2068]   Hypertext Transfer Protocol -- HTTP/1.1. R.
            Fielding, J. Gettys, J. Mogul, H. Frystyk, T.
            Berners-Lee. January 1997. (Format: TXT=378114
            bytes) (Status: PROPOSED STANDARD)

[RFC2369]   The Use of URLs as Meta-Syntax for Core Mail List
            Commands and their Transport through Message
            Header Fields. G. Neufeld, J. Baer. July 1998.
            (Format: TXT=30853 bytes) (Status: PROPOSED
            STANDARD)

17.   Author's address

Jacob Palme                          Phone: +46-8-16 16 67
Stockholm University/KTH             Fax: +46-8-783 08 29
Skeppargatan 73                      E-mail: [email-masked]
S-115 30 Stockholm, Sweden

------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

Date: 18 Feb 1999 15:19:35 -0500
From: Tim Showalter <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

> Date: Thu, 11 Feb 1999 17:28:46 -0800 (PST)
> From: Ned Freed <[email-masked]>

> Recently we've run into a number of cases where agents will simply refuse
> messages with 8bit in the headers. Judging from the variety of error messages
> there are at least three implementations out there that have this
> characteristic.  One of them appears to be something associated with local
> delivery in Cyrus, [...]

The behavior is that Cyrus's deliver program exits with EX_DATAERR if it
sees 8-bit in the headers, causing sendmail to respond with a bounce
containing "Data format error".  Cyrus versions <= 1.5.10 have this behavior.

Versions >= 1.5.14 don't, although there's a config option for it.
These versions change 8-bit characters to something else.

Many sites, especially in Europe, have disabled the code that checks for 
this, and have been fine.

The paranoia was implemented in part because the character set can not
be deduced.

-- 
Tim Showalter <[email-masked]>

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: Transmission issues for transition to UTF-8 Headers 
Date: Mon, 15 Feb 1999 10:24:03 -0500
Sender: [email-masked]

> Two Q-P MUAs can tunnel 8-bit message bodies through a 7-bit MTA.
> 
> Q-P proponents pointed to this fact in 1991 and said how wonderful it
> would be to avoid the costs of adding 8-bit support to MTAs. What they
> ignored was the cost of adding Q-P support to MUAs.

No, this wasn't ignored.  At the time it was widely believed to
be more difficult to upgrade MTAs than MUAs, since a user could
choose his MUA but the MTA was imposed by the user's environment.
What actually happened is that several well-publicized security 
bugs caused people to upgrade their MTAs much more quickly than 
anticipated.

> Sending an unencoded 8-bit message body has a much higher chance of
> success than sending Q-P. You're much less likely to encounter a 7-bit
> MTA than to encounter a non-Q-P MUA.

That's today.  It wasn't true in 1991 when the decision was made.

Keith

To: [email-masked]
Xref: clerew local.mime:99
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Mon, 15 Feb 1999 11:07:17 GMT
Lines: 17
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>Charles Lindsey writes:
>> So long as the reader
>> displays _something_ distinct for every possible combination of octets
>> received, nothing is going to break.

>Replacing the proper 8859-1 characters with ``_something_'' will annoy
>the users.

8859-1 characters are not "proper" in this context.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:100
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Transmission issues for transition to UTF-8 Headers
Date: Mon, 15 Feb 1999 11:10:09 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> "Bart Schaefer" <[email-masked]> writes:

>I think I agree with Keith that a new header field doesn't really help.
>I'm not a UTF-8 expert by any means, but it sounds like recognizing the
>strings individually is sufficient.  We've got these cases:

>1.  Plain unlabeled (non-UTF-8) octets.
>2.  A UTF-8 string, which can be recognized by its structure.
>3.  An RFC2047 string, which can be recognized and the character set
>    extracted.
>4.  An RFC2231 string, which can be recognized and the character set
>    and language extracted.

There is also case 5:

5.   The header cannot be recognised as any of the above.

In which case, it is broken and all bets are off. If some agent manages to
produce an interpretation which makes sense to its readers, then good luck
to it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: 13 Feb 1999 00:31:07 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Transmission issues for transition to UTF-8 Headers
Sender: [email-masked]

Two Q-P MUAs can tunnel 8-bit message bodies through a 7-bit MTA.

Q-P proponents pointed to this fact in 1991 and said how wonderful it
would be to avoid the costs of adding 8-bit support to MTAs. What they
ignored was the cost of adding Q-P support to MUAs.

Sending an unencoded 8-bit message body has a much higher chance of
success than sending Q-P. You're much less likely to encounter a 7-bit
MTA than to encounter a non-Q-P MUA.

This is why just-send-8 is a much more popular strategy for SMTP clients
than convert-to-Q-P (or bounce). This is the behavior of qmail, for
example, and exim, and sendmail versions from 1993 through 1995, and
sendmail versions after 1995 with the 8 flag (standard in Europe).

Of course, implementors can't learn any of this from the RFCs. RFC 821
assumes 7-bit messages. The 8BITMIME specification prohibits clients
from sending 8-bit data except in MIME messages to 8BITMIME servers. The
IESG refuses to admit that servers rejecting unencoded 8-bit bodies are
laughed out of the marketplace.

An interesting phenomenon here is that all the MTAs mentioned above
declare 8BITMIME--even though, contrary to the 8BITMIME religion, they
_don't_ apply Q-P conversion. Why do they declare 8BITMIME? Because this
prevents Q-P conversion from religious SMTP clients.

Dave Crocker writes:
> The recent changes involving MIME and ESMTP represent an 
> extraordinarily successful piece of work,

There are certainly some good things that one can say about MIME. The
8BITMIME SMTP extension is not one of them.

> Even the smallest change is measured in years.  The longest in decades.

Indeed, the IETF has managed to screw up the 8-bit transition so badly
that, two decades after the demand for 8-bit mail was widely recognized,
implementors are still being told that 7-bit MTAs are acceptable.

---Dan

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Fri, 12 Feb 1999 17:58:53 -0500
Sender: [email-masked]

> > So long as the reader
> > displays _something_ distinct for every possible combination of octets
> > received, nothing is going to break.
> 
> Replacing the proper 8859-1 characters with ``_something_'' will annoy
> the users.

As explained earlier, while this might be a problem with other character
sets, this is extremely unlikely to happen with 8859/1.

Keith

From: Keith Moore <[email-masked]>
To: "Bart Schaefer" <[email-masked]>
cc: [email-masked]
Subject: Re: Transmission issues for transition to UTF-8 Headers 
Date: Fri, 12 Feb 1999 17:57:42 -0500
Sender: [email-masked]

> On Feb 12,  7:27pm, Dave Crocker wrote:
> } Subject: Transmission issues for transition to UTF-8 Headers
> }
> } For moving to UTF-8 headers, we need an esmtp option and, I suppose, an 
> } 822(bis) header.
> 
> I think I agree with Keith that a new header field doesn't really help.
> I'm not a UTF-8 expert by any means, but it sounds like recognizing the
> strings individually is sufficient.  We've got these cases:
> 
> 1.  Plain unlabeled (non-UTF-8) octets.
> 2.  A UTF-8 string, which can be recognized by its structure.
> 3.  An RFC2047 string, which can be recognized and the character set
>     extracted.
> 4.  An RFC2231 string, which can be recognized and the character set
>     and language extracted.

I don't know of a case where it's valid to include 2231 and 2047 
in the same header field.  2231 is for parameter values and 
machine-readable text which aren't necessarily displayed to the
recipient; 2047 is for human-readable strings in "plain text" fields
which are intended for display to the recipient.

For similar reasons, I don't think that 2231 strings should be 
translated to or from UTF-8, and it's not clear that we need
to ever send them in "raw" form even if they are originated
in UTF-8.  Filenames etc. need to stay intact end-to-end.
 
Keith

Date: 12 Feb 1999 18:12:11 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

Charles Lindsey writes:
> So long as the reader
> displays _something_ distinct for every possible combination of octets
> received, nothing is going to break.

Replacing the proper 8859-1 characters with ``_something_'' will annoy
the users.

They'll ask for the feature to be removed, or at least turned off by
default.

You may think that they're overreacting. But you'll have to pay
attention to them if you don't want an unnecessarily slow transition.

---Dan

From: "Bart Schaefer" <[email-masked]>
Date: Fri, 12 Feb 1999 10:05:46 -0800
Comments: In reply to Dave Crocker <[email-masked]> "Transmission issues for transition to UTF-8 Headers" (Feb 12,  7:27pm)
To: [email-masked]
Subject: Re: Transmission issues for transition to UTF-8 Headers
Sender: [email-masked]

On Feb 12,  7:27pm, Dave Crocker wrote:
} Subject: Transmission issues for transition to UTF-8 Headers
}
} For moving to UTF-8 headers, we need an esmtp option and, I suppose, an 
} 822(bis) header.

I think I agree with Keith that a new header field doesn't really help.
I'm not a UTF-8 expert by any means, but it sounds like recognizing the
strings individually is sufficient.  We've got these cases:

1.  Plain unlabeled (non-UTF-8) octets.
2.  A UTF-8 string, which can be recognized by its structure.
3.  An RFC2047 string, which can be recognized and the character set
    extracted.
4.  An RFC2231 string, which can be recognized and the character set
    and language extracted.

Due to vagaries of old software, one might end up with several of those in
the same header field; I think only 2047 and 2231 end up being mutually
exclusive (?).

What information is the new header field going to carry to help an MTA or
an MUA deal with this situation?  Who's responsible for canonicalizing a
mess like that so that a single header field could describe it?

-- 
Bart Schaefer                                                   Zanshin
http://www.well.com/user/barts                   http://www.zanshin.com

To: [email-masked]
Xref: clerew local.mime:88
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Fri, 12 Feb 1999 11:29:47 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>Charles Lindsey writes:
>> Just what you you propose that a type-2 reader should do when it sees this
>> special header that it would not do if this header was absent?

>Here are the two reader strategies that I've mentioned:

>   (1) Display every 8-bit header field as UTF-8.

>   (2) Display all the 8-bit header fields in a message as UTF-8 _if_
>       the message contains the following special header field: ...

Yes, but readers and display are not the problem. So long as the reader
displays _something_ distinct for every possible combination of octets
received, nothing is going to break. If it looks ugly, then user pressure
will force it to look prettier over time.

The question is whether MTAs (sendmail and friends) can do anything useful
when they see this header that they could not have done without it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Fri, 12 Feb 1999 19:27:23 +0800
To: [email-masked]
From: Dave Crocker <[email-masked]>
Subject: Transmission issues for transition to UTF-8 Headers
Sender: [email-masked]

Folks,

I've changed the subject field so that matters about UA display, and the 
like, can be treated separately from the meta-question of interacting email 
services, namely what is the basic approach that should be used when a 
UTF-8 host wants to sent to some other host.

This subject line is intended to focus just on the question of coordination 
between sending and receiving systems, in other words on the exchange 
protocol, SMTP, and the object being exchanged, in this case 822 or 822bis 
since headers are the issue.

This particular community has quite a large amount of experience making 
making changes to a a very installed base.  It has upgraded that base more 
than once.  The recent changes involving MIME and ESMTP represent an 
extraordinarily successful piece of work, and we should not have to 
re-invent, experiment, or otherwise spend a great deal of time on a topic 
that is quite clearly a repeat of these earlier times.

The transition to UTF-8 headers has all of the characteristics of the 
transition to MIME and ESMTP and we should use what we've learned.

What we've learned:

1.  If you insist on "just sending" the new stuff, without getting a 
statement of support by the receiver, then you MUST send it in a fashion 
which is safe for old recipients.  That is the model used for MIME and that 
is the reason that MIME has such a horrendous appearance, as well as such a 
safe record of deployment.  A computer scientist shudders at MIME's 
ugliness.  An engineer marvels at its successful deployment over an 
existing, global base of users.

2.  If there is any real chance of impact on the recipient, such as 
breaking their software, the sender MUST get permission before initiating a 
'new' behavior.  That is what ESMTP options are for.  They work 
dandy.  Since 8-bit encoding is well-known to be a source of problems for 
recipients not ready for it, it is clear that sending 8-bit headers needs 
either to have 7-bit encoding or explicit recipient go-ahead.  We fought 
the "just send 8 bits" wars years ago.  We should not have to fight them again.

3.  Unlabeled defaults tend to be problematic, since there is no detecting 
when they are changed.  While it offends one's sense of "efficiency" it's 
worth the extra bits to label a long-lived object explicitly.

At 01:50 AM 2/5/99 +0000, D. J. Bernstein wrote:
>The short-term goal is to allow messages with unencoded UTF-8 in the

4.  When making a transition for a global installed base (and actually the 
rule applies equally for a much smaller base) there is no semantic content 
to the term "near-term".

Even the smallest change is measured in years.  The longest in 
decades.  Hence, serious efforts at transition must design a "transition" 
mode which is capable of being used permanently.

MIME's 7-bit encoding is an example.  While one might claim that 8bitmime 
is a disaster, that claim misses the larger fact that we exchange binary 
data with each other today quite comfortably but could not do so before the 
MIME effort was started.  The fact that we send it around "inefficiently" 
is irritating to some, but is clearly secondary to the fact that we can and 
do use the exchange mechanism successfully.

>The long-term goal is to eliminate the implementation burden of multiple
>character sets and =??=. This takes one extra step:
>
>   (4) Mail writers have to convert all outgoing messages from the local
>       character set to unencoded UTF-8.
>
>Eventually all character-set markers can be removed.

5.  Eventually, COBOL will disappear.  Eventually, no one will ever program 
in assembly language.  Eventually, we will have world peace...

So how do we factor in "eventually" to practical planning efforts?

>Chris Newman writes:
>> * Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
>>   both top level headers and nested message/rfc822 headers.
>
>That doesn't survive a cost-benefit analysis.

Right.  Instead it just survives the test of global, practical demonstration.

In other words, the alternative doesn't survive a credibility analysis, 
unless you want to declare the world's movement to X.400-based UA's a success.

>Your conversion is safe in a fantasy world where all message readers

"fantasy"?

gosh.

why would anyone find it distasteful to interact with such a 
participant?  I just can't imagine.

ANYHOW...

For moving to UTF-8 headers, we need an esmtp option and, I suppose, an 
822(bis) header.

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PRESIDENTS DAY OPEN HOUSE, 2/13
<http://www.brandenburg.com/misc/presday/presday-invite.gif>

Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400  MALAYSIA
Brandenburg Consulting
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464             675 Spruce Dr., Sunnyvale, CA 94086  USA

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: 8-bit message bodies, Q-P conversion, and 8BITMIME 
Date: Fri, 12 Feb 1999 02:44:38 -0500
Sender: [email-masked]

> Keith Moore claims that there were SMTP servers in 1991 that _crashed_
> when they saw 8-bit bodies.
> 
> Over the last few years I have repeatedly asked Moore for evidence to
> back up this claim. He has never provided any such evidence.

This is my memory of what people were saying at the time.  Various
problems were reported with 8bit bodies, and some of those reports
were of mailers that would stop functioning - refusing to process
other messages - when presented with 8bit input.    

> In 1997 I reviewed the old ietf-smtp archives and found a 1991 message
> from Robert Ullmann on the same issue. Guess what? Contrary to Moore's
> claims, nobody could find any SMTP servers that crashed on 8-bit bodies.

Face to face discussions aren't recorded in the archives.

> Amazingly enough, the IETF _still_ doesn't admit that SMTP servers are,
> as a practical matter, required to accept 8-bit messages.

This is probably an oversight which should be corrected.  But at the
time that the SMTP 8BITMIME stuff was defined, there was nothing
resembling clear consensus that it was acceptable to just-send-8,
nor was there consensus that it was acceptable to require MTAs to 
accept 8bit and downgrade (when necessary) to 7bit.  And the discussion
on those topics was so long and heated (even without your help, Dan!)
that there was for many years a reluctance to revisit the issue.

Keith

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Fri, 12 Feb 1999 02:36:18 -0500
Sender: [email-masked]

> Keith has mentioned two more strategies:
> 
>    (3) Display an 8-bit header field as UTF-8 _if_ it is a valid UTF-8
>        string.
> 
>    (4) Display all the 8-bit header fields in a message as UTF-8 _if_
>        all the fields are valid UTF-8 strings _and_ the message contains
>        the following special header field: ...

Both of these are mis-interpretations.  more precise descriptions:

3) if a phrase, comment, or *text field containing 8bit characters is 
   a valid UTF-8 string in its entirety, display it as UTF-8.  
   otherwise display it in a local character set.

4) if a message does not contain a special header field, display all
   8bit characters in a local character set.  if the message does
   contain the special header field, follow rule 3.

note that #4 is just for comparison; I don't consider the extra
header field useful enough to be worth the trouble.

you have to make the display-as-UTF-8 vs. display-as-local decision 
with smaller granularity than a header field, because each 
phrase/address/comment in an address field may come from a different 
source, using a different character set.

Keith

Date: 12 Feb 1999 07:17:07 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

Dave Crocker writes:
> declaring your objections to lack working group support.

What objections are you talking about?

I was merely making some historical comments, here in ietf-822, to
explain to Charles Lindsey why the 822bis grammar is so painful.

You then started claiming ``VERY strong consensus'' as the result of
``deliberations'' that nobody else seems to remember participating in.

> What IS a fact is that your recent efforts have received NOT ONE
> message of support from anyone else.

Why would I need any support? All of my comments can easily be verified
from the DRUMS archives.

---Dan

Date: Fri, 12 Feb 1999 06:21:43 +0100
To: Ned Freed <[email-masked]>
From: Patrik =?iso-8859-1?Q?F=E4ltstr=F6m?=  <[email-masked]>
Subject: Re: The transition to UTF-8 header fields
Cc: [email-masked]
Sender: [email-masked]

At 17.28 -0800 1999-02-11, Ned Freed wrote:
> Recently we've run into a number of cases where agents will simply refuse
> messages with 8bit in the headers.

Nowadays I have only seen the problem when talking about gateways 
from SMTP to some other "messaging plattform" which uses a different 
character set than US-ASCII. The problems arises when the final 
message on the "other side" of the gateway consists of information 
from both the headers and the body of a message. Normally, the From: 
and the Subject: headers are used for information for the receiver -- 
and those do often include characters which doesn't exist in the 
character set used on the other side of the gateway.

The plattforms I am talking about in perticular are gateways to SMPP 
which runs on X.25, and is used when sending SMS messages in the GSM 
system.

The combination of limited character set and new technology (which 
means non-email-aware developers) makes those products which pop up 
from whatever rock you lift pretty bad.

By talking to the developers that they have to support the 8BITMIME 
ESMTP extension, it is easy to force them to have a look at character 
set issues in general.

    Patrik

Date: Thu, 11 Feb 1999 17:28:46 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: The transition to UTF-8 header fields
To: Patrik =?iso-8859-1?Q?F=E4ltstr=F6m?= <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Gateways to non-SMTP environments generally can not handle 8-bit
> characters in headers. With can not I talk about anything between
> crashing to non-delivery of that specific message, to just passing on
> the data as being corrupt. The gateway itself can though handle QP or
> Base64 CTE in the body and headers correctly.

Back in the early 90's we used to see problems with SMTP servers going into
infinite loops or getting desynchronized during the SMTP session when they were
presented with 8bit in message headers. As far as we could tell the former was
associated with a particular sendmail version/configuration -- we never
determined specificaly why it behaved this way, only that it did, and that the
sites where it happened were seriously pissed when they found their load
averages above 30 as a result of all the looping processes out there.

The latter case of session desynchronization was even more problematic in that
it didn't seem to be repeatable. I don't recall which MTA was associated with
the problem, assuming we ever figured that out.

There was also a desynchronization problem not specific to headers that
resulted from some SMTP servers having code in them to do TELNET options
negotiation.

We were forced as a result of this to stop emitting 8bit in message headers.
PMDF is 8-bit clean internally, but when customers yell at you for something
that's obviously a standards violation and which is causing problems, you don't
have a lot of choice in what you do. So we adopted a complex strategy, using
RFC 2047 encodings in cases where they are allowed and using mnemonics and such
in others, to avoid 8bit in message headers.

Of course once you stop doing something illegal you cannot tell if the
installed base is getting better at tolerating it or not. But  as it turns out
our 8bit downgrading logic wasn't perfect -- there were a couple of cases I
missed where 8bit could sneak through. And this has given us some good
indications that things are getting worse, not better.

Recently we've run into a number of cases where agents will simply refuse
messages with 8bit in the headers. Judging from the variety of error messages
there are at least three implementations out there that have this
characteristic.  One of them appears to be something associated with local
delivery in Cyrus, one of them appears to be associated with some gateway to or
from Lotus Notes, and the third we've never been able to identify. And
remember, the cases are coming to light when messages contain 8bit
in very peculiar places, since we catch the obvious ones. (And of course
now we're modified our code to even get the peculiar ones that used to
get through.)

				Ned

Date: Fri, 12 Feb 1999 09:24:23 +0800
To: "D. J. Bernstein" <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: Why the 822bis grammar is so painful
Cc: [email-masked]
Sender: [email-masked]

At 12:02 AM 2/12/99 +0000, D. J. Bernstein wrote:
>on removing the tokenizer. Nobody else on DRUMS seems to remember ever
>supporting Resnick's removal of the tokenizer.

Again, Dan, you are making a statement that goes beyond the facts.  It's 
curious how consistently you do that.

Only one person has made an explicit statement about their memory of the 
precise details.  (It wasn't me, because I'm not willing to go down that 
rat-hole with you.)  Yet you persist in making a positive claim of group 
failed memories.

What IS a fact is that your recent efforts have received NOT ONE message of 
support from anyone else.  Since this has gone on for a some days already, 
there should have been some other mail in support of your claims and none 
has appeared.

This creates a prima facie case for declaring your objections to lack 
working group support.  Hence, it provides that much more basis for 
terminating the endless delay the working group has suffered through and, 
instead, for shipping its product.

If the documents are so horribly flawed, as you believe, there will be 
clear evidence from the community.

>How can _I_ prevent publication? I'm not in charge of the specs. I've
>merely been pointing out problems.

A penalty for the IETF's approach to openness is that it must and does bend 
over backwards to attend to all input.  Many participants even feel 
obligated to put up with all manner of personal crap, just so there is 
sufficient technical due diligence.

My own, whimsical phrasing is that the IETF is quite good at putting up 
with people who are obnoxious or ignorant... but not both.  Only when a 
person shows a very sustained pattern of being both are they (finally) 
largely ignored.

Alas, you frequently provide legitimate input, however difficult it is to 
handle the packaging it comes in.  Most of us are obnoxious only part of 
the time, or at least we are not obnoxious ALL of the time.

Only rarely do we experience someone who seems to lack any control at 
all.  (Hope springs eternal.  About 25 years ago, the community experienced 
its first email non-stop flamer, and he seemed unable to control it, but 
things eventually got much better; it only took him about 18 years to calm 
down.  By contrast, some of us have always had SOME control, but never 
quite graduate to developing ENOUGH...)

In any event, the group's desire to do good work obligates it to attend to 
that input, in spite of the real and deep pain of dealing with that packaging.

But part of the social contract of participating in an effort like this is 
a community desire to make forward progress, while giving credence to all 
participants.  This requires real and difficult compromise by ALL 
participants, which often means losing particular battles that one or 
another person feels is important.

In other words, the social contract is to choose the battles, balancing 
various factors.  Someone who only lobbies for their own point of view and 
who shows no willingness to compromise and participate in the rest of the 
balancing act is inherently problematic.

IETF specifications are always flawed.  They are always compromises.  There 
is always room for improvement.  As with any development effort, it becomes 
necessary to balance the problems of the flaws with the need to ship product.

>It's not my fault that, for example, Klensin is continuning to ignore

Alas, Dan, nothing is ever YOUR fault.  Worse, you always see things as 
SOMEONE's "fault".  You always translate issues into ad hominems.

>No. When you talk about every use of a person's name as ``ad hominem''
>you simply make yourself sound illiterate.

When you over-interpret the nature and meaning of a statement, you show 
yourself to be psychologically disturbed, probably in the realm of 
schizoid, but I'm not current on modern classifications.

Now, gosh, wasn't THAT a productive micro-exchange, Dan?

It is not the use of a person's name that I am citing as the basis for 
pointing out your constant devolution to ad hominems.  It's those curious 
qualifiers and claims that you associate with the names.  It's the context, 
tone and semantics in which the names are used.  Surely you can comprehend 
the distinction, Dan?

At any rate, you are pursuing a concern about DRUMS, so the topic needs to 
be pursued on that list or with the IESG, per IETF rules.  Further 
discussion of your discomforts about DRUMS, on this mailing list, are 
inappropriate.

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PRESIDENTS DAY OPEN HOUSE, 2/13
<http://www.brandenburg.com/misc/presday/presday-invite.gif>

Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400  MALAYSIA
Brandenburg Consulting
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464             675 Spruce Dr., Sunnyvale, CA 94086  USA

Date: 12 Feb 1999 01:01:08 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: 8-bit message bodies, Q-P conversion, and 8BITMIME
Sender: [email-masked]

Keith Moore claims that there were SMTP servers in 1991 that _crashed_
when they saw 8-bit bodies.

Over the last few years I have repeatedly asked Moore for evidence to
back up this claim. He has never provided any such evidence.

In 1997 I reviewed the old ietf-smtp archives and found a 1991 message
from Robert Ullmann on the same issue. Guess what? Contrary to Moore's
claims, nobody could find any SMTP servers that crashed on 8-bit bodies.

Keith Moore writes:
> At the time MIME was defined there were reports of several MTAs in 
> deployment which could not handle 8bit characters in the message 
> body - some would corrupt the message; others would get wedged or 
> crash.

Corrupt 8-bit bodies, yes. Crash, no.

> I don't think it's fair to brand this as a failure, because it slowed
> the deployment of "just send 8",

sendmail started doing just-send-8 by default in 1993. See
http://pobox.com/~djb/proto/smtp/8bitmime.html for further discussion.

Amazingly enough, the IETF _still_ doesn't admit that SMTP servers are,
as a practical matter, required to accept 8-bit messages.

---Dan

Date: 12 Feb 1999 00:22:28 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

Charles Lindsey writes:
> Just what you you propose that a type-2 reader should do when it sees this
> special header that it would not do if this header was absent?

Here are the two reader strategies that I've mentioned:

   (1) Display every 8-bit header field as UTF-8.

   (2) Display all the 8-bit header fields in a message as UTF-8 _if_
       the message contains the following special header field: ...

Keith has mentioned two more strategies:

   (3) Display an 8-bit header field as UTF-8 _if_ it is a valid UTF-8
       string.

   (4) Display all the 8-bit header fields in a message as UTF-8 _if_
       all the fields are valid UTF-8 strings _and_ the message contains
       the following special header field: ...

---Dan

Date: 12 Feb 1999 00:02:51 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

Crocker says ``The DRUMS working group deliberated on this point quite
carefully'' but won't provide a shred of evidence. Nobody else on DRUMS
seems to remember any such deliberations.

Crocker and Newman say that there was, and still is, consensus in DRUMS
on removing the tokenizer. Nobody else on DRUMS seems to remember ever
supporting Resnick's removal of the tokenizer.

There's extensive evidence in the DRUMS archives that Resnick considered
the removal of the tokenizer to be his own personal decision, within his
editorial discretion. I don't see any indication of tokenizer discussion
in any of the subsequent DRUMS meeting minutes.

When did these ``deliberations'' take place? Where are the people who
participated? Where are the people who support Crocker's ``consensus''?
If Crocker is telling the truth, why can't he answer these questions?

> as it is, one could argue that you are pursuing a deprivation of service 
> attack, by preventing publication of the new IETF specs,

How can _I_ prevent publication? I'm not in charge of the specs. I've
merely been pointing out problems.

It's not my fault that, for example, Klensin is continuning to ignore
the working group consensus on the QUIT issue, and the EHLO issue, and
the VRFY issue. It's not my fault that Klensin is trying to use 821bis
as a weapon against some popular client behavior that offends him. It's
not my fault that Klensin recommended that clients use a new EHLO syntax
that will fail with some existing servers, preventing mail delivery.

> It means introducing personal content.

No. When you talk about every use of a person's name as ``ad hominem''
you simply make yourself sound illiterate.

---Dan

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Thu, 11 Feb 1999 14:37:20 -0500
Sender: [email-masked]

> Keith Moore writes:
> > No, but I fail to see how having an extra header (or not) would
> > affect that outcome.
> 
> MUA implementors are generally willing to change the interpretation of a
> message _if_ a new header field is supplied.
> 
> This doesn't affect current users. The important question is whether the
> likely future benefits outweigh the implementation and deployment costs.
> 
> Without the header field it's more difficult to evaluate the current
> costs. In some cases such a change would impose huge extra costs on
> current users; if you ask implementors to do that then you lose.
> 
> Again, I'm not saying that the extra costs of what you're proposing are
> obviously larger than the benefits. But it _is_ obvious how the costs
> _could_ drastically slow down the transition. More study is required.

A similar argument was made in MIME for the MIME-Version header.   
Based on our experience with that, I'd say it has been more trouble 
than it was worth.  

Most implementors, I think, would accept UTF-8 without an extra header
if the RFC provided an adequate discussion of the issues and explained
why it isn't expected to be a problem.  But to some degree this  analysis
might need to be based on assumptions which character sets are used without 
encoding by the installed base.  The koi-8 charset, for instance, might 
be a problem if it's widely used in 8bit headers.  So I agree that more 
study would be useful.

Part of the additional study that you want could consist of an Experimental 
period for the RFC, in which implementors were encouraged to give feedback 
about the negative impact of using UTF-8 without an additional header.  

> > We will almost certainly need a UTF8 SMTP option anyway,
> 
> Why? Explain the benefits and costs. Remember that the argument for MTA
> Q-P conversion failed in the real world because it wildly understated
> the costs of Q-P support.

At the time MIME was defined there were reports of several MTAs in 
deployment which could not handle 8bit characters in the message 
body - some would corrupt the message; others would get wedged or 
crash.   Though the "downgrade to quoted-printable" advice seems
overly conservative today, it was right for the time.  I don't
think it's fair to brand this as a failure, because it slowed the
deployment of "just send 8", allowing MTA vendors to fix their
8bit body problems with minimal disruption.

But 8bit headers are more difficult.  These days, I don't hear many 
reports about 8bit body problems (other than some MTAs which refuse 
to relay 8bit mail to an MTA which doesn't support 8BITMIME).  
But I'm still hearing reports of MTAs failing in 'catastrophic' ways
when presented with 8bit headers.  It's hard to pin these down because
I'm just being told about them - I'm not seeing them firsthand.  But 
we need to be very careful about deploying something which might 
cause a significant number of failures in the mail system.

> Data points: qmail is 8-bit-clean. sendmail (8.8.0 and above) removes
> bytes 128-159. The Cyrus IMAP toaster, apparently, bounces any message
> with an 8-bit header field. Any other reports?

Current versions aren't nearly as relevant as what is deployed.
And like Y2K bugs, having a short list of things known not to break
doesn't give you confidence that other things won't break.  If you 
really want to know whether it is safe to just send 8 bits in headers, 
then you need to do a survey of all versions of all mail software 
with significant deployment.  This is beyond the resources of IETF.  
Instead, we go with the consensus of a set of experts.

Keith

Date: Thu, 11 Feb 1999 20:08:47 +0100
To: "D. J. Bernstein" <[email-masked]>, [email-masked]
From: Patrik =?iso-8859-1?Q?F=E4ltstr=F6m?=  <[email-masked]>
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

At 18.51 +0000 1999-02-11, D. J. Bernstein wrote:
>> the vast majority of deployed MTAs cannot handle 8bit headers,
>
> Data points: qmail is 8-bit-clean. sendmail (8.8.0 and above) removes
> bytes 128-159. The Cyrus IMAP toaster, apparently, bounces any message
> with an 8-bit header field. Any other reports?

Gateways to non-SMTP environments generally can not handle 8-bit 
characters in headers. With can not I talk about anything between 
crashing to non-delivery of that specific message, to just passing on 
the data as being corrupt. The gateway itself can though handle QP or 
Base64 CTE in the body and headers correctly.

   Patrik

Date: 11 Feb 1999 18:51:10 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

Keith Moore writes:
> No, but I fail to see how having an extra header (or not) would
> affect that outcome.

MUA implementors are generally willing to change the interpretation of a
message _if_ a new header field is supplied.

This doesn't affect current users. The important question is whether the
likely future benefits outweigh the implementation and deployment costs.

Without the header field it's more difficult to evaluate the current
costs. In some cases such a change would impose huge extra costs on
current users; if you ask implementors to do that then you lose.

Again, I'm not saying that the extra costs of what you're proposing are
obviously larger than the benefits. But it _is_ obvious how the costs
_could_ drastically slow down the transition. More study is required.

> We will almost certainly need a UTF8 SMTP option anyway,

Why? Explain the benefits and costs. Remember that the argument for MTA
Q-P conversion failed in the real world because it wildly understated
the costs of Q-P support.

> the vast majority of deployed MTAs cannot handle 8bit headers,

Data points: qmail is 8-bit-clean. sendmail (8.8.0 and above) removes
bytes 128-159. The Cyrus IMAP toaster, apparently, bounces any message
with an 8-bit header field. Any other reports?

---Dan

To: [email-masked]
Xref: clerew local.mime:70
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Thu, 11 Feb 1999 10:55:01 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> OK, so what would that look like? Presumably it would have to include
>> downgrading of some sort.

>one thing at a time.  first, you need a good spec for the format,

Well I recently posted here a summary of what USEFOR is proposing -
essentially all protocol words must be ASCII, stuff for humans may be
non-ASCII (phrases, unstructureds, values of some parameters, etc). There
will be a new USEFOR draft out shortly with detailed syntax (though still
needing some details fixing). That could be a good starting point.

>probably along with some strong discouragement to ship
>UAs that generate raw UTF-8 by default 
>(but support for reading it might not be such a bad idea)

>the downgrading document can follow  - actually it should specify
>translation in both directions.

Yes, something like that.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:71
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Thu, 11 Feb 1999 10:56:51 GMT
Lines: 13
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>These two strategies have exactly the same effect with type-1 readers.
>The second strategy works with type-2 readers; the first does not.

Just what you you propose that a type-2 reader should do when it sees this
special header that it would not do if this header was absent?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: Why the 822bis grammar is so painful 
Date: Thu, 11 Feb 1999 09:35:33 -0500
Sender: [email-masked]

> > Note, instead, that you are the only person submitting any messages in 
> > support of your current position.
> 
> Perhaps that's because people are sick of spending huge amounts of time
> writing comments that are ignored by the DRUMS management.

It's at least as possible that people are sick of arguing with you,
or being personally attacked by you in the process of arguing with you,
or watching other people being personally attacked by you.

> Then how come you're having so much trouble scraping up any evidence of
> that consensus? I accept that _you_ don't want the tokenizer any more,
> but where are all the other people who agree with you?

And where are all the other people who support your position?
I see a couple, but nothing like "rough consensus".

Keith

Date: Thu, 11 Feb 1999 15:56:56 +0800
To: "D. J. Bernstein" <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: Why the 822bis grammar is so painful
Cc: [email-masked]
Sender: [email-masked]

At 07:24 AM 2/11/99 +0000, D. J. Bernstein wrote:
>I've posted a missing-deliberations notice on DRUMS. Anybody who has
>spotted Dave Crocker's lost deliberations should let me know.

Always glad when Dan deviates fully from technical content and ventures his 
preferred realm of aberrant human behavior.  It is of course much more 
comfortable ground for me, too.

Unfortunately he takes a failure to respond to an irrelevant question as a 
"lost deliberation".  Such excessive interpretation of non-data is always 
curious to observe.

>Dave Crocker writes:
>> ad hominem
>
>You keep using that word. I do not think that word means what you think
>it means.
>
>An ad-hominem argument ignores the issues in favor of a personal attack
>against the opponent. For example, when Johnny Eriksson responded to a
>new SMTP client feature report by saying ``Idiot,'' that was an
>ad-hominem argument.

Danny boy, the term means "to the person".  It means introducing personal 
content.  It means distracting the discussion from real content and, 
instead, focusing on the person.  That is what you do.  Frequently.  You 
aren't the only one to do it.  Many of us have that weakness.  Some of us 
have had it far longer than you, so we know whereof its failings lie.

In any event, a personal "attack" is not always required for a statement to 
be "ad hominem".

>Perhaps that's because people are sick of spending huge amounts of time
>writing comments that are ignored by the DRUMS management.

Perhaps they are sick of your constantly introducing delays.

Many interpretations of human behavior are possible.  That's why ad hominem 
analysis of group processes like this are irrelevant.  What matters is the 
technical content of the process and the support for them.  At the moment, 
you are receiving no support for your objections.  With luck, others will 
prefer to see closure rather than the endless delay you seem to perfer.

>822bis and 821bis, in their current form, don't have a chance in hell of
>competing with my web pages. Why not ignore them and let them die of

your web pages?

ok.

but perhaps you meant the IETF specs?

If your pages are really that wonderful, they will become the dominant 
reference for email on the net and you will replace that function in the 
IETF.  sounds like a fair form of competition.  i'd be delighted to see it 
pursued.

as it is, one could argue that you are pursuing a deprivation of service 
attack, by preventing publication of the new IETF specs, so that we could 
then find out whether the community prefers those specs or your pages.

hmmm, sounds like you have a conflict of interest and that all these delays 
you are introducing in the IETF process actually serve YOUR interests 
rather than the community's.

>You know, legitimate standards organizations don't have this problem.

by gosh, you are right.  they don't.  it's only the pissy, unprofessional, 
illegitimate IETF that ever has problems like these, perhaps because it has 
such a nasty desire for true openness.

So, perhaps you would be happier playing in the x.400 email standards arena?

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PRESIDENTS DAY OPEN HOUSE, 2/13
<http://www.brandenburg.com/misc/presday/presday-invite.gif>

Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400  MALAYSIA
Brandenburg Consulting
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464             675 Spruce Dr., Sunnyvale, CA 94086  USA

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Thu, 11 Feb 1999 02:39:39 -0500
Sender: [email-masked]

> > Your argument is irrelevant because it assumes that is some value
> > in the type 2 approach.  There is not.
>   [ ... ]
> > It is pointless to choose a strategy that creates an extra header 
> > which adds no value.
> 
> Do you want another disaster like 8BITMIME? 

No, but I fail to see how having an extra header (or not) would
affect that outcome.  The header doesn't help the user agent
distinguish UTF-8 from other 8bit text because the text in the
message header comes from a variety of sources. The header
doesn't help the MTA because the MTA can just as easily scan
for UTF8 as it can scan for the new header.

We will almost certainly need a UTF8 SMTP option anyway, because
the vast majority of deployed MTAs cannot handle 8bit headers,
some of them failing miserably.  I don't like this conclusion 
because adding complexity to MTAs makes them less reliable.  
I would like to find a better way, but adding another field
to the message header doesn't solve the problem of MTAs that
break when you feed them 8bit headers.

> If not, you should be more
> careful in your cost-benefit analyses.
> 
> Reader strategy 1 interferes with many existing messages. Reader
> strategy 2 does not. 

I disagree.  Strategy 1 does not "interfere" with the operation
of anything, though in rare cases existing text may be mis-displayed.  

Such cases are very rare with 8859/1 text, because UTF-8 characters
for values greater than 127 are represented by sequences of from
2 to 6 octets, each of which has the most significant bit set.
All but the first octet must be in the range 128-191.
In 8859/1 these correspond mostly to special characters and
upper case vowels with accents or diacritical marks.  In all of 
the languges that I know of that use the Latin alphabet, a string 
of more than two or three characters which consists entirely of 
special characters and upper case vowels, is very unlikely to 
occur in practice in the human-readable portion of a message header.
Nobody has a name which contains 1/2 or the copyright symbol, and 
very few words in the Subject field can be spelled entirely with upper
case accented letters.

Text in other 8-bit character sets, notably those with a Latin alphabet 
in the 0-127 range and a non-Latin alphabet in the 128-255 range, is 
somewhat more likely to produce a valid UTF-8 string.  But even then 
this is not terribly likely, because the number of octets in a character
is encoded in the upper bits of the first octet, and all subsequent 
octets in that character must be in the range 128-191.  And I am
of the impression that the largest portion of the installed base
which uses 8bit characters in message headers, uses 8859/1.

> You're advocating another strategy. Your strategy entails certain risks
> that aren't present in strategy 2. 

Yes, but strategy 2 also entails certain risks.  At best, adding
an extra header will tell a user agent that a particular sequence
of 8-bit characters in the header *might* be UTF-8; in general, 
it cannot guarantee that it *is* UTF-8.  Trusting the header alone 
is less reliable than trusting just the validity of the UTF-8 sequence.
Trusting both the header and the sequence validity (which you would
do anyway, since how do you display an invalid UTF-8 sequence?)
doesn't improve the reliability much, and the additional reliability 
comes at the cost of having to scan the entire message header before 
displaying anything.   I doubt sincerely that many implementors would
bother doing this.

> It's _possible_ that the resulting
> damage is so small that it is outweighed by the tiny costs of an extra
> header field; but the necessary studies have not yet been performed.

I am of the understanding from I18N experts that this feature was a 
design goal of UTF-8, and that the "necessary studies" have indeed 
been performed.  I haven't tried to look up those studies myself,
because I trust their judgement and because my back-of-the-envelope 
analysis seems sufficient.

Keith

Date: 11 Feb 1999 07:24:41 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

I've posted a missing-deliberations notice on DRUMS. Anybody who has
spotted Dave Crocker's lost deliberations should let me know.

Dave Crocker writes:
> ad hominem

You keep using that word. I do not think that word means what you think
it means.

An ad-hominem argument ignores the issues in favor of a personal attack
against the opponent. For example, when Johnny Eriksson responded to a
new SMTP client feature report by saying ``Idiot,'' that was an
ad-hominem argument.

> Note, instead, that you are the only person submitting any messages in 
> support of your current position.

Perhaps that's because people are sick of spending huge amounts of time
writing comments that are ignored by the DRUMS management.

822bis and 821bis, in their current form, don't have a chance in hell of
competing with my web pages. Why not ignore them and let them die of
neglect? Because I still think it's possible to turn them into something
that would actually be useful for future implementors.

> I believe that an original author of the removed construct, the chair of 
> the current working group, and both area directors are of the view that 
> there is strong working group consensus on this point.

Then how come you're having so much trouble scraping up any evidence of
that consensus? I accept that _you_ don't want the tokenizer any more,
but where are all the other people who agree with you?

[resounding silence]

You know, legitimate standards organizations don't have this problem.
The procedures are designed so that it isn't possible to (mistakenly or
maliciously) misrepresent the consensus of the group.

---Dan

Date: 11 Feb 1999 06:29:42 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

Keith Moore writes:
> Your argument is irrelevant because it assumes that is some value
> in the type 2 approach.  There is not.
  [ ... ]
> It is pointless to choose a strategy that creates an extra header 
> which adds no value.

Do you want another disaster like 8BITMIME? If not, you should be more
careful in your cost-benefit analyses.

Reader strategy 1 interferes with many existing messages. Reader
strategy 2 does not. Many implementors prefer strategy 2 over strategy 1
for exactly this reason.

You're advocating another strategy. Your strategy entails certain risks
that aren't present in strategy 2. It's _possible_ that the resulting
damage is so small that it is outweighed by the tiny costs of an extra
header field; but the necessary studies have not yet been performed.

Kletnieks gave an example of a situation that's difficult to handle with
strategy 2. But the same situation is just as difficult to handle with
strategy 1, and with your strategy.

---Dan

Date: Thu, 11 Feb 1999 11:26:08 +0800
To: "D. J. Bernstein" <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: Why the 822bis grammar is so painful
Cc: [email-masked]
Sender: [email-masked]

At 11:50 PM 2/10/99 +0000, D. J. Bernstein wrote:
>If Pete Resnick had the approval of the working group, why did he never
>say so? He spent lots of time defending _his_ removal of the tokenizer.

Dan,

I believe that an original author of the removed construct, the chair of 
the current working group, and both area directors are of the view that 
there is strong working group consensus on this point. Yes, I now you keep 
claiming otherwise, but doesn't that seem a tad... well, counterproductive?

Once again, I join the others who urge you to move past ad hominem issues, 
like your above text.

Note, instead, that you are the only person submitting any messages in 
support of your current position.

Whatever you might argue about the past, where is CURRENT support for the 
objection you are raising?

Absent other such submissions, I suggest this thread be allowed to die it's 
natural death.

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PRESIDENTS DAY OPEN HOUSE, 2/13
<http://www.brandenburg.com/misc/presday/presday-invite.gif>

Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400  MALAYSIA
Brandenburg Consulting
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464             675 Spruce Dr., Sunnyvale, CA 94086  USA

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Wed, 10 Feb 1999 22:23:21 -0500
Sender: [email-masked]

> This message is about possible future strategies for _writers_:
> 
>    (A) Translate 8-bit characters from the local character set to
>        unencoded UTF-8 in outgoing messages.
> 
>    (B) Translate 8-bit characters from the local character set to
>        unencoded UTF-8 in outgoing messages, and include the following
>        special header field: ...
> 
> These two strategies have exactly the same effect with type-1 readers.
> The second strategy works with type-2 readers; the first does not.

Your argument is irrelevant because it assumes that is some value
in the type 2 approach.  There is not.

> > Some of those names may be in 2047 format, 
> > others in raw 8859/*, and others in UTF-8.  A single header field, 
> > particularly one which is not supported by everyone's user agent,
> > cannot handle all of those cases.
> 
> Logic error. You face exactly the same problem without the header field.
> This is orthogonal to the choice of strategy.

It is pointless to choose a strategy that creates an extra header 
which adds no value.

Keith

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: Why the 822bis grammar is so painful 
Date: Wed, 10 Feb 1999 22:20:28 -0500
Sender: [email-masked]

> Keith Moore writes:
> > Maybe it's not there, because the discussion happened in person.  
> 
> When did it happen, Keith? Which IETF meeting? 

I don't remember which meeting.  There have been quite a few of them, you know.

> Where did it appear in
> the meeting minutes that were sent to the DRUMS mailing list?

Perhaps it didn't appear.   Maybe the minute taker didn't think it 
was relevant.  It doesn't matter.  The only thing that matters at
present is whether the group has rough consensus on the current approach.

> If Pete Resnick had the approval of the working group, why did he never
> say so? He spent lots of time defending _his_ removal of the tokenizer.

Life is too short for me to try to talk you out of your delusions.

Keith

Date: 11 Feb 1999 02:49:05 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

> > Which approach, 1 or 2, causes more brain damage if sent

1 and 2 aren't ``sent.'' They are current strategies for _readers_.

This message is about possible future strategies for _writers_:

   (A) Translate 8-bit characters from the local character set to
       unencoded UTF-8 in outgoing messages.

   (B) Translate 8-bit characters from the local character set to
       unencoded UTF-8 in outgoing messages, and include the following
       special header field: ...

These two strategies have exactly the same effect with type-1 readers.
The second strategy works with type-2 readers; the first does not.

> Some of those names may be in 2047 format, 
> others in raw 8859/*, and others in UTF-8.  A single header field, 
> particularly one which is not supported by everyone's user agent,
> cannot handle all of those cases.

Logic error. You face exactly the same problem without the header field.
This is orthogonal to the choice of strategy.

---Dan

Date: 10 Feb 1999 23:50:16 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

Pete Resnick, 9 December 1996: ``I should mention that noone brought
this up during the WG session... I didn't remove the entire idea of
tokenizing the input...''

Keith Moore writes:
> Maybe it's not there, because the discussion happened in person.  

When did it happen, Keith? Which IETF meeting? Where did it appear in
the meeting minutes that were sent to the DRUMS mailing list?

If Pete Resnick had the approval of the working group, why did he never
say so? He spent lots of time defending _his_ removal of the tokenizer.

---Dan

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: Why the 822bis grammar is so painful 
Date: Wed, 10 Feb 1999 17:26:15 -0500
Sender: [email-masked]

> > The DRUMS working group deliberated on this point quite carefully -- 
> 
> When? Where in the DRUMS archives can we find the public record of these
> deliberations?

Maybe it's not there, because the discussion happened in person.  
It doesn't matter as long as there was rough consensus, and enough
people were there to support that consensus.

Keith

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Wed, 10 Feb 1999 17:22:59 -0500
Sender: [email-masked]

> That's all fine and good. Unfortunately, what I thought Keith was
> talking about was which one causes more problems in the *boundary
> cases*, such as:
> 
> A) Which approach, 1 or 2, causes more brain damage if sent to an MUA
> that has *NOT* been upgraded yet?
> 
> B) Which approach, 1 or 2, deals with weirdness such as trying to do
> an UTF-8 style reply to an unflagged ISO8859-23 message?

This is close.  The problem with using an extra header is that for 
a variety of reasons, different parts of the message header are
generated by different entities.  Replies are a good example -
consider a message thread which has had a number of people reply
to it, and has a long CC list.  Each person's name in that CC 
list may have been copied from a From header supplied by a 
different user agent.  Some of those names may be in 2047 format, 
others in raw 8859/*, and others in UTF-8.  A single header field, 
particularly one which is not supported by everyone's user agent,
cannot handle all of those cases.

An effective strategy for displaying headers might be:

- if it's in 2047 format, decode and display per 2047

- if a phrase, *text, comment, or quoted-string is a valid UTF-8 
  string, display as UTF-8 

  (the first byte of each UTF-8 character has the length of the
   character encoded in it, and each subsequent byte within that
   character has certain bits set, so it's fairly unlikely that
   something that looks like a valid UTF-8 string is actually a 
   string from some other charset)

- otherwise, display in the recipient's native or default character set

You will (almost certainly) still need a way to negotiate in
SMTP whether the next MTA can deal with 8bit UTF-8 headers, 
and to downgrade to 2047 format if this is not possible.
But an extra header field doesn't help you there - by the time 
you scan the message header looking for the extra field, you can
as easily scan the message header looking for 8bit characters.

Keith

Date: 10 Feb 1999 18:15:30 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

Dave Crocker writes:
> On the matter of removing the tokenizer approach, there was VERY strong 
> working group consensus.

That isn't what Myers said in March 1996. Nor is it what Resnick said in
December 1996.

> The DRUMS working group deliberated on this point quite carefully -- 

When? Where in the DRUMS archives can we find the public record of these
deliberations?

The December 1996 discussion involved precisely six people: three who
strongly objected to Resnick's removal of the tokenizer; two who merely
said that they wanted a formal grammar; and Resnick.

---Dan

To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
From: [email-masked]
Date: Wed, 10 Feb 1999 10:55:56 -0500
Sender: [email-masked]

--==_Exmh_1258636672P
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

On 10 Feb 1999 00:19:05 GMT, you said:
> I mentioned two strategies for UTF-8 header field display in message
> readers:
> =

>    (1) Display every 8-bit header field as UTF-8.
> =

>    (2) Display 8-bit header fields as UTF-8 _if_ the message contains
>        the following special header field: ...
> =

> I said that many implementors would be much happier doing #2 than #1,
> because #1 creates new problems for current users and #2 doesn't.
> =

> ``Wrong,'' Keith said.
> =

> So I've checked with some implementors. The ones who've responded so fa=
r
> are happier doing #2 than #1. Evidently I was right and Keith was wrong=
=2E

Remember that in case 1 *or* 2, if the implementor does what we decide
to do, the result is users that have an MUA that supports whatever it
is we decided on.  Now, the implementors say they would rather
implement 2.

That's all fine and good. Unfortunately, what I thought Keith was
talking about was which one causes more problems in the *boundary
cases*, such as:

A) Which approach, 1 or 2, causes more brain damage if sent to an MUA
that has *NOT* been upgraded yet?

B) Which approach, 1 or 2, deals with weirdness such as trying to do
an UTF-8 style reply to an unflagged ISO8859-23 message?

Remember, there's probably enough manure-ware out there sending untagged
8-bit headers already that you need to worry about this.  A quick test
of the last 1,500 or so pieces of mail I received, 3 had characters that
failed an isascii() test.  Amazingly enough, one of them contained these
headers:

Content-Type: text/plain; charset=3DISO-8859-2
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by netspace.org id OA=
A25799
From: Piotr Strzy=BFewski <[email-masked]>

This one managed to go through 3 sendmails without anything noticing the
lack of RFC2184 quoting on the character following the 'y' in the From:
which originally showed up as a raw unquoted x'BF'.

Yowza.  And no, my MUA (exmh) didn't display it right, because it
looks for 2184 quoting for cues on non-ASCII.

-- =

				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_1258636672P
Content-Type: application/pgp-signature

--==_Exmh_1258636672P--

Date: Wed, 10 Feb 1999 22:26:32 +0800
To: "D. J. Bernstein" <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: Why the 822bis grammar is so painful
Cc: [email-masked]
Sender: [email-masked]

At 01:54 AM 2/7/99 +0000, D. J. Bernstein wrote:
>Pete Resnick, over the objections of several implementors on DRUMS,
>threw away the RFC 822 tokenizer. He wrote a new ABNF grammar that

Just to clarify this particular point:  It is not uncommon to have 
objections from one or more participants.  So the fact of such dissension 
is not automatically significant.

What matters is rough consensus, meaning a sufficient base of support for a 
decision.

On the matter of removing the tokenizer approach, there was VERY strong 
working group consensus.

The tokenizer was a nice idea for RFC733, attempting to provide a nice, 
academically clean model.  I thoroughly enjoyed the time I spent 
researching the topic while we were writing 733.

But it turned out not to work very well, because the mail header 
environment has too much context sensitivity.  All that academic purity 
simply added confusion and, therefore, errors.

The DRUMS working group deliberated on this point quite carefully -- 
considering both retention and removal -- and the agreement to do the 
removal was quite strong.

I'll also add that it is not only counter-productive to put the issue in ad 
hominem terms, it is just plain wrong.

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PRESIDENTS DAY OPEN HOUSE, 2/13
<http://www.brandenburg.com/misc/presday/presday-invite.gif>

Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400  MALAYSIA
Brandenburg Consulting
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464             675 Spruce Dr., Sunnyvale, CA 94086  USA

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Wed, 10 Feb 1999 08:31:54 -0500
Sender: [email-masked]

> OK, so what would that look like? Presumably it would have to include
> downgrading of some sort.

one thing at a time.  first, you need a good spec for the format,
probably along with some strong discouragement to ship
UAs that generate raw UTF-8 by default 
(but support for reading it might not be such a bad idea)

the downgrading document can follow  - actually it should specify
translation in both directions.

Keith

To: [email-masked]
Xref: clerew local.mime:57
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Why the 822bis grammar is so painful
Date: Wed, 10 Feb 1999 10:26:07 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Chris Newman <[email-masked]> writes:

>I actually figured out the 822bis grammar changes necessary to resolve the
>one-atom-vs-two parsing distinction for fun.  But the problem is so minor
>that fixing it isn't worth the complexity it would add to the grammar,
>IMHO.  In the bigger scheme of things, the current grammar sufficies to
>determine if headers are valid or invalid; and that's what's important. 

Is it that complex? Surely the particular case I highlighted only requires:

phrase = word *( CFWS word) / obs-phrase

Of course, there may be other cases, and I think the whole of that syntax
is badly designed and liable to be misunderstood, but I think all you can
do at this point is fix known errors while there is still time.

The real problems are going to come when other standards (Mime extensions,
UTF-8, etc) start trying to build on the DRUMS basis.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:56
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Wed, 10 Feb 1999 10:17:39 GMT
Lines: 17
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>We probably could do some sort of public pronouncement, but I think that
>such a pronouncement should probably be an Experimental protocol spec
>for UTF-8 in email headers, not just a "we're going to migrate to 
>UTF-8 someday" statement.  If we do the latter we risk people going
>off and implementing UTF-8 in an ad-hoc fashion, and even greater
>divergence of implementations than we have now.

OK, so what would that look like? Presumably it would have to include
downgrading of some sort.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Tue, 09 Feb 1999 23:48:30 -0500
Sender: [email-masked]

> I mentioned two strategies for UTF-8 header field display in message
> readers:
> 
>    (1) Display every 8-bit header field as UTF-8.
> 
>    (2) Display 8-bit header fields as UTF-8 _if_ the message contains
>        the following special header field: ...
> 
> I said that many implementors would be much happier doing #2 than #1,
> because #1 creates new problems for current users and #2 doesn't.
> 
> ``Wrong,'' Keith said.

I don't care what your implementors think.  Your approach is broken.

Keith

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: Why the 822bis grammar is so painful 
Date: Tue, 09 Feb 1999 23:47:00 -0500
Sender: [email-masked]

> My conclusion, after several case studies, is that most message-format
> interoperability problems are caused by implementors who don't know what
> RFC 822 says.

I mostly agree.

Keith

Date: 10 Feb 1999 00:19:05 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

I mentioned two strategies for UTF-8 header field display in message
readers:

   (1) Display every 8-bit header field as UTF-8.

   (2) Display 8-bit header fields as UTF-8 _if_ the message contains
       the following special header field: ...

I said that many implementors would be much happier doing #2 than #1,
because #1 creates new problems for current users and #2 doesn't.

``Wrong,'' Keith said.

So I've checked with some implementors. The ones who've responded so far
are happier doing #2 than #1. Evidently I was right and Keith was wrong.

Perhaps what Keith _meant_ to say was that he thinks another strategy
(such as the one we used in FTP) would be even better than #2. But that
doesn't make #1 better than #2.

---Dan

Date: 9 Feb 1999 23:45:30 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

I wrote my own message-header-format reference manual for the benefit of
future implementors, after I realized that 822bis had a different goal:

   http://pobox.com/~djb/proto/immhf.html

The feedback from other implementors has been overwhelmingly positive.
So I don't think that Newman's pessimism is justified.

Chris Newman writes:
> The parsing details of RFC 822 were widely misimplemented in numerous ways
> in the field.  The only reasonable conclusion was that the grammar model
> in 822 was inadequate for real world needs.

No. Your conclusion doesn't explain the fact that most broken parsers
are unable to handle the _examples_ in RFC 822.

My conclusion, after several case studies, is that most message-format
interoperability problems are caused by implementors who don't know what
RFC 822 says. (Exception: Mark Crispin has read RFC 822 but insists that
DCA prohibited all the formats that Pine fails to recognize.)

Will 822bis be more inviting to implementors than 822? I doubt it. There
aren't enough examples; reading the grammar is an exercise in masochism;
the organization doesn't match the structure of common implementations.

> the current grammar sufficies to determine if headers are valid or invalid;

No, it doesn't---the 822bis grammar still has bugs!

In December 1996, after Pete Resnick said that the point of his grammar
was automated verification, I suggested using C instead of ABNF. C code
can actually be tested and used. How many people have ABNF compilers?

> I actually figured out the 822bis grammar changes necessary to resolve the
> one-atom-vs-two parsing distinction for fun.

Great! Now show us the grammar changes necessary to prohibit fields such
as "To: anything I want".

---Dan

Date: Tue, 09 Feb 1999 11:08:30 -0800 (PST)
From: Chris Newman <[email-masked]>
Subject: Re: Why the 822bis grammar is so painful
Originator-info: login-id=chris; server=THOR.INNOSOFT.COM
To: [email-masked]
Sender: [email-masked]

Discussions about DRUMS topics on this list will not be taken into account
within the DRUMS WG.

The 822bis document passed WG last call -- meaning it had WG rough
concensus.  Unless the IESG determines something is unacceptable during
IETF last call and sends it back to the WG for revision, the subject is
closed in DRUMS. 

The parsing details of RFC 822 were widely misimplemented in numerous ways
in the field.  The only reasonable conclusion was that the grammar model
in 822 was inadequate for real world needs.

Personally, I think the 822bis grammar is a bit painful, but I believe it
is less painful than any alternative I saw and will result in fewer errors
in the field than the original 822 grammar model did.  While people with
academic backgrounds in parsing, such as myself, might find the split
tokenize-parse model attractive; a number of key software developers in
the email world lack this background and need something more
straightforward.

If someone had politely suggested a concrete and reasonable alternative in
1996/1997, perhaps we could have had something better.  But most of the
objections were of the form "I don't like this" and lacked a concrete
alternative.  Such comments have little value when trying to fix problems.

I actually figured out the 822bis grammar changes necessary to resolve the
one-atom-vs-two parsing distinction for fun.  But the problem is so minor
that fixing it isn't worth the complexity it would add to the grammar,
IMHO.  In the bigger scheme of things, the current grammar sufficies to
determine if headers are valid or invalid; and that's what's important. 

		- Chris

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Tue, 09 Feb 1999 08:31:31 -0500
Sender: [email-masked]

> I think if raw UTF-8 is going to be the eventual interpretation of headers
> with 8th bits set in them (as opposed to treating it as iso-8859-1, which
> people can get away with now) then there needs to be something said sooner
> rather than later just in order to take possession of the high ground
> before anyone else does. A few newsreaders that took the UTF-8
> interpretation might work wonders just by setting the right precedents.

Yes, I agree with that.

> But total lack of public pronouncement on the issue will just open the way
> for iso-8859-1 to win by default.
> 
> Yes, I know there is presently no machinery for making such public
> pronouncements ...

We probably could do some sort of public pronouncement, but I think that
such a pronouncement should probably be an Experimental protocol spec
for UTF-8 in email headers, not just a "we're going to migrate to 
UTF-8 someday" statement.  If we do the latter we risk people going
off and implementing UTF-8 in an ad-hoc fashion, and even greater
divergence of implementations than we have now.

Keith

To: [email-masked]
Xref: clerew local.mime:53
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Tue, 9 Feb 1999 10:50:32 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>Yes, but we may be better off to have less variation in practice.
>I'd rather have "just" some combination of encoded-words and raw 
>8859/1 to deal with, than to prematurely add UTF-8 to the mix.

I think if raw UTF-8 is going to be the eventual interpretation of headers
with 8th bits set in them (as opposed to treating it as iso-8859-1, which
people can get away with now) then there needs to be something said sooner
rather than later just in order to take possession of the high ground
before anyone else does. A few newsreaders that took the UTF-8
interpretation might work wonders just by setting the right precedents.

But total lack of public pronouncement on the issue will just open the way
for iso-8859-1 to win by default.

Yes, I know there is presently no machinery for making such public
pronouncements ...

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Mon, 08 Feb 1999 18:31:18 -0500
Sender: [email-masked]

> >> sendmail, starting in version 8.8.0, simply drops bytes 128-159 from all
> >> incoming header fields.
> >> 
> >> The underlying problem is that sendmail's code is far from 8-bit-clean.
> >> sendmail's entire rewriting mechanism works with a string format that
> >> assigns special meanings to bytes 129, 130, 133, etc. It can't even deal
> >> with an 8-bit name in the From line.
> 
> >This could actually turn out to be a Good Thing in that it will
> >slow down those who want to propagate 8-bit headers until they
> >are properly defined and their implications are understood. 
> 
> Actually, it will be a Bad Thing, because it will encourage people to
> continue the illegal use of ISO-8859-1, which does not use those
> characters and who therefore "get away" with it presently.

Yes, but we may be better off to have less variation in practice.
I'd rather have "just" some combination of encoded-words and raw 
8859/1 to deal with, than to prematurely add UTF-8 to the mix.

Keith

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Mon, 08 Feb 1999 18:26:36 -0500
Sender: [email-masked]

> Keith Moore writes:
> > But if users rely on nonstandard features, they deserve whatever they get.  
> 
> Your standards don't provide the features that the users want. You lose.

There's more than one way to lose.  You can lose because you invest
in nonstandard solutions and/or solutions that don't scale.  
You can also lose because the standards don't meet your needs.

> The current reality is that many users rely on unencoded ISO-8859-1. If
> you ignore them, they will ignore you---and they're the ones in charge.

Users will use whatever works best for them, as soon as they find it.
Given competition between a standard that works and a nonstandard that
doesn't work, the standard will probably win out eventually.

> > Wrong.
> 
> I can't imagine what you could possibly be disagreeing with. The special
> header field doesn't show up in any current messages.

Your lack of imagination is not my concern.

If you want to know why adding a header field doesn't work, go back 
and read the archives of this mailing list from 1991 or so.

Keith

Date: 8 Feb 1999 17:15:26 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

Charles Lindsey writes:
> I am worried about the receivers that don't understand UTF-8, of
> which sendmail seems the chief offender.

Hmmm. I was using ``receiver'' to refer to the final message reader. If
all the message readers understand =?utf-8?= then you can just use that
without running into any trouble with sendmail.

> Sorry, it seems I was wrong, though actually only the first on your list
> seems to be legal UTF-8 according to RFC 2044.

It takes extra work for a UTF-8 reader to exclude the other encodings.
Does anyone have the relevant excerpt from the real UTF-8 definition?

---Dan

To: [email-masked]
Xref: clerew local.mime:43
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Mon, 8 Feb 1999 10:25:51 GMT
Lines: 43
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>sendmail, starting in version 8.8.0, simply drops bytes 128-159 from all
>incoming header fields.

>The underlying problem is that sendmail's code is far from 8-bit-clean.
>sendmail's entire rewriting mechanism works with a string format that
>assigns special meanings to bytes 129, 130, 133, etc. It can't even deal
>with an 8-bit name in the From line.

Yes, that seems real ugly :-( .

>I thought you were worried about receivers that _don't_ understand =??=.
>In this case, downgrading will fail. You have at least some chance of
>success if you go ahead and send 8-bit characters.

Wrong. I am worried about the receivers that don't understand UTF-8, of
which sendmail seems the chief offender. MTAs that _don't_ understand =??=
are no problem (they simply have to pass it through). Even user agents
that don't understand it are not show stoppers, because they just render
it as received - ugly, but nothing breaks.

>> The (slightly) good news is that you can translate iso-8859-1 into UTF-8
>> without producing any of the characters in 128-159.

>Hmmm? Character 193 ("A'"), 11000001 binary, is encoded as one of

>   11000011 10000001
>   11100000 10000011 10000001
>   11110000 10000000 10000011 10000001
>   11111000 10000000 10000000 10000011 10000001
>   11111100 10000000 10000000 10000000 10000011 10000001

Sorry, it seems I was wrong, though actually only the first on your list
seems to be legal UTF-8 according to RFC 2044.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:44
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Mon, 8 Feb 1999 10:29:18 GMT
Lines: 31
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> sendmail, starting in version 8.8.0, simply drops bytes 128-159 from all
>> incoming header fields.
>> 
>> The underlying problem is that sendmail's code is far from 8-bit-clean.
>> sendmail's entire rewriting mechanism works with a string format that
>> assigns special meanings to bytes 129, 130, 133, etc. It can't even deal
>> with an 8-bit name in the From line.

>This could actually turn out to be a Good Thing in that it will
>slow down those who want to propagate 8-bit headers until they
>are properly defined and their implications are understood. 

Actually, it will be a Bad Thing, because it will encourage people to
continue the illegal use of ISO-8859-1, which does not use those
characters and who therefore "get away" with it presently.

>> I'm concerned with the features that users rely on, not just the
>> features guaranteed by the IETF.

>I don't want to gratuitously break things unless they cause harm.  But 
>if users rely on nonstandard features, they deserve whatever they get.

Yes, I agree.
  
-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: 8 Feb 1999 07:12:17 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

Keith Moore writes:
> But if users rely on nonstandard features, they deserve whatever they get.  

Your standards don't provide the features that the users want. You lose.

The current reality is that many users rely on unencoded ISO-8859-1. If
you ignore them, they will ignore you---and they're the ones in charge.

> > MUAs with poor 8-bit support and without any European users can get away
> > with broad rules such as
> >    Interpret all 8-bit characters in the header as UTF-8.
> > But other implementors will be much happier with
> >    Interpret all 8-bit characters in the header as UTF-8 _if_ you see
> >    the following special header field: ...
> > This avoids creating any new problems for current users.
> Wrong.

I can't imagine what you could possibly be disagreeing with. The special
header field doesn't show up in any current messages.

---Dan

Date: Mon, 8 Feb 1999 07:15:10 +0100
To: "D. J. Bernstein" <[email-masked]>, [email-masked]
From: Patrik =?iso-8859-1?Q?F=E4ltstr=F6m?=  <[email-masked]>
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

At 04.24 +0000 1999-02-08, D. J. Bernstein wrote:
> Keith Moore writes:
>> Different people will have different recollections.
>
> I sincerely doubt that. The DRUMS mailing list archive supports every
> detail of my statements. Your claimed consensus is wishful thinking.

Dan, and others.

Speaking as Area Director for Applications Area, I ask you to keep
the discussions focused and constructive on what to do next and not
continue with the tone and personal attacks that I now see. If it is
the case that you want closure on these documents -- and my
recollection is that we all do -- then you have to concentrate on the
documents, and and move forward.

   Patrik

------------------------------------------------------------------
Area Director, Applications Area               Email: [email-masked]
IETF                                             URL: http://paf.se
                                          PGP Key ID: 0xBD236602

   In theory there is no difference between theory and practice,
   but in practice, there is.

Date: 8 Feb 1999 04:24:25 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

Keith Moore writes:
> Different people will have different recollections.

I sincerely doubt that. The DRUMS mailing list archive supports every
detail of my statements. Your claimed consensus is wishful thinking.

---Dan

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: Why the 822bis grammar is so painful 
Date: Sun, 07 Feb 1999 20:19:17 -0500
Sender: [email-masked]

Dan,

Different people will have different recollections.  Also, you weren't
at the face-to-face meetings where other opinions were expressed.

Also, your "bullshit" comments and other personal attacks are not 
constructive.  You express your opinions in such an obnoxious way that 
other people, who agree with some of what you say, don't want to 
publically state this for fear of lending support to your extremism.
In this way your technical contributions are eroded.

Keith

Date: 8 Feb 1999 00:39:06 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Why the 822bis grammar is so painful
Sender: [email-masked]

Keith Moore writes:
> DRUMS decided that 822 was too liberal in allowing white
> space and comments, and that the revised mail specification needed
> to discourage certain such uses, and therefore the grammar needed to be
> able to explicitly specify where white space and comments were legal.

No, Keith, that isn't what happened.

DRUMS mailing list discussion, early 1996: Pine incorrectly truncates
the valid RFC 822 field

   From: joe@heaven. af.mil

after the first dot.

Almost everyone agrees that this is unacceptable---but that 822bis
should recommend, for interoperability, that message generators avoid
inserting whitespace between address tokens.

DRUMS mailing list discussion, August 1996: There's a big difference
between

   (1) a single atom token containing two digits and
   (2) two atom tokens, each containing one digit.

Whitespace is prohibited in #1 but required in #2.

RFC 822 uses ``2DIGIT'' to mean #1, not #2, but if you refuse to look at
anything other than the formal grammar then you can't figure this out.

Four people suggest adding special notations to the grammar to indicate
where free insertion of whitespace is allowed. Two people suggest
sticking to the simple RFC 822 model of a separate tokenizer and parser,
and stating the definition of 2DIGIT within that model.

DRUMS mailing list discussion, October 1996: I give some examples of
RFC 822 syntax features not expressed in the RFC 822 formal grammar. One
example is that atoms extend until the next delimiter. Another example
is that whitespace may be freely inserted between tokens.

``I hope all will be pleased with my solution,'' Resnick says, without
saying what his solution is.

DRUMS mailing list discussion, November 1996: Pete Resnick has just
released his first attempt at an RFC 822 revision, with dozens of
changes not authorized by DRUMS.

In particular, Resnick has thrown away the RFC 822 tokenizer in favor of
explicit insertion of dozens and dozens of spaces, tabs, and comments
throughout the grammar.

Paul Overell, Kai Henningsen, and I object to the destruction of the
tokenizer. Resnick claims that he's trying to support automated testing
of whether a message is syntactically valid; he also claims that English
is error-prone while ABNF is not. Jim Conklin and Chris Newman state
that they agree.

> However, Dan's mistaken to blame Pete for this.  A large number 
> of DRUMS participants - probably a rough consensus - argued for 
> the changes to the grammar notation,

Bullshit.

Pete Resnick explicitly stated that nobody had discussed this with him.
He repeatedly talked about the destruction of the tokenizer as his own
personal decision. When he was explaining ``why I wrote [the grammar]
the way I did'' he never claimed that he had approval from DRUMS.

---Dan

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: Why the 822bis grammar is so painful 
Date: Sun, 07 Feb 1999 01:12:08 -0500
Sender: [email-masked]

> Pete Resnick, over the objections of several implementors on DRUMS,
> threw away the RFC 822 tokenizer. 

For better or worse, the RFC 2234 BNF was designed to be usable in 
other protocol specifications besides those describing email headers. 
Many non-email specifications use 822's ABNF.  Most of those that
do use 822's ABNF have ambiguities or errors, because few of these
protocols are intended to allow the same liberal use of white space 
and comments allowed by RFC 822. 

Indeed, DRUMS decided that 822 was too liberal in allowing white
space and comments, and that the revised mail specification needed
to discourage certain such uses, and therefore the grammar needed to be
able to explicitly specify where white space and comments were legal.
And since 822's "#" notation implicitly allows white space and
comments, that had to go also.

One might say that both 822's ABNF and 2234's ABNF share the 
same deficiency - the failure to separate lexical analysis
from parsing.  (to be fair, 822 does have a prose description
of its lexical analysis step, but most of the specifications 
that borrowed 822's ABNF notation did not describe their own 
lexical analysis).

And while 2234's ABNF has been used successfully to describe
other protocols, I don't like the result for 822bis.  I agree
that it makes the grammar unnecessarily difficult to read
or verify for correctness, and there are bound to be a few
bugs left.  (and I'm not looking forward to doing a detailed 
review for IESG)

However, Dan's mistaken to blame Pete for this.  A large number 
of DRUMS participants - probably a rough consensus - argued for 
the changes to the grammar notation, and to deprecate 822's
liberal acceptance of comments and white space.  This opinion
held despite the objections of some members with more experience 
in language design that this would make the grammar too complicated 
and introduce errors.

Keith

Date: 7 Feb 1999 01:54:30 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Why the 822bis grammar is so painful
Sender: [email-masked]

RFC 822 says that structured field bodies are parsed as follows:

   characters
   (e.g., "6 Feb 1999")

     |
     | tokenize
     V

   spaces, tabs, comments,
   "<", ">", ",", ";", ":", "@", ".", atoms, domain literals, quoted strings
   (e.g., atom 6, space, atom Feb, space, atom 1999)

     |
     | remove spaces, tabs, comments
     V

   "<", ">", ",", ";", ":", "@", ".", atoms, domain literals, quoted strings
   (e.g., atom 6, atom Feb, atom 1999)

     |
     | parse
     V

   higher-level data

It's easy to give precise English descriptions of each of these steps.
See http://pobox.com/~djb/proto/immhf.html.

Pete Resnick, over the objections of several implementors on DRUMS,
threw away the RFC 822 tokenizer. He wrote a new ABNF grammar that
starts from sequences of characters, rather than sequences of tokens.
ABNF is a weak programming language in which simple lexing steps such as

   Read as many characters as possible, stopping before the first ...

and

   Now remove all comments

are a royal pain to handle correctly, so it's hardly a surprise that
Resnick made some big mistakes in his grammar, and that the result is
much more difficult to read than RFC 822.

Charles Lindsey writes:
  [ ``foobar'' being parsed as two atoms ]
> However, I gather Pete Resnick has spotted this discussion and is
> taking it up on the DRUMS list. I hope they fix it.

I raised the same issue on the DRUMS mailing list in 1996.

Resnick was claiming that English was error-prone while ABNF was not:
``Having everything in the grammar leaves no ambiguity, and having them
in the prose is almost guaranteeing it.''

I pointed out that the evidence was against him: ``Really? How come your
grammar allows `To: anything I want'? How come your grammar allows the
string `foo' to be parsed as three atoms?'' Of course, these ambiguities
are extremely difficult to eliminate from the formal grammar.

Resnick's response: ``I don't think it makes a difference, unless you do
something silly like try to feed tokens you've received and parsed into
something that's going to put comments or whitespace between them.''

I said that an incorrect spec was unacceptable, and suggested writing
the grammar in C instead of ABNF. Resnick didn't respond.

---Dan

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Sat, 06 Feb 1999 18:44:21 -0500
Sender: [email-masked]

> Charles Lindsey writes:
> > And does it
> > barf when the nasty characters are in phrases or comments in those
> > headers?
> 
> sendmail, starting in version 8.8.0, simply drops bytes 128-159 from all
> incoming header fields.
> 
> The underlying problem is that sendmail's code is far from 8-bit-clean.
> sendmail's entire rewriting mechanism works with a string format that
> assigns special meanings to bytes 129, 130, 133, etc. It can't even deal
> with an 8-bit name in the From line.

This could actually turn out to be a Good Thing in that it will
slow down those who want to propagate 8-bit headers until they
are properly defined and their implications are understood. 

> > Well any message that currently uses unencoded ISO-8859-1 (except within
> > the protection of a Content-Type specifying that charset) is not
> > conforming already, so we are not necessarily obliged to recognise it
> 
> I'm concerned with the features that users rely on, not just the
> features guaranteed by the IETF.

I don't want to gratuitously break things unless they cause harm.  But 
if users rely on nonstandard features, they deserve whatever they get.  
This is true no matter whether those features are propagated by sendmail,
qmail, or a 300-pound gorilla in Redmond.

> MUAs with poor 8-bit support and without any European users can get away
> with broad rules such as
> 
>    Interpret all 8-bit characters in the header as UTF-8.
> 
> But other implementors will be much happier with
> 
>    Interpret all 8-bit characters in the header as UTF-8 _if_ you see
>    the following special header field: ...
> 
> This avoids creating any new problems for current users.

Wrong.  Go back and do the case analysis.

Keith

Date: 6 Feb 1999 19:33:52 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: The transition to UTF-8 header fields
Sender: [email-masked]

Charles Lindsey writes:
> And does it
> barf when the nasty characters are in phrases or comments in those
> headers?

sendmail, starting in version 8.8.0, simply drops bytes 128-159 from all
incoming header fields.

The underlying problem is that sendmail's code is far from 8-bit-clean.
sendmail's entire rewriting mechanism works with a string format that
assigns special meanings to bytes 129, 130, 133, etc. It can't even deal
with an 8-bit name in the From line.

It might be possible to reversibly encode the unsafe 8-bit characters
when a header is read, and undo the encoding when a header is written,
without extensive changes to the sendmail code. But, no matter how minor
the changes are, it will take a lot of effort to deploy the new code.

On the bright side, there are lots of 8-bit-clean MTAs. Also, sendmail
before 8.8.0 can at least tolerate 128-159 in Subject fields.

> Well any message that currently uses unencoded ISO-8859-1 (except within
> the protection of a Content-Type specifying that charset) is not
> conforming already, so we are not necessarily obliged to recognise it

I'm concerned with the features that users rely on, not just the
features guaranteed by the IETF.

MUAs with poor 8-bit support and without any European users can get away
with broad rules such as

   Interpret all 8-bit characters in the header as UTF-8.

But other implementors will be much happier with

   Interpret all 8-bit characters in the header as UTF-8 _if_ you see
   the following special header field: ...

This avoids creating any new problems for current users.

> Surely, it is
> the good, clean agent that is then responsible to do the downgrading.

If =??= is acceptable to the receiving user, why don't you just send it
in the first place?

I thought you were worried about receivers that _don't_ understand =??=.
In this case, downgrading will fail. You have at least some chance of
success if you go ahead and send 8-bit characters.

Remember that the current situation doesn't work. We have a bunch of
unhappy 8-bit users. How do we make everybody happy?

Widespread support for 8859-1 isn't enough: many users need characters
outside 8859-1. The benefits of UTF-8 are obvious.

The =??= encoding is unsatisfactory, for the reasons you've mentioned.
The benefits of _unencoded_ UTF-8 are obvious.

How do we make unencoded UTF-8 work as quickly as possible? By replacing
the mailers that have trouble with it. Newman's SMTP extension, like
8BITMIME, would make this process much slower and much more expensive.

> The (slightly) good news is that you can translate iso-8859-1 into UTF-8
> without producing any of the characters in 128-159.

Hmmm? Character 193 ("A'"), 11000001 binary, is encoded as one of

   11000011 10000001
   11100000 10000011 10000001
   11110000 10000000 10000011 10000001
   11111000 10000000 10000000 10000011 10000001
   11111100 10000000 10000000 10000000 10000011 10000001

all of which use byte 129. Is there a repetition of 8859-1 somewhere
else in the 10646 character set?

---Dan

From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked]
Subject: Re: The transition to UTF-8 header fields 
Date: Fri, 05 Feb 1999 18:23:26 -0500
Sender: [email-masked]

> The short-term goal is to allow messages with unencoded UTF-8 in the
> header. This takes three steps:
> 
>    (1) Mailers with restrictions on 8-bit characters, notably sendmail,
>        have to be fixed.
> 
>    (2) Message readers have to learn how to convert common UTF-8
>        characters to the local character set.

Or more to the point, they have to learn how to display them.
it's not at all unusual these days for a GUI to be able to
display characters in other than the default local character set.

>    (3) Message readers have to turn on conversion for some messages.
>        Many existing messages use unencoded ISO-8859-1, so the new
>        messages have to be marked, for example with a new header field.

It is nearly always possible to distinguish a UTF-8 string of more
than a few characters' length, from a string in some other character 
set.  So a reasonable strategy would be to display valid UTF-8 
strings as UTF-8, and display other unencoded 8-bit character 
strings using a local default character set.

> The long-term goal is to eliminate the implementation burden of multiple
> character sets and =??=. This takes one extra step:
> 
>    (4) Mail writers have to convert all outgoing messages from the local
>        character set to unencoded UTF-8.
> 
> Eventually all character-set markers can be removed.

This is *very* long-term.  For several years, mail readers would
still need to be able to handle 2047.  At some point after enough
MTAs became 8-bit header-friendly and enough MUAs became UTF-8
header-friendly, sending UTF-8 in headers would be considered an 
acceptable risk for the sender.  At some much later point, not 
including support for 2047 in your user agent would be considered 
an acceptable risk for a MUA vendor.

> > * Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
> >   both top level headers and nested message/rfc822 headers.
> 
> That doesn't survive a cost-benefit analysis.
> 
> Your conversion is safe in a fantasy world where all message readers
> magically understand =??=. But in that fantasy world you can use
> _encoded_ UTF-8. Your SMTP extension is completely unnecessary.

I would say it differently - having this in SMTP requires the SMTP
server to know what kind of UA it is delivering a message to, so 
that it knows whether to downgrade or not.  For many environments
this just doesn't work, and for the environments where it does work,
it's easy enough to just detect the presence of UTF-8 in the message 
header.  There's no need for an SMTP extension.
 
> The real world is more complicated. There's no safe way to send UTF-8.
> Many readers don't handle =??=. Many readers don't _want_ to handle
> =??=. They want unencoded 8-bit characters. But sendmail can't handle
> characters between 128 and 159 in address fields.

Arbitrary 8-bit characters in address fields are a completely separate 
issue, and probably won't happen within our lifetimes.  It is more 
likely that the world will standardize on English (not that I consider
this likely).

> Fixing sendmail isn't easy, but it's the fastest way to eliminate this
> problem. 

This is one case where sendmail is not broken.  No change to any single 
MTA will have a great effect on transition times.

Keith

To: [email-masked]
Xref: clerew local.mime:37
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: The transition to UTF-8 header fields
Date: Fri, 5 Feb 1999 15:39:06 GMT
Lines: 64
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>The short-term goal is to allow messages with unencoded UTF-8 in the
>header. This takes three steps:

>   (2) Message readers have to learn how to convert common UTF-8
>       characters to the local character set.

They also need to know of some agreed escape notation to render any UTF-8
character they are unable to display using the local facilities.

They also have to do the same thing with headers in Mime multiparts and
message/rfc822 (and actually, that is easier because sendmail is not a
problem there AFAIK).

>   (3) Message readers have to turn on conversion for some messages.
>       Many existing messages use unencoded ISO-8859-1, so the new
>       messages have to be marked, for example with a new header field.

Well any message that currently uses unencoded ISO-8859-1 (except within
the protection of a Content-Type specifying that charset) is not
conforming already, so we are not necessarily obliged to recognise it (it
should have been done as an encoded-word, though I agree it often isn't).

>Chris Newman writes:
>> * Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
>>   both top level headers and nested message/rfc822 headers.

>The real world is more complicated. There's no safe way to send UTF-8.
>Many readers don't handle =??=. Many readers don't _want_ to handle
>=??=. They want unencoded 8-bit characters. But sendmail can't handle
>characters between 128 and 159 in address fields.

Could you be more specific about that? Which headers are affected? To:
From: Sender: Reply-To: Cc: ... ? I know that sendmail delights in
rewriting such headers under control of its .cf file :-((( ). And does it
barf when the nasty characters are in phrases or comments in those
headers?

>Fixing sendmail isn't easy, but it's the fastest way to eliminate this
>problem. What you're talking about is fixing sendmail _and_ adding a
>painful conversion procedure to sendmail _and_ adding the same painful
>conversion procedure to dozens of 8-bit-clean MTAs; that's much more
>expensive, and will add years to the transition.

So you are saying that if you have to fix sendmail (and other agents) to
do downgrading then you might as well fix it to pass the UTF-8 characters
cleanly in the first place? Sounds a plausible argument, but what about
the case where a good, clean, upgraded agent is negotiating using EHLO
with some ancient agent that does not support UTF8HEADER?  Surely, it is
the good, clean agent that is then responsible to do the downgrading.
Trouble is, those ancient agents are going to be around for a long time to
come, and the decent agents are still going to have to speak to them.

The (slightly) good news is that you can translate iso-8859-1 into UTF-8
without producing any of the characters in 128-159. Does that help at all?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:35
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Fri, 5 Feb 1999 14:30:02 GMT
Lines: 114
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>I'm not sure what you could mean by "undoing the first mistake".  The email 
>world is still not ready for UTF-8 in headers, though if we manage the 
>transition carefully it might be ready in a few years.

Well that is where I am hearing conflicting stories. My immediate problem
is that news is going to go to UTF-8 long before it becomes common in mail.
So we need some means to tunnel news articles safely through existing mail
systems.

Now I have proposed two Mime mechanisms for doing that, one of which has
been described as "ugly", and the other of which is agreed to be
"exceedingly ugly". Instead, I have been told that the "proper" way to do
this would be to assume that UTF-8 would become legal in mail, and that we
should instead use the downgrading mechanism that would be associated with
that. (Actually, allowing it in multipart headers rather that full mail
headers would go a long way towards fixing our problem).

And, moreover again, Ned stated on this very list that such could easily
be worked out within six months (and, one would think, he should know). On
that sort of time scale, news could probably wait, but if this problem is
not going to be fixed for "a few years", then news is going to have
to come up with an interim solution _now_.

> ...   And it would seem 
>silly to replace encoded-words with a different and slightly less broken 
>mechanism for encoding 8bits in ASCII (but which might work for parameter 
>values also) when we already have an investment in encoded-words and 
>we'll want to go to UTF-8 in a few years anyway.  It also seems silly
>to declare "it's okay to use encoded-words in quoted strings" when 
>this will cause a fair amount of disruption.

Yes, the purpose of my message was to try and find how big that
"disruption" might actually be. You already said yourself that many
(most?) software already supported it, though not legally so.

>> Right! Let us suppose, just be way of Thinking-Out-Loud, that some
>> extension of RFC2047 (or perhaps some 2047bis) were made that allowed
>> encoded-words within a quoted-string (with perhaps a prohibition remaining
>> within addr-specs and msg-ids). What calamities would ensue?

>People would start using them in addr-specs, expecting them to be
>decoded before display.

No, I carefully said they must be excluded in addr-specs (one day, UTF-8
will be allowed in domain-names, and that transition will lead to all
sorts of interesting happenings - but that is certainly NOT part of
anything under consideration now). I don't really care how they are
displayed, so long as they go to the right place. Obviously, the display
SHOULD reflect the actual address to which the mail goes. And actually, I
don't see why people would try to use them in addr-specs since valid
addresses do not contain 8bit characters anywhere that I know of.

> ...   Some gateways would translate them to 
>raw characters, and fail to translate them back, which would make 
>replies fail, as well as causing failures with other tools that 
>recognize addresses.

No, because they would only be allowed in the phrase part of an address,
and that part does not affect the routeing. Some mailers that present mail
to you and include the 'real name' in the menu might get it wrong, but I
suspect many (most?) existing mailers might suddenly start getting it
right.

> ...  Some gateways would translate them to raw 
>characters, and translate them back in a way that didn't match the 
>original name, and cause similar (but more subtle) failures.

Yes, one of my concerns with RFC2047 is that there is no canonical way to
encode a string. Software that cares about such things ought to work with
the unencoded form (one of the other things I am looking at at the moment
is digital signing of headers, and that is one of the concerns to be
addressed there).

>Systems that use name=  or content-disposition filenames would
>behave inconsistently - some would store the file under a bizarre
>name, while others would store the file with the decoded name.

It does not matter particularly if a file gets stored under the bizarre
name on one machine and under the decoded name on another. The important
thing is that it gets stored someplace. But I suspect that systems that do
not yet implement RFC2231 are facing this problem already.

>All of these things happen right now to some degree due to broken
>implementations.  But by changing direction, we would make the
>overall behavior worse.

Or better? 

>I don't see the point in changing encoded-words at this point, especially
>not if the purpose is to make eventual downgrading to UTF-8 easier.  
>Yes, the downgrading rules will be somewhat complex, with lots of special 
>cases.

I think the point to do it would be when the UTF-8 stuff comes along, in
one grand RFC that brought all of RFC2047, RFC 2232 and others together in
one document.

The real question is when that is going to be. Ned says six months. I was
dubious when he said it, and am even more dubious now. But if not then,
then what is news supposed to do in the meantime?

But either way, it is not unreasonable to look at some possibilities that
might or might not work, even if they don't get implemented for some
longer time.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:36
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Fri, 5 Feb 1999 14:36:07 GMT
Lines: 23
Sender: [email-masked]

In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:

>Charles Lindsey writes:
>> I was working from the syntax in DRUMS

>Not a good idea. The ABNF+822bis syntax is an untested computer program
>in a weak, error-prone programming language.

Well, if DRUMS is going to supersede RFC822 as the official standard in
the not-too-distant future, then it had better become "a good idea"
pronto. However, I gather Pete Resnick has spotted this discussion and is
taking it up on the DRUMS list. I hope they fix it.

Yes, writing good syntax in BNF-like languages is hard work, as those of
us who have worked in program language design well know. Perhaps that is
where I developed a good nose for smelling these things out :-) .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: 5 Feb 1999 01:50:07 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: The transition to UTF-8 header fields
Sender: [email-masked]

The short-term goal is to allow messages with unencoded UTF-8 in the
header. This takes three steps:

   (1) Mailers with restrictions on 8-bit characters, notably sendmail,
       have to be fixed.

   (2) Message readers have to learn how to convert common UTF-8
       characters to the local character set.

   (3) Message readers have to turn on conversion for some messages.
       Many existing messages use unencoded ISO-8859-1, so the new
       messages have to be marked, for example with a new header field.

The long-term goal is to eliminate the implementation burden of multiple
character sets and =??=. This takes one extra step:

   (4) Mail writers have to convert all outgoing messages from the local
       character set to unencoded UTF-8.

Eventually all character-set markers can be removed.

Chris Newman writes:
> * Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
>   both top level headers and nested message/rfc822 headers.

That doesn't survive a cost-benefit analysis.

Your conversion is safe in a fantasy world where all message readers
magically understand =??=. But in that fantasy world you can use
_encoded_ UTF-8. Your SMTP extension is completely unnecessary.

The real world is more complicated. There's no safe way to send UTF-8.
Many readers don't handle =??=. Many readers don't _want_ to handle
=??=. They want unencoded 8-bit characters. But sendmail can't handle
characters between 128 and 159 in address fields.

Fixing sendmail isn't easy, but it's the fastest way to eliminate this
problem. What you're talking about is fixing sendmail _and_ adding a
painful conversion procedure to sendmail _and_ adding the same painful
conversion procedure to dozens of 8-bit-clean MTAs; that's much more
expensive, and will add years to the transition.

---Dan

Date: Thu, 4 Feb 1999 16:07:37 -0600
To: DRUMS <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Atoms and phrases (Was: UTF-8 in headers)
Sender: [email-masked]

I have Bcc'ed this message to [email-masked] since this discussion 
originated there, but have directed the discussion to DRUMS, since 
this is a DRUMS issue. Please direct future followup on this subtopic 
to the DRUMS list.

On 2/4/99 at 2:23 PM -0500, Keith Moore wrote:

>> I think not. I was working from the syntax in DRUMS (admittedly my 
>> version was dated March 1998, so it may be different now). The 
>> syntax I have gives:
>>
>> atom = [CFWS] 1*atext [CFWS]
>>
>> so "foo" and "bar" are atoms (the CFWS is optional).
>
> This probably points to a need for a clarification in DRUMS -
> an atom is intended to be the entire contiguous sequence of
> atext.
>> so "foobar" is a phrase consisting of two words side-by-side, as 
>> allowed by the syntax. It can also be legitimately parsed in many 
>> other ways, of course.
>
> this may be allowed by the grammar, but was not intended.

It was not intended. Back in draft -05, there was a sentence in the 
syntax discussion of the atom definition that said "Two atoms have to 
be separated by some other token, since putting two atoms next to 
each other would create a single atom." It was pointed out to me that 
this was incorrect, since the atom token itself includes the optional 
CFWS, so I removed the sentence. However, in the semantics 
description of atom, it says: "Semantically, the optional comments 
and FWS surrounding the rest of the characters are not part of the 
token; the token is only the run of atext characters in an atom..."

If further clarification is needed in the atom definition, or in the 
definition of phrase, to disambiguate, I'm open for suggestions.

pr
--
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (619)651-4478, Fax: (619)651-1102

Date: 4 Feb 1999 20:08:40 -0000
Mail-Followup-To: [email-masked]
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: UTF-8 in headers
Sender: [email-masked]

>From http://pobox.com/~djb/proto/immhf/token.html:

   An atom is a string of one or more characters terminated by the end
   of the field value or by any of these characters: space, tab, @, <,
   >, [, left parenthesis, comma, semicolon, colon, dot, double quote.
   (Note that adjacent atom tokens must be separated by at least one
   comment, space, or tab.)

RFC 822 says the same thing: atoms ``are delimited by the
self-delimiting symbols and by linear-white-space.''

Charles Lindsey writes:
> I was working from the syntax in DRUMS

Not a good idea. The ABNF+822bis syntax is an untested computer program
in a weak, error-prone programming language.

---Dan

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: UTF-8 in headers 
Date: Thu, 04 Feb 1999 14:23:44 -0500
Sender: [email-masked]

> >> Hmm! I doubt you will persuade the world to go along with that.
> 
> >Perhaps not.  But it would be helpful to know why this kind of 
> >approach would or would not work.  Or are you just saying that
> >it would be difficult to establish this convention in people's
> >minds?
> 
> Yes. The convention is just too unlike the way headers currently get written.

okay, understood.  of course it doesn't have to be that specific
convention, but *some* convention that people would be willing
to adopt.  granted that the idea that "I can put anything I want
into my header field" may be too widespread...

> >> 3. Within a <phrase>, I find the syntax ambiguous. Clearly foobar is a
> >> phrase, but is it one <word> or two (foo and bar are both <word>s, and the
> >> syntax allows adjacent <word>s in a <phrase>)? So can I put
> >> 	foo=?iso-8859-1?Q?bar?=   ?
> 
> >In "foobar" there is exactly one word.  Lexical symbols in structured
> >header fields are delimited by white space, comments, or specials.
> >(no, this isn't crystal clear in RFC 822, but it is there if you look
> >closely enough)
> 
> I think not. I was working from the syntax in DRUMS (admittedly my version
> was dated March 1998, so it may be different now). The syntax I have gives:
> 
> atom = [CFWS] 1*atext [CFWS]
> 
> so "foo" and "bar" are atoms (the CFWS is optional).

This probably points to a need for a clarification in DRUMS -
an atom is intended to be the entire contiguous sequence of
atext.

> so "foobar" is a phrase consisting of two words side-by-side, as allowed
> by the syntax. It can also be legitimately parsed in many other ways, of
> course.

this may be allowed by the grammar, but was not intended.

> >That's correct.  This was an explicit decision of the working group,
> >made during an IETF meeting in Santa Fe, if I recall.  The idea was 
> >that encoded-words were their own quoting mechanism, and therefore
> >should not be combined with double quotes.
> 
> Because at the time it didn't appear useful? Experience shows that
> designing syntax with particular cases excluded from otherwise general
> mechanisms usually leads to grief later on :-( .

The question is, how much grief would we have suffered had we done it
the other way?  Hindsight doesn't help us much with that one.
We make the best decisions we can at the time, and then we deal with them.
Sooner or later the amount of entropy due to old design decisions gets 
too large, and we wipe the slate clean and start more-or-less from scratch.  
(e.g. we put UTF-8 in message headers)

> >In hindsight, we might regret that decision, because it was one of
> >the considerations that forced us to invent a different syntax
> >for MIME parameters.
> 
> Or you could say that, having made one mistake, you compounded the error
> by making a second mistake, rather than undoing the first one :-( .

I'm not sure what you could mean by "undoing the first mistake".  The email 
world is still not ready for UTF-8 in headers, though if we manage the 
transition carefully it might be ready in a few years.   And it would seem 
silly to replace encoded-words with a different and slightly less broken 
mechanism for encoding 8bits in ASCII (but which might work for parameter 
values also) when we already have an investment in encoded-words and 
we'll want to go to UTF-8 in a few years anyway.  It also seems silly
to declare "it's okay to use encoded-words in quoted strings" when 
this will cause a fair amount of disruption.  I'd rather minimize the 
number of design changes (and transitions) than fix every "mistake" 
as soon as possible.

Also it seems gratuitously abusive to call 1342 a "mistake".  UTF-8 was not
available at the time; we were aware of 10646 but realized (correctly) that
it was not sufficiently mature.  We looked at several other alternatives
and all were rejected. In hindsight, I don't think we would make a different
decision.  

Several years ago I was invited to give a talk on MIME.  My last slide was
entitled "where we blew it" and I mentioned the failure of MIME to provide
a general way to put any octet string in a parameter.  This, I think,
was the real "mistake" (I prefer the word "oversight").  Had we thought 
of it at the time I think we would have made the MIME parameter syntax 
slightly more general.  But we would still have needed a 1342-like fix 
to put non-ASCII characters in existing headers.

It is unfortuante that people are trying to use encoded-words for things 
that they were never designed to do, but is not a failure of the design 
of encoded-words.

> >  (there were others also; in particular,
> >encoded-words were designed to encode human-readable text, and
> >were never intended to encode very long strings of characters
> >such as might be found in a filename)
> 
> Yes, I can see more weight in that part of the argument, though I see that
> DRUMS does not put any limit on line lengths in headers (and allows up to
> 998 in bodies).

This is another omission in DRUMS which should be fixed.
(remember, DRUMS is still an Internet-draft, so you're supposed to cite
it as "work in progress" :)

SMTP still imposes a line length of 1000 characters, and DRUMS messages
should be compatible with SMTP.  (The practical limit is much less
than 1000 characters; many user agents still display more-or-less
raw headers and either do not wrap long lines, or do not wrap headers 
appropriately).

> >> 8. Whilst one can see the thinking that lead to RFC 2231, I must point out
> >> that, in a parameter of a Mime header, the following is already legal:
> >> 	filename="=?iso-8859-1?Q?my-funny-file-name?="
> 
> >I don't know what you mean by "legal", since RFCs 1342, 1522, and 2047
> >all prohibit the interpretation of this string as an encoded-word.
> >(because it is in quotes). (and encoded-words are not legal MIME
> >parameter values outside of quotes because they contain = signs)
> 
> It is "legal" in the sense that "=?iso-8859-1?Q?my-funny-file-name?=" is a
> syntactically correct quoted-string. Of course, its semantics is not what
> you would expect (it will contruct a file with a peculiar name 35
> characters long on a strictly-conforming implementation).

right.

> >There are problems with long filenames and header wrapping - an 
> >encoded-word has a maximum length and some mailers still have line
> >length limitations.  And people are reluctant to change a 
> >long-established specification.  (I've asked people before if
> >encoded-words should be changed to be more uniform and couldn't 
> >get much support for the idea)
> 
> But it seems we are turning up more and more situations that are going to
> run into this problem.
> 
> Right! Let us suppose, just be way of Thinking-Out-Loud, that some
> extension of RFC2047 (or perhaps some 2047bis) were made that allowed
> encoded-words within a quoted-string (with perhaps a prohibition remaining
> within addr-specs and msg-ids). What calamities would ensue?

People would start using them in addr-specs, expecting them to be
decoded before display.   Some gateways would translate them to 
raw characters, and fail to translate them back, which would make 
replies fail, as well as causing failures with other tools that 
recognize addresses.  Some gateways would translate them to raw 
characters, and translate them back in a way that didn't match the 
original name, and cause similar (but more subtle) failures.
Systems that use name=  or content-disposition filenames would
behave inconsistently - some would store the file under a bizarre
name, while others would store the file with the decoded name.

All of these things happen right now to some degree due to broken
implementations.  But by changing direction, we would make the
overall behavior worse. 

I don't see the point in changing encoded-words at this point, especially
not if the purpose is to make eventual downgrading to UTF-8 easier.  
Yes, the downgrading rules will be somewhat complex, with lots of special 
cases.  But that's not an argument for making still more changes to
existing software (and dealing with yet another set of early 
implementation bugs).  Rather, we should get to work defining the 
eventual use of UTF-8 in email and managing *that* transition. 

Keith

To: [email-masked]
Xref: clerew local.mime:30
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Thu, 4 Feb 1999 15:36:06 GMT
Lines: 163
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> Hmm! I doubt you will persuade the world to go along with that.

>Perhaps not.  But it would be helpful to know why this kind of 
>approach would or would not work.  Or are you just saying that
>it would be difficult to establish this convention in people's
>minds?

Yes. The convention is just too unlike the way headers currently get written.
> 

>> 3. Within a <phrase>, I find the syntax ambiguous. Clearly foobar is a
>> phrase, but is it one <word> or two (foo and bar are both <word>s, and the
>> syntax allows adjacent <word>s in a <phrase>)? So can I put
>> 	foo=?iso-8859-1?Q?bar?=   ?

>In "foobar" there is exactly one word.  Lexical symbols in structured
>header fields are delimited by white space, comments, or specials.
>(no, this isn't crystal clear in RFC 822, but it is there if you look
>closely enough)

I think not. I was working from the syntax in DRUMS (admittedly my version
was dated March 1998, so it may be different now). The syntax I have gives:

atom = [CFWS] 1*atext [CFWS]

so "foo" and "bar" are atoms (the CFWS is optional).

word = atom / quoted-string

so "foo" and "bar" are words

phrase = 1*word / obs-phrase

so "foobar" is a phrase consisting of two words side-by-side, as allowed
by the syntax. It can also be legitimately parsed in many other ways, of
course.

So the syntax is ambiguous, but does not actually make any difference
within DRUMS itself (but does when you introduce encoded-words later on).
So is this a bug in DRUMS? It would seem so. Indeed there are many
examples of ambiguities-that-do-not-matter in DRUMS.

>> but not
>> 	"=?iso-8859-1?Q?Charles=20H.=20Lindsey?="

>That's correct.  This was an explicit decision of the working group,
>made during an IETF meeting in Santa Fe, if I recall.  The idea was 
>that encoded-words were their own quoting mechanism, and therefore
>should not be combined with double quotes.

Because at the time it didn't appear useful? Experience shows that
designing syntax with particular cases excluded from otherwise general
mechanisms usually leads to grief later on :-( .

>In hindsight, we might regret that decision, because it was one of
>the considerations that forced us to invent a different syntax
>for MIME parameters.

Or you could say that, having made one mistake, you compounded the error
by making a second mistake, rather than undoing the first one :-( .

>  (there were others also; in particular,
>encoded-words were designed to encode human-readable text, and
>were never intended to encode very long strings of characters
>such as might be found in a filename)

Yes, I can see more weight in that part of the argument, though I see that
DRUMS does not put any limit on line lengths in headers (and allows up to
998 in bodies).

>See above.  I think you'll find that in practice many (most?) decoders
>will decode encoded-words even if they appear in quoted-strings.

>> 8. Whilst one can see the thinking that lead to RFC 2231, I must point out
>> that, in a parameter of a Mime header, the following is already legal:
>> 	filename="=?iso-8859-1?Q?my-funny-file-name?="

>I don't know what you mean by "legal", since RFCs 1342, 1522, and 2047
>all prohibit the interpretation of this string as an encoded-word.
>(because it is in quotes). (and encoded-words are not legal MIME
>parameter values outside of quotes because they contain = signs)

It is "legal" in the sense that "=?iso-8859-1?Q?my-funny-file-name?=" is a
syntactically correct quoted-string. Of course, its semantics is not what
you would expect (it will contruct a file with a peculiar name 35
characters long on a strictly-conforming implementation).

>> So what would be wrong with letting it mean what it looks as though it
>> means?

>There are problems with long filenames and header wrapping - an 
>encoded-word has a maximum length and some mailers still have line
>length limitations.  And people are reluctant to change a 
>long-established specification.  (I've asked people before if
>encoded-words should be changed to be more uniform and couldn't 
>get much support for the idea)

But it seems we are turning up more and more situations that are going to
run into this problem.

Right! Let us suppose, just be way of Thinking-Out-Loud, that some
extension of RFC2047 (or perhaps some 2047bis) were made that allowed
encoded-words within a quoted-string (with perhaps a prohibition remaining
within addr-specs and msg-ids). What calamities would ensue?

1. Existing strictly-conforming implementations would display the phrase
"=?iso-8859-1?Q?Charles=20H.=20Lindsey?=" as-is. Yes, it is not a pretty
sight, but those of us without RFC2047-conforming implementations have been
seeing such things for years. It does not seem a show stopper.

2. Many (most?) existing but non-conforming implementations would suddenly
become conforming (and would correctly decode that phrase).

3. Every syntactically correct header, under either the old or the new
syntax, would still be syntactically correct under either old or new
implementations. So nothing is going to blow up in your face that did not
blow up in your face before.

4. You could immediately use encoded-words in Mime parameters. This make
some (not all) of the features of RFC2231 redundant. Maybe you declare
those features "deprecated".

5. For really long parameters, you keep the mechanism described in section
3 of RFC2231, and obviously you keep the language-specification within
encoded-words feature of RFC2231.

6. You advise all people developing new headers that they should include
quoted-string as an option in any human-writeable fields that they invent,
or to include encoded-word in their syntax explicitly, or both. And you
advise them to make provision for long fields if necessary (so, for
example, we might have to look at long newsgroup-names).

7. When you invent UTF-8 headers as an extension to mail, you enact that
all downgrading is to be done by RFC2047(bis).

So, what other problems might there be? I see no show stoppers so far.

>>  "X-" header, I can use RFC2047 stuff wherever I like, whether it
>> fits with any supposed syntax or not. But what about other headers, not
>> defined in RFC822 or DRUMS, but defined in extensions? That is unclear to
>> me.

>It's difficult to answer the question.  The definition for a particular
>field should specify whether encoded-words are allowed.  Unfortunately,
>this means that the encoded-word decoder has to know how to parse each 
>field, and different fields may be either structured or unstructured,
>may allow comments or not, and have different notions of special characters. 
>In general, encoded-words should only be used for text intended to
>be presented to humans, not for machine-readable text like filenames.
>In practice, most decoders will decode encoded-words anywhere they see them.

I think your last sentence sums it up (what harm could such an approach
do?). It might be better to encorage designers of new fields to ensure
that new encoded-words only arose within "..." .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: UTF-8 in headers 
Date: Wed, 03 Feb 1999 19:24:35 -0500
Sender: [email-masked]

> >Perhaps we could establish something of a uniform syntax for all new 
> >headers, and along with it, uniform rules for converting to 7bit.
> >For example, declare that all human-readable strings in future fields 
> >are to be enclosed in single quotes.  Then it would be easy for a 
> >converter to know how to translate between UTF-8 and 7bit - convert 
> >the strings within quotes and leave everything else alone.
> 
> Hmm! I doubt you will persuade the world to go along with that.

Perhaps not.  But it would be helpful to know why this kind of 
approach would or would not work.  Or are you just saying that
it would be difficult to establish this convention in people's
minds?
 
> >RFC 2047 attempts to define this very carefully, but that hasn't stopped
> >people from trying to use encoded-words where they don't fit - such as 
> >within a quoted string.
> 
> I still find the RFC2047 rules bizarre.

To some degree, so do I, even though I (mostly) understand the reasons
that they're there.
 
> 1. It seems that in an <unstructured> an encoded-word must have whitespace in
> front of it and behind it. 

Yes.   I believe the original idea in RFC 1342 was that you could parse 
any header field by just looking at the words surrounded by white space
and seeing whether they matched the pattern =\?[^?]+?[BQbq]?[^?]+\?=
But this overlooked that in structured fields you can have certain 
specials immediately adjacent to a 'word' with no intervening white space
(the usual case is a word in a phrase which is normally separated
by white space from surrounding special characters, but this is 
not required).  So the rules got changed for structured fields.
Yes, this is a pain.

> So if I really really want my Subject: to be
> "=?iso-8859-1?Q?my=20text?=" I can put "\=?iso-8859-1?Q?my=20text?", though
> it is not clear that reading agents are expected to hideaway the "\".

Or you could encoded the encoded-word as an encoded-word.
Not that this case happens very often...

"\" affects only how the field is parsed, not how it is displayed.
Presentation of the "\" is not defined.

> 2. But in a comment this is not so. My encoded-word can have other
> characters adjacent to it. But preceding it by a "\" will definitely stop
> it being decoded, and it is more likely that the reading agent will hide
> the "\".

It's not clear what would happen in practice (the combination of "\"
with encoded-words is an ambiguity that should probably be cleared up), 
but I don't know of any mail readers that hide "\"s.

> 3. Within a <phrase>, I find the syntax ambiguous. Clearly foobar is a
> phrase, but is it one <word> or two (foo and bar are both <word>s, and the
> syntax allows adjacent <word>s in a <phrase>)? So can I put
> 	foo=?iso-8859-1?Q?bar?=   ?

In "foobar" there is exactly one word.  Lexical symbols in structured
header fields are delimited by white space, comments, or specials.
(no, this isn't crystal clear in RFC 822, but it is there if you look
closely enough)

> 4. It seems the following are allowed in a <phrase>
> 	Charles Lindsey
> 	Charles "H." Lindsey
> 	"Charles H. Lindsey"
> 	=?iso-8859-1?Q?Charles=20H.=20Lindsey?=
> but not
> 	"=?iso-8859-1?Q?Charles=20H.=20Lindsey?="

That's correct.  This was an explicit decision of the working group,
made during an IETF meeting in Santa Fe, if I recall.  The idea was 
that encoded-words were their own quoting mechanism, and therefore
should not be combined with double quotes.

In hindsight, we might regret that decision, because it was one of
the considerations that forced us to invent a different syntax
for MIME parameters.  (there were others also; in particular,
encoded-words were designed to encode human-readable text, and
were never intended to encode very long strings of characters
such as might be found in a filename)  But the rule has been 
that way for about 8 years now, through three document revisions,
so it seems too late to change it.

> 5. It seems that the syntax was designed so that an encoded-word would
> always be syntactically correct in the places where it was allowed even in
> the absence of RFC2047. Which would seem to allow reading agents not to
> decode them, for whatever reason. Presumably it was this policy that lead
> to RFC 2231.

The syntax was designed so that ordinary RFC 822 parsers would not
be confused by the presence of an encoded-word in a structured header
field - an encoded-word looks like an 'atom' to an RFC 822 parser.
It was also designed to be unusual enough that it was unlikely
to appear in a header field unless it were actually intended to represent
non-ASCII text.  Finally, the characters within an encoded word were 
chosen to survive translation between all known character sets.
(lots of 7-bit ISO 646 national character sets were in use at the time,
not to mention EBCDIC)

> 6. The rule that an encoded-word cannot occur in a <quoted-string> I find
> particularly odd. In fact that <phrase> I quoted above 
> 	"=?iso-8859-1?Q?Charles=20H.=20Lindsey?="
> IS allowed. It fits the syntax of a <quoted-string>. But reading agents
> would not be allowed to decode it. AFAICS, the only place where allowing
> an encoded-word within a <quoted-string> would be an embarassment would be
> in the <local-part> of an <addr-spec> - a problem which could surely be
> fixed in other ways. 

Though in hindsight the decision may seem like it was short-sighted,
avoiding embarassment was not the justification for the decision.

> In fact, part of the problem appears to be that the
> syntax of DRUMS is set out in such a way as not to facilitate dealing with
> problems that do not arise within DRUMS itself (my foobar example above is
> a case in point).

DRUMS was deliberately scoped to not look at MIME (because the task
was already very difficult. There was also (for awhile) an assumption 
that the DRUMS documents might go directly to full Internet Standard,
without first going through Proposed Standard and Draft Standard.
Hence it was necessary to keep DRUMS from referencing MIME, because
MIME is only a Draft Standard, and a full Standard cannot reference
a Draft Standard.

> 7. So can someone please explain to me why the use of encoded-words in a
> <quoted-string> was outlawed, and what evils would ensue from letting them
> in?

See above.  I think you'll find that in practice many (most?) decoders
will decode encoded-words even if they appear in quoted-strings.

> 8. Whilst one can see the thinking that lead to RFC 2231, I must point out
> that, in a parameter of a Mime header, the following is already legal:
> 	filename="=?iso-8859-1?Q?my-funny-file-name?="

I don't know what you mean by "legal", since RFCs 1342, 1522, and 2047
all prohibit the interpretation of this string as an encoded-word.
(because it is in quotes). (and encoded-words are not legal MIME
parameter values outside of quotes because they contain = signs)

RFC 2157 suggests that FTBP parameters can be mapped as encoded-words,
but points out that this 'is not the normal usage of encoded-words'.
RFC 2388 says that encoded-words can be used in Content-Disposition,
but this is clearly an error as it contradicts RFC 2183.

> So what would be wrong with letting it mean what it looks as though it
> means?

There are problems with long filenames and header wrapping - an 
encoded-word has a maximum length and some mailers still have line
length limitations.  And people are reluctant to change a 
long-established specification.  (I've asked people before if
encoded-words should be changed to be more uniform and couldn't 
get much support for the idea)

>  "X-" header, I can use RFC2047 stuff wherever I like, whether it
> fits with any supposed syntax or not. But what about other headers, not
> defined in RFC822 or DRUMS, but defined in extensions? That is unclear to
> me.

It's difficult to answer the question.  The definition for a particular
field should specify whether encoded-words are allowed.  Unfortunately,
this means that the encoded-word decoder has to know how to parse each 
field, and different fields may be either structured or unstructured,
may allow comments or not, and have different notions of special characters. 
In general, encoded-words should only be used for text intended to
be presented to humans, not for machine-readable text like filenames.
In practice, most decoders will decode encoded-words anywhere they see them.

> I am asking all these questions just to make sure I understand the present
> situation correctly, because it would be useless to try to look for
> solutions to the more general problem without doing that first.

right.

Keith

To: [email-masked]
Xref: clerew local.mime:28
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Wed, 3 Feb 1999 18:01:33 GMT
Lines: 89
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>Perhaps we could establish something of a uniform syntax for all new 
>headers, and along with it, uniform rules for converting to 7bit.
>For example, declare that all human-readable strings in future fields 
>are to be enclosed in single quotes.  Then it would be easy for a 
>converter to know how to translate between UTF-8 and 7bit - convert 
>the strings within quotes and leave everything else alone.

Hmm! I doubt you will persuade the world to go along with that.

>> Surely, a more sensible approach would be to define more carefully where
>> RFC2047 could be used. For example, that no protocol word or protocol
>> character (i.e. anything specified by an explicit "..." in the syntax)
>> could be included within an encoded-word. OK, that is too simplistic as it
>> stands, but surely there MUST be some uniform principles that could be
>> applied.

>RFC 2047 attempts to define this very carefully, but that hasn't stopped
>people from trying to use encoded-words where they don't fit - such as 
>within a quoted string.

I still find the RFC2047 rules bizarre.

1. It seems that in an <unstructured> an encoded-word must have whitespace in
front of it and behind it. So if I really really want my Subject: to be
"=?iso-8859-1?Q?my=20text?=" I can put "\=?iso-8859-1?Q?my=20text?", though
it is not clear that reading agents are expected to hideaway the "\".

2. But in a comment this is not so. My encoded-word can have other
characters adjacent to it. But preceding it by a "\" will definitely stop
it being decoded, and it is more likely that the reading agent will hide
the "\".

3. Within a <phrase>, I find the syntax ambiguous. Clearly foobar is a
phrase, but is it one <word> or two (foo and bar are both <word>s, and the
syntax allows adjacent <word>s in a <phrase>)? So can I put
	foo=?iso-8859-1?Q?bar?=   ?

4. It seems the following are allowed in a <phrase>
	Charles Lindsey
	Charles "H." Lindsey
	"Charles H. Lindsey"
	=?iso-8859-1?Q?Charles=20H.=20Lindsey?=
but not
	"=?iso-8859-1?Q?Charles=20H.=20Lindsey?="

5. It seems that the syntax was designed so that an encoded-word would
always be syntactically correct in the places where it was allowed even in
the absence of RFC2047. Which would seem to allow reading agents not to
decode them, for whatever reason. Presumably it was this policy that lead
to RFC 2231.

6. The rule that an encoded-word cannot occur in a <quoted-string> I find
particularly odd. In fact that <phrase> I quoted above 
	"=?iso-8859-1?Q?Charles=20H.=20Lindsey?="
IS allowed. It fits the syntax of a <quoted-string>. But reading agents
would not be allowed to decode it. AFAICS, the only place where allowing
an encoded-word within a <quoted-string> would be an embarassment would be
in the <local-part> of an <addr-spec> - a problem which could surely be
fixed in other ways. In fact, part of the problem appears to be that the
syntax of DRUMS is set out in such a way as not to facilitate dealing with
problems that do not arise within DRUMS itself (my foobar example above is
a case in point).

7. So can someone please explain to me why the use of encoded-words in a
<quoted-string> was outlawed, and what evils would ensue from letting them
in?

8. Whilst one can see the thinking that lead to RFC 2231, I must point out
that, in a parameter of a Mime header, the following is already legal:
	filename="=?iso-8859-1?Q?my-funny-file-name?="
So what would be wrong with letting it mean what it looks as though it
means?

9. In any "X-" header, I can use RFC2047 stuff wherever I like, whether it
fits with any supposed syntax or not. But what about other headers, not
defined in RFC822 or DRUMS, but defined in extensions? That is unclear to
me.

I am asking all these questions just to make sure I understand the present
situation correctly, because it would be useless to try to look for
solutions to the more general problem without doing that first.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Tue, 02 Feb 1999 19:24:40 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: UTF-8 in headers
To: [email-masked]
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> Ned Freed <[email-masked]> writes:

> > > > On Sun, 24 Jan 1999, Charles Lindsey wrote:

> > > So I can see a great danger that the grand canonical method of downgrading
> > > UTF-8 headers (which is subject of this thread) is likely to degenerate
> > > into a large collection of special cases, all done differently. And I do
> > > not see any immediate clean answer to this :-( .

> > There is no clean answer short of writing out explicit rules, which is what
> > I've been saying from the start has to be done.

> I cannot believe that anyone could seriously suggest that the way out of
> this problem is to invent a separate ad hoc solution for every situation.
> That is just not how sensible systems are designed.

Sigh. Nowhere did I say, or even imply, this. Explicit rules can be generic in
form. The difference here is between not having rules that clearly say what to
do and having them.

As such, this is purely a strawman of your own making.

> New headers for mail and news are being invented all the time. Often, they
> are deployed and widely adopted before the RFC gets written (yes, that is
> not supposed to happen, but it does). And now you are saying that everyone
> who invents a new headers has also to specify how it gets downgraded to
> 7bit?

Nope, that's not what I'm saying at all. What I'm saying is that we need
explicit downgrading rules -- not having any will lead to chaos, as you
yourself said, and that's just not acceptable, if for no other reason that
interoperability is an absolute requirement of the process.

> Surely, a more sensible approach would be to define more carefully where
> RFC2047 could be used. For example, that no protocol word or protocol
> character (i.e. anything specified by an explicit "..." in the syntax)
> could be included within an encoded-word. OK, that is too simplistic as it
> stands, but surely there MUST be some uniform principles that could be
> applied.

Uniform principles are what I said we should have.

> If not, then my suggestion of encapsulating headers and bodies as
> multiparts, which you all described as "ugly" should be looked at again.
> At least, it is uniform and applicable for all situations.

However, my guess is that this is a complete operational nonstarter.

> BTW, is RFC 2231 actually deployed and in regular use in meatspace yet?

There is some deployment, but not a lot. It also depends on what parts of RFC
2231 you're talking about -- I don't believe I've ever seen the language
tagging facilities used, for example.

				Ned

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: UTF-8 in headers 
Date: Tue, 02 Feb 1999 22:08:15 -0500
Sender: [email-masked]

> >> >On Sun, 24 Jan 1999, Charles Lindsey wrote:
> 
> >> So I can see a great danger that the grand canonical method of downgrading
> >> UTF-8 headers (which is subject of this thread) is likely to degenerate
> >> into a large collection of special cases, all done differently. And I do
> >> not see any immediate clean answer to this :-( .
> 
> >There is no clean answer short of writing out explicit rules, which is what
> >I've been saying from the start has to be done.
> 
> I cannot believe that anyone could seriously suggest that the way out of
> this problem is to invent a separate ad hoc solution for every situation.
> That is just not how sensible systems are designed.
> 
> New headers for mail and news are being invented all the time. Often, they
> are deployed and widely adopted before the RFC gets written (yes, that is
> not supposed to happen, but it does). And now you are saying that everyone
> who invents a new headers has also to specify how it gets downgraded to
> 7bit?

Perhaps we could establish something of a uniform syntax for all new 
headers, and along with it, uniform rules for converting to 7bit.
For example, declare that all human-readable strings in future fields 
are to be enclosed in single quotes.  Then it would be easy for a 
converter to know how to translate between UTF-8 and 7bit - convert 
the strings within quotes and leave everything else alone.  

> Surely, a more sensible approach would be to define more carefully where
> RFC2047 could be used. For example, that no protocol word or protocol
> character (i.e. anything specified by an explicit "..." in the syntax)
> could be included within an encoded-word. OK, that is too simplistic as it
> stands, but surely there MUST be some uniform principles that could be
> applied.

RFC 2047 attempts to define this very carefully, but that hasn't stopped
people from trying to use encoded-words where they don't fit - such as 
within a quoted string.

Keith

To: [email-masked]
Xref: clerew local.mime:25
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Tue, 2 Feb 1999 10:39:48 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> Ned Freed <[email-masked]> writes:

>> >On Sun, 24 Jan 1999, Charles Lindsey wrote:

>> So I can see a great danger that the grand canonical method of downgrading
>> UTF-8 headers (which is subject of this thread) is likely to degenerate
>> into a large collection of special cases, all done differently. And I do
>> not see any immediate clean answer to this :-( .

>There is no clean answer short of writing out explicit rules, which is what
>I've been saying from the start has to be done.

I cannot believe that anyone could seriously suggest that the way out of
this problem is to invent a separate ad hoc solution for every situation.
That is just not how sensible systems are designed.

New headers for mail and news are being invented all the time. Often, they
are deployed and widely adopted before the RFC gets written (yes, that is
not supposed to happen, but it does). And now you are saying that everyone
who invents a new headers has also to specify how it gets downgraded to
7bit?

Surely, a more sensible approach would be to define more carefully where
RFC2047 could be used. For example, that no protocol word or protocol
character (i.e. anything specified by an explicit "..." in the syntax)
could be included within an encoded-word. OK, that is too simplistic as it
stands, but surely there MUST be some uniform principles that could be
applied.

If not, then my suggestion of encapsulating headers and bodies as
multiparts, which you all described as "ugly" should be looked at again.
At least, it is uniform and applicable for all situations.

BTW, is RFC 2231 actually deployed and in regular use in meatspace yet?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5