Subject: [EAI] UTF-8 in a Subject header field?

From: Dave CROCKER <[email-masked]>
Date: Fri, 24 Dec 2010 09:30:11 -0800

Hi.

I am reading through draft-ietf-eai-rfc5335bis-07.txt and can see modifications 
to support Unicode in addresses and Received header fields.  However I do not 
see support for UTF-8 or Unicode in the Subject header field or any other header 
fields.

Since my understanding of the goal of the draft is to provide native UTF-8 
support for the email header, I assume it is meant for all user-visible data.

So I thought I should ask how the working group expects UTF-8 Subject data and 
data in other free-form header-fields to be handled.

Thanks.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: John Levine <[email-masked]>
Date: 24 Dec 2010 18:47:08 -0000

>So I thought I should ask how the working group expects UTF-8 Subject
>data and data in other free-form header-fields to be handled.

Isn't that covered by RFC 2047 ?

  Subject: =?UTF-8?Q?Joyeux=20No=C3=ABl?=

I realize that if we're going to allow bare UTF-8 in addresses, it
might seem reasonable to allow it everywhere, but it's better to avoid
multiple ways to do the same thing.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Dec 2010 14:20:56 -0500

Dave,

First of all, thanks for your careful and comprehensive review
(and, to others, for theirs).  I'm trying to consolidate most of
the more substantive review comments and at least my personal
(as participant and sometime author, but not as WG Chair)
responses to them.  It has turned out to be a larger task than I
had originally estimated, so let me respond to this one note
separately...

--On Friday, December 24, 2010 09:30 -0800 Dave CROCKER
<[email-masked]> wrote:

> Hi.
> 
> I am reading through draft-ietf-eai-rfc5335bis-07.txt and can
> see modifications to support Unicode in addresses and Received
> header fields.  However I do not see support for UTF-8 or
> Unicode in the Subject header field or any other header fields.
> 
> Since my understanding of the goal of the draft is to provide
> native UTF-8 support for the email header, I assume it is
> meant for all user-visible data.

The WG's intent, as you have deduced, is that one should be able
to simply use UTF-8 in all unstructured header fields,
especially those in which encoded words are permitted by RFC
2047 and its successors.  That intent should be clear from the
second paragraph of Section 2 ("The traditional format...").
Trace fields and fields containing address information are
singled out for special treatment only because of their
interaction with the envelope specifications in rfc5336bis.

The specific problem in this case is that, while the document
specifies many header fields as being UTF-8-capable by virtue of
the changes in Section 4.3, particularly to <ctext>, the RFC
5322 definition of <subject>, <comments>, and <optional-field>s
(and some obsolete equivalents) rely on <unstructured>, which,
in turn, relies on the RFC 5234 definition of VCHAR.  And the
5234 definition of VCHAR is definitely ASCII-only.

So, Section 4.3 needs an updated production for <unstructured>
as well.

The document would, IMO, benefit from a tad more tutorial/
overview text that better explains what header fields are and
are not extended, but that should not be strictly necessary.

This situation raises an aspect of one or two more general
problems that we should probably move to the apps-discuss list:

When working on the previous (experimental) set of documents,
the WG discovered that the IETF has no standardized model or
agreed-upon practices for how to handle the ABNF if an extension
to one document modifies its syntax rules.  The WG struggled, at
length, with the question of how to do that in a way that would
minimize confusion and concluded that the approach it took (old
production names with new definitions) was probably the least
bad.  That may not have been the right choice, and it does lend
itself to errors like this one, but I'd encourage anyone who
wants to complain about the approach taken to think through, and
suggest, alternatives.  Note that one possible alternative would
be to simply replicate the entire RFC 5322 production set, as
extended, in this document, possibly with new names for
everything, but the size of that grammar and the likelihood of
become unsynchronized if the 5322 syntax ever changes, probably
does not recommend that approach.

The other, possibly-separate, problem is that none of the
authors/editors of the EAI core documents had significant prior
experience producing RFC-ready IETF specs, much less specs with
sensitive ABNF.   That is a problem that we need to deal with,
possibly with more educational efforts or metalanguage tutorials
as well as with patience during the review process, unless we
want to doom the IETF to a situation in which only metalanguage
experts and the same old group of people (in my case and maybe
yours, with both meanings of "old") can write new or updated
documents.

best wishes to all for whatever you are celebrating at this time
of year.
   john

> 
> So I thought I should ask how the working group expects UTF-8
> Subject data and data in other free-form header-fields to be
> handled.

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Dec 2010 14:41:31 -0500

--On Friday, December 24, 2010 18:47 +0000 John Levine
<[email-masked]> wrote:

> I realize that if we're going to allow bare UTF-8 in
> addresses, it might seem reasonable to allow it everywhere,
> but it's better to avoid multiple ways to do the same thing.

Actually, one of the EAI goals has been to gradually retire
"multiple ways to do the same thing" by having encoded words
become gradually obsolete (see Section 7.2 of
draft-ietf-eai-frmwrk-4952bis).

FWIW, at some level, one could "avoid multiple ways to do the
same thing" --and have a much more robust mail environment in
the short and medium term-- by declaring address (and other
envelope-related) information as ASCII forever and confining
internationalization efforts to display names and other textual
subfields that could be expressed as encoded words.   For better
or worse, that option is  clearly not acceptable to large, and
growing, groups of Internet users -- the choices were between
having the IETF standardize an internationalization model that
would eventually work (and interoperate) worldwide and having a
lot of certainly-leaky local protocol extensions that would
divide the Internet mail environment back up into islands
connected by gateways.  

Stated more aggressively, much of the design of the EAI work has
been to move toward a target in which Internet email is as fully
internationalized as possible, with a minimum of kludges left
over from either legacy ASCII or the transition process.
Encoded words were, IMO, an ingenious solution given the
constraints of the time.  But, if we are going to move toward
internationalized email, rather than treating all address and
header elements as ASCII-forever protocol elements, then the
assumption under which EAI was chartered, and the conclusion of
the WG, has been that we needed to figure out how to actually do
that, rather than carrying old i18n-supporting kludges around
forever.

best,
     john

---

From: "John R Levine" <[email-masked]>
Date: 24 Dec 2010 15:57:17 -0500

> draft-ietf-eai-frmwrk-4952bis).

Hmmn.  Now that I look at all the documents, I see that.

With that in mind, I'd suggest that the EAI documents would benefit by 
being shorter and more brutal: anywhere you can use arbitrary ASCII you 
can now use arbitrary UTF-8.  All other encodings are deprecated, no 
charsets other than UTF8 in top-level MIME parts, A-labels in addresses 
will cause a razzberry to come out of your speaker.  If you want to send 
mail to somebody who doesn't grok EAI, it's your MUA's job to try to 
create a mail-classic version.

It also suggests that STARTEAI to confirm that we are now in the world of 
EAI would be better than trying to flag every command that might have an 
argument or response.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Claudio Allocchio <[email-masked]>
Date: Fri, 24 Dec 2010 22:41:10 +0100 (CET)

> best wishes to all for whatever you are celebrating at this time

Thank you, and let me add my best wishes, too!

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: "Daniel Taharlev" <[email-masked]>
Date: Fri, 24 Dec 2010 21:48:07 +0000

+1

Daniel Taharlev
[email-masked]

-----Original Message-----
From: Claudio Allocchio <[email-masked]>
Sender: [email-masked]
Date: Fri, 24 Dec 2010 22:41:10 
To: John C Klensin<[email-masked]>
Cc: Claudio Allocchio<[email-masked]>; <[email-masked]>; Shawn Steele<[email-masked]>; <[email-masked]>
Subject: [EAI] Seasons' Greetings!

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Dec 2010 19:43:14 -0500

--On Friday, December 24, 2010 15:57 -0500 John R Levine
<[email-masked]> wrote:

> With that in mind, I'd suggest that the EAI documents would
> benefit by being shorter and more brutal: anywhere you can use
> arbitrary ASCII you can now use arbitrary UTF-8.  All other
> encodings are deprecated, no charsets other than UTF8 in
> top-level MIME parts, A-labels in addresses will cause a
> razzberry to come out of your speaker.  If you want to send
> mail to somebody who doesn't grok EAI, it's your MUA's job to
> try to create a mail-classic version.

Some variations on this theme were considered at length by the
WG, probably circa 2006.  The problem is that it creates a
nasty, probably fatal, transition problem at a level that would
make "new mail protocol on new port and just try one, then the
other" attractive.   The WG decided that was a bad idea; the
current model has now effectively been through IETF Last Call
and consensus three times: approval of what became RFC 4942
(March 2007), approval of what became RFC 5335, 5336, and 5337
(July 2008), and approval of draft-ietf-eai-frmwrk-4952bis (last
September).  Those are not merely WG decisions but decisions
that the IETF has had multiple chances to review over a nearly 3
1/2 year period, with published Experimental protocols and
implementations in the middle.   

At some point, it has to be too late to redesign the basic
model, at least absent empirical demonstration of showstoppers.

> It also suggests that STARTEAI to confirm that we are now in
> the world of EAI would be better than trying to flag every
> command that might have an argument or response.

There is not, as far as I can tell, an "every command"
situation.  The commands that might plausibly be flagged are:

	-- MAIL (as proposed by multiple reviewers but not yet
	accepted by the WG -- see subsequent notes)
	
	-- VRFY and EXPN. 

That combination comes right out of the "mail transaction" model
of RFC 821 that was made more explicit in 2821.  It is not
exactly an EAI invention, nor is it "every command".  One could
do a STARTEAI instead but it would require extra turnarounds in
the non-pipeline case (and, without doing much analysis, I think
even in the pipeline one).

    best,
     john

---

From: "John R Levine" <[email-masked]>
Date: 24 Dec 2010 22:05:56 -0500

> WG, probably circa 2006. ...

But I don't see much difference from what the docs say now.  If you're 
sending EAI, you can put UTF-8 in the envelope and pretty much anywhere in 
the headers.  If you're not, you can't, give or take RFC2045 encoded 
strings and A-labels.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Sun, 26 Dec 2010 13:36:36 -0500

--On Friday, December 24, 2010 10:05 PM -0500 John R Levine
<[email-masked]> wrote:

>>> benefit by being shorter and more brutal: anywhere you can
>>> use arbitrary ASCII you can now use arbitrary UTF-8.  All
>>> other encodings are deprecated, no charsets other than UTF8
>>> in top-level MIME parts, A-labels in addresses will cause a
> But I don't see much difference from what the docs say now.
> If you're sending EAI, you can put UTF-8 in the envelope and
> pretty much anywhere in the headers.  If you're not, you
> can't, give or take RFC2045 encoded strings and A-labels.

No razzberries for A-labels, no deprecation of content-type
charsets other than UTF-8 (probably a good idea at some point,
but out of scope).  But certainly, if you want to send something
with explicit non-ASCII (i.e., UTF-8 with codepoints about
U+007F) in the headers, or with explicit non-ASCII in the
envelope, without the extension being offered in the next hop,
then the sender has to cope.  And "cope" is limited to:

	(1) For the MUA, do whatever it is that MUAs might do,
	taking advantage of using whatever out-of-band
	information (such as alternate ASCII addresses or
	transliteration rather than encodings)might be available
	(and, I trust obviously, not using or depending on any
	out-of-band information that isn't available).
	
	(2) For the MSA, act on behalf of the MUA as needed and
	previously negotiated between the two or with the sender
	(that negotiation can be completely out of band, rather
	than a matter of protocols), again utilizing whatever
	out-of-band information is available and whose use is
	authorized by the sender.
	
	(3) For a gateway, do whatever it is that gateways might
	do.
	
	(4) For anything else, reject or bounce the message as
	undeliverable.

I assume the WG would appreciate advice as to whether things
really need to be spelled out in that much detail -- certainly
some of it is a rephrasing of rules that already exist in RFC
5321 about who can change what -- and, if so, where it should go.

      john

---

From: "John R Levine" <[email-masked]>
Date: 26 Dec 2010 21:08:37 -0500

>> But I don't see much difference from what the docs say now. ...

> but out of scope).

True.

> without the extension being offered in the next hop,

Here's where brutality could be helpful.  Rather than calling out possible 
ways to sneak an EAI message through other than EAI channels, just say 
bounce unless you happen to have a non-standard way to get the message to 
the recipient.  And perhaps leave out the last part of the sentence, since 
you can always do whatever non-standard extensions you want.

Are there actually MSAs that downgrade EAI messages?  I have no idea, 
never having had to address this particular problem.

Regards,