Subject: [EAI] I-D ACTION:draft-ietf-eai-downgrade-03.txt

From: [email-masked]
Date: Thu, 08 Mar 2007 15:50:04 -0500

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.
This draft is a work item of the Email Address Internationalization Working Group of the IETF.

	Title		: Downgrading mechanism for Email Address Internationalization
	Author(s)	: Y. Yoneya, K. Fujiwara
	Filename	: draft-ietf-eai-downgrade-03.txt
	Pages		: 18
	Date		: 2007-3-8
	
Traditional mail systems handle only US-ASCII characters in SMTP
   envelope and mail header fields.  The Email Address
   Internationalization is implemented by allowing UTF-8 characters in
   SMTP envelope and mail header fields (UTF8SMTP).  To deliver Non-
   ASCII mail address via UTF8SMTP non-compliant environment, some sort
   of converting mechanism (i.e. downgrading) is required.  This
   document describes requirements for downgrading, SMTP downgrading,
   Email header downgrading and implementation consideration.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-eai-downgrade-03.txt

To remove yourself from the I-D Announcement list, send a message to 
[email-masked] with the word unsubscribe in the body of 
the message. 
You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce 
to change your subscription settings.

Internet-Drafts are also available by anonymous FTP. Login with the 
username "anonymous" and a password of your e-mail address. After 
logging in, type "cd internet-drafts" and then 
"get draft-ietf-eai-downgrade-03.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-eai-downgrade-03.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Kari Hurtta <[email-masked]>
Date: 09 Mar 2007 07:03:45 +0200

>    Email header downgrading and implementation consideration.

|   other headers:
|      All other headers which contains UTF-8 characters are preserved in
|      Downgraded: header and removed.

Removing of MIME's  Content-Type -header field produces incorrect result.

Basically mime-type part (type/subtype) of that header-field must be preserved.
Also following mime paramaters need to be preserved: charset and boundary
These parameter values should be ASCII -only.  Also type/subtype value
should be ASCII -only.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 09 Mar 2007 07:18:25 +0200

>    Email header downgrading and implementation consideration.

| o  If each mail header has [RFC2047] encoded part and which encoding
|      is "UTF-8", it may be a downgraded header, so decode it.

That algorithm not necessary produce original result. 

But perhaps it is determined that sequence first downgrade and then upgrade
does not need produce original result ?

Because on downgrade  only header field which was RFC2047 encodes was:
|   Subject:
|      Encode the header by [RFC2047] with UTF-8 tag and replace it.

that upgrade rule can be:

| o  If Subject: header field has [RFC2047] encoded part and which encoding
|      is "UTF-8", it may be a downgraded header, so decode it.

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 09 Mar 2007 17:15:49 -0000

> These parameter values should be ASCII -only.  Also type/subtype value

Which just leaves possible <parameter>s with UTF-8 in theor <value>s. We  
should define that these are to be downgraded using RFC 2231.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 09 Mar 2007 17:18:56 -0000

> That algorithm not necessary produce original result.

What differences can arise? I can see that folding may get changed, and  
whitespace may get mucked about, but is there anything else?

Changes of folding should be acceptable (and will even lead to correct  
DKIM signature interpretation is the 'relaxed' canonicalization is used).

---

From: Kari Hurtta <[email-masked]>
Date: 09 Mar 2007 19:36:10 +0200

> What differences can arise? I can see that folding may get changed,
> and  whitespace may get mucked about, but is there anything else?
> 

Also header field is not necessarly downgraded header field.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 09 Mar 2007 20:21:11 +0200

>    Email header downgrading and implementation consideration.

chaprer "6.  Email header Downgrading"

|   Message-ID:, Date:, In-Reply-To:
|      "ID"s and "Date"s does not contain UTF-8 characters.

RFC 2822:

message-id      =       "Message-ID:" msg-id CRLF

msg-id          =       [CFWS] "<" id-left "@" id-right ">" [CFWS]

CFWS            =       *([FWS] comment) (([FWS] comment) / FWS)

draft-ietf-eai-utf8headers-03.txt:

   comment = "(" *([FWS] utf8-ccontent) [FWS] ")"

So I'm not completely sure that Message-ID: -header field
does not contain UTF-8 characters.

Same apply also to Date -header field.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: 09 Mar 2007 20:23:36 +0200

"Charles Lindsey" <[email-masked]> writes in gmane.ietf.ima:

> Which just leaves possible <parameter>s with UTF-8 in theor
> <value>s. We  should define that these are to be downgraded using RFC
> 2231.

And do not forget comments.

|   comment = "(" *([FWS] utf8-ccontent) [FWS] ")"
|
|   word    = utf8-atom / utf8-quoted-string
|
|   This means that all the RFC 2822 constructs that build upon these
|   will permit UTF-8 characters, including comments and quoted strings.

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 10 Mar 2007 17:21:40 +0100

Kari Hurtta wrote:

> chapter "6.  Email header Downgrading"

> |      "ID"s and "Date"s does not contain UTF-8 characters.

Yeah, there are tons of [CFWS] in 2822, and the "C" is for
<comment>.  Proposed fix:

-   Message-ID:, Date:, In-Reply-To:
-      "ID"s and "Date"s does not contain UTF-8 characters.
+   In-Reply-To:, Message-ID:, References:, Resent-Message-ID:
+      RFC 2822 <msg-id>s don't contain UTF-8 characters.
+   Date:, Resent-Date:
+      RFC 2822 <day-of-week>, <date>, and <time> as used in
+      a <date-time> don't contain UTF-8 characters.

Maybe we should limit our efforts to header fields relevant
for EAI (anything with an address), and stay from redefining
CFWS everywhere.

Quick shot:  We could introduce <IFWS> for all header fields
containing addresses, where the "I" is an "internationalized
comment".  Or better <UFWS> for an "UTF-8 comment".  Or maybe
that's a bad idea, but I certainly don't want any "UFWS" or
"IFWS" or UTF-8 in *_MIME_version_1.0_* part headers (i.e. in
any Content-* header fields of a message/utf-8 part).

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 10 Mar 2007 17:33:12 +0100

> Also header field is not necessarly downgraded header field.

In other words any valid 2321 or 2047 =?UTF-8...?= encoded word could
be encoded in the original message/utf-8 (before downgrading).  Then
"upgrading" it would replace the encoded word by native UTF-8.  And
that could cause havoc for header signatures.  But we know this, the
issue has to be noted somewhere (maybe as "security consideration").

After that folks living behind a downgrade + upgrade setup have to
deal with the potential side-effects, it's anyway a shaky setup.

Frank

---

From: Kari Hurtta <[email-masked]>
Date: 10 Mar 2007 20:02:22 +0200

RFC 2822:

obs-in-reply-to =       "In-Reply-To" *WSP ":" *(phrase / msg-id) CRLF

phrase          =       1*word / obs-phrase

draft-ietf-eai-utf8headers-04.txt:

   word    = utf8-atom / utf8-quoted-string

( or ... are I already sent this? )

/ Kari Hurtta

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 10 Mar 2007 20:15:02 +0100

Kari Hurtta wrote:

 [2822]

> phrase          =       1*word / obs-phrase

 [I-D.ietf-eai-utf8headers-04]
>    word    = utf8-atom / utf8-quoted-string

Introducing UTF-8 in any <obs-cenities> is a dubious plan.  Here's
another example in the same direction:

| obs-bcc         =       "Bcc" *WSP ":" (address-list / [CFWS]) CRLF

The "C" in CFWS again.  Getting a clean split between terms used (also)
in <obs-cenities> and terms used (only) in no-nonsense sections (ch. 3)
in ABNF would be a big effort.

We better solve it once and for all in prose, something like this:

| The revised ABNF specified here is not intended to be used in obsolete
| forms of the relevant header fields or terms defined in the Internet
| Message Format chapter 4 [RFC 2822].  As stated in [RFC 2822] these
| obsolete forms MUST NOT be generated.
|
| Implementations supporting message/utf-8 MAY accept UTF-8 in obsolete
| forms of header fields and terms where the corresponding revised ABNF
| specified here suggests it, but this is not required.  The statement
| in [RFC 2822] that all implementations MUST accept the obsolete forms
| does not affect the use of UTF-8 in these obsolete forms.

> ( or ... are I already sent this? )

Don't think so.  I've got a bunch (about six or eight) messages from you
where I first thought that they might be "courtesy copies" showing up on
the list later, but that didn't happen.  If that was supposed to be
private mail please mention "off list" somewhere.  I could also try to
forward these mails to the list, but you probably have your own archive
of "sent" mails (list, unintentionally off list, or really off list).

Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 11 Mar 2007 11:38:41 +0100

>    comment = "(" *([FWS] utf8-ccontent) [FWS] ")"

I suggest that this is best addressed by adding "except in comments".
Having two kinds of comments in UTF8SMTP headers is asking for even more 
trouble than UTF8SMTP is currently asking for.

                Harald

---

From: Kari Hurtta <[email-masked]>
Date: 12 Mar 2007 10:14:07 +0200

> deal with the potential side-effects, it's anyway a shaky setup.

Yes. That is why tress that IMAP and POP should provide method
for UTF8SMTP capable clients to access messages without automatic
upgrading -- ie. on they "original forms".

   From: Kari Hurtta <[email-masked]>
   Subject: Respawn "Messages on original form" (Re: I-D
	ACTION:draft-ietf-eai-imap-utf8-01.txt)
   Newsgroups: gmane.ietf.ima
   To: [email-masked]
   Date: 07 Mar 2007 19:41:00 +0200

When RFC Editor queue is open, I probably post draft which
also sidesteps this issue (i.e. Message Store requirements. )
( That should be 19.3.2007 if I remember correctly.)

/ Kari Hurtta

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 12 Mar 2007 11:16:30 -0000

> issue has to be noted somewhere (maybe as "security consideration").

Which implies that any RFC2047 stuff should be unscrambled before  
computing the hash for the signature. That would mean a different  
canonicalization algorithm, but if a special canonicalization algorithm is  
needed for DKIM-signing of UTF8SMTP messages, then that is not a  
showstopper.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 12 Mar 2007 11:21:10 -0000

> trouble than UTF8SMTP is currently asking for.

+1

I think it is well accepted that <comment>s and <phrase>s can contain
UTF-8 anywhere, and always get downgraded (except perhaps in Received
headers),

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 12 Mar 2007 11:24:59 -0000

+1

The obs-syntax was always intended, AIUI, for archives of ancient email  
messages that might exist. It should be seen on current any wire, and for  
sure there are no archives of ancient UTF8SMTP messages left over from the  
days before RFC 2822.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 12 Mar 2007 19:48:32 +0100

> canonicalization algorithm, but if a special canonicalization algorithm
> is needed for DKIM-signing of UTF8SMTP messages, then that is not a
> showstopper.

It's interesting for my "master plan" to keep all-ASCII in Content-*,
you can't "unscramble" it where that's forbidden in MIME version 1.0.

And _any_ 2047 (or even 2231) stuff isn't possible, implementations
would be forced to know _any_ charset.  And "unscrambling" 2231 would
remove the language tags, unless you intend to use the Unicode tags 
in plane 14 (officially deprecated).

This EAI effort is a can of worms, kill one, find three new.. :-(

Frank