Subject: Fw: [idn] IDNs in email message bodies

From: "Edmon" <[email-masked]>
Date: Mon, 26 Mar 2001 19:45:56 -0500

> > > as far as I'm concerned, RFC 2047 encoding is a non-starter for domain
> > > > names in protocol elements.  2047 was specifically designed to
encode
> > > > text whose only purpose was to be presented to a user; and it
> therefore
> > > > completely skirts all issues related to normalization,
> canonicalization,
> > > > etc.
> > >
> > > While I agree to a certain degree (within the protocol and
> > > transportation/routing), I would like to say that in the context of
this
> > > discussion where we are disucssing the "message headers" and "content
> body"
> > > of an email, the information is in fact for display purpose.
> >
> > no.  not only is this not "in fact", it's simply incorrect.
> > the addresses in the message header are intended both for display
> > and for use in forming replies.
> >
> > also, in practice, RFC 2047 encoded-words are routinely decoded,
> > translated, re-encoded, and/or corrupted by mail transports and
> > delivery systems.
> >
>
> Yes they are used for both display and for "forming" replies, and it is
> during this "forming" process where the ACE conversion should take place.
> The information itself within the field is for display and we should not
> confuse it with the actual protocol for transportation.  Just like typing
a
> domain name in the URL bar or copy and pasting into the URL bar, you
should
> see the multilingual domain there! and not be forced to type an ACE
domain.
> Edmon

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 27 Mar 2001 05:59:43 +0000

Edmon <[email-masked]> wrote:

> Yes they are used for both display and for "forming" replies, and it
> is during this "forming" process where the ACE conversion should take
> place.  The information itself within the field is for display and we
> should not confuse it with the actual protocol for transportation.

When someone replies to a message, the MUA extracts the email address
from the From: field, and hands this address to the MTA (which is often
built into the MUA), which then separates the domain from the local
part and does a DNS lookup on the domain.  If the MUA and MTA are not
IDN-aware, and the address in the From: field is a non-ACE IDN, then
this lookup will fail.

The header fields are not just for display.  Message headers are
primarily a protocol used by MUAs to communicate with other MUAs, just
as SMTP is a protocol used by MTAs to communicate with other MTAs, and
the message body is for the humans to communicate with other humans.
(Exceptions:  Several of the header fields, like From:, are of interest
to humans in addition to MUAs, and the Subject: field is only for the
humans.  The Received: fields are primarily for the MTAs.  MIME gets the
MUAs more involved with the message body.)

AMC

---

From: "Edmon" <[email-masked]>
Date: Tue, 27 Mar 2001 12:10:23 -0500

> built into the MUA),

Are you saying that it doesnt get displayed first on the user interface?
How then could I add additional recipients to the "reply" mail?

 which then separates the domain from the local
> MUAs more involved with the message body.)

But if we are talking about an IDNA approach, "A"pplications or "A"gents are
assumed to have to change.  I would not expect that a non IDNA aware browser
to be able to resolve an IDNA name even if it was cut and pasted from an
IDNA aware MUA...

Edmon

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 27 Mar 2001 12:49:07 -0800

> from the From: field, and hands this address to the MTA (which is
> often built into the MUA), which then separates the domain from the
> local part and does a DNS lookup on the domain.  If the MUA and MTA
> are not IDN-aware, and the address in the From: field is a non-ACE
> IDN, then this lookup will fail.

But this specific scenario won't fail due to ACE, it will fail due to the
mailer not conforming with the standards which were already in place
(ASCII only in domain names). In this case, the original sender (which the
reply is being directed to) should not have specified their address as a
charset-encoded IDN, but instead should have written it as a charset@ASCII
sequence and rejected anything that wasn't @ASCII.

Going forward, it seems that the best way to handle these particular
fields is to promote ACE-encoded domain names to the same level as charset
encoding (although they of course have different mechs).

For example, if a user enters <JPuserid>@<JPdomain.dom> then that should
be encoded as <iso-2022-jp>@<ACE-encoding> on message SEND, with the
mailer doing an ACE conversion on the domain part at the same time as it
does an RFC-2047 conversion on the user part.

Applications that display messages to users should know that they can
reverse the encoded data for display just like they would do with RFC-2047
encoded headers, but they have to generate the right data when they craft
up the headers.

Newer apps which are aware of these rules will have to do this if they
want to present a rich experience to the user. Legacy apps should not
allow users to enter these domain names into the relevant fields, or they
should barf if such a name is encountered on load/send. If they are
accepting illegal domain names that is their problem; they shouldn't be
breaking the rules they know about, which is @ASCII only.

More problematic is extra-config data, such as a hand-coded Reply-To
header field in the message, as this data will bypass the mailer's
boundary checks. Not much to say about that, other than to point out that
it could happen and that it should commonly be treated as message data and
therefore covered by charset header. Any conversion that needs to be
applied to this data must be applied when the data is used to send a
message, and should not converted to ACE on behalf of the user prior to
that point.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 27 Mar 2001 21:39:45 +0000

Edmon <[email-masked]> wrote:

> But if we are talking about an IDNA approach, "A"pplications or
> "A"gents are assumed to have to change.  I would not expect that a non
> IDNA aware browser to be able to resolve an IDNA name even if it was
> cut and pasted from an IDNA aware MUA...

This explains why we've been having trouble understanding each other.
As I see it, one of the biggest reasons to use ACE is that it allows
someone with today's non-IDN-aware software to reply to a message from
someone with an IDN-address.  I don't see how else you're going to
deploy IDN.  If people try out the new & improved IDN stuff and discover
that some of the messages they send cannot not be replied to, a lot of
them will probably give up on the IDN stuff and go back to using what
worked.

"Eric A. Hall" <[email-masked]> wrote:

> > If the MUA and MTA are not IDN-aware, and the address in the From:
> > field is a non-ACE IDN, then this lookup will fail.
>
> But this specific scenario won't fail due to ACE, it will fail due to
> the mailer not conforming with the standards which were already in
> place (ASCII only in domain names).

Yes, I totally agree.  I'm trying to convince Edmon that domain names in
message headers need to be ACE.

> In this case, the original sender (which the reply is being directed
> to) should not have specified their address as a charset-encoded IDN,
> but instead should have written it as a charset@ASCII sequence and
> rejected anything that wasn't @ASCII.

Well, it's not clear whether the local part should use RFC 2047
encoded-words.  RFC 2047 forbids the use of encoded-words in any part of
an addr-spec.  Encoded-words are designed for parts of the headers that
are strictly for human consumption, not the machine-readable parts.  ACE
might be better choice for the local part of email addresses (but with a
slightly different nameprep that allows all ASCII characters).

> boundary checks.

I don't see why.  According to RFC 822 the Reply-To: field has the exact
same syntax as the To: field, and MUAs routinely do syntax checking on
the To: field, so they should do the same checking on the Reply-To:
field.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 27 Mar 2001 14:21:45 -0800

> Encoded-words are designed for parts of the headers that
> are strictly for human consumption, not the machine-readable parts.

Yeah, should have read "2047-encoding" <ascii@ACE-encoding>

The point is still the same, ACE-encoding/decoding can occur at the same
time as 2047-encoding/decoding, when the display data is converted into
protocol data.

> I don't see why.  According to RFC 822 the Reply-To: field has the
> exact same syntax as the To: field, and MUAs routinely do syntax
> checking on the To: field, so they should do the same checking on
> the Reply-To: field.

I am thinking about command line tools and automated mail generators which
will not view Reply-To as a protocol operation.

---

From: Keith Moore <[email-masked]>
Date: Tue, 27 Mar 2001 18:18:00 -0500

> does an RFC-2047 conversion on the user part.

no, that would be insane.

1. as already stated, RFC 2047 frequently gets munged by things that
   handle mail.  this would corrupt the local-part of the address

2. as already stated, there is no canonical form of RFC 2047 encoding -
   the same string can be encoded in an arbitrary number of ways.  And
   SMTP servers aren't equipped to decode RFC 2047 before comparing
   a recipient address with a list of valid addresses.
 
the requirements for local-parts of email addresses are quite similar
to those of domain names.  ACE is much closer to this than RFC 2047.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 27 Mar 2001 18:32:35 -0500

> time as 2047-encoding/decoding, when the display data is converted into

there are two problems with this statement.

the first is that address processing is done at many different places,
not all of which are concerned with presentation.  an SMTP server might
need to do ACE decoding on addresses independently of any user agent.

the second is that in many current implementations rfc 2047 is done at the 
wrong time - a message is RFC 2047 headers is often "converted" to 
use raw characters at some point prior to presentation, and those
raw characters often leak out in replies, forwarded messages, etc.

I'd agree that if a user agent were implemented properly, it would decode
rfc 2047 and ace domain names *for the purpose of presentation* at the 
same time - but this might not be the same time that rfc 2047 is decoded
by a current implementation of that user agent.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 27 Mar 2001 15:42:11 -0800

> > The point is still the same, ACE-encoding/decoding can occur at the
> > same  time as 2047-encoding/decoding, when the display data is
> > converted into protocol data.
> 

Why is that a problem?

> the second is that in many current implementations rfc 2047 is done
> at the wrong time - a message is RFC 2047 headers is often "converted"
> to use raw characters at some point prior to presentation, and those

I expect that ACE leakage will be a problem forever, just as 2047 leakage
is going to be a problem forever. Not a major problem though if the
addresses are verified when the envelope is created. If an implementation
isn't following the old rules there's not much we can do about it.

> I'd agree that if a user agent were implemented properly, it would
> decode rfc 2047 and ace domain names *for the purpose of presentation*
> at the same time - but this might not be the same time that rfc 2047
> is decoded by a current implementation of that user agent.

Agreed. For the purpose of message composition, address books and the
like, the IDN representaionn should be the visible form (just like 2047)
while the underlying mechs and structures will use ACE (just like 2047).