Subject: [EAI] A collision free ACE for internationlized email addresses

From: "Chris Walker" <[email-masked]>
Date: Mon, 19 Nov 2007 12:26:20 -0700

So many examples that purport to demonstrate why an internationalized 
email address can’t be ACE encoded have relied on examples that are 
really only encodings of the local PART. Why did I capitalize part, to 
reinforce the concept that the local part is part of an email address, 
the other side of the @ sign, of course with the domain part, together 
they constitute an email address.

Due to vulgarities of technology, and long standing tradition that IETF 
documents minimally appear in text, I’ll assume 7bit text;  I will use 
examples with short strings of internationalized characters represented 
in U+hex notation.  Additionally, local parts, even in UTF8 examples 
will be quoted and contain a period character.

First what can NON internationalized email addresses look like:
"a.b"@a.b.com
"a.b"@[192.168.44.239]  (non routable IP chosen for example)
"a.b"@[2001:0db8:0000:0000:0000:0000:1428:57ab]  (non zero collapsed 
IPv6)
and non internationalized email addresses with internationalized domain 
parts:
"a.b"@ U+90F5 U+4EF6 . U+5546 U+52D9  (actual UTF8 characters 
represented by U+hex)
"a.b"@ xn--5nqv22n.xn--lhr59c  (same domain part as above, in ACE form)

All of the above addresses are capable of transmission over STMP, the 
internationalized domain one, only in its ACE form. Whether or not these 
addresses will be accepted, understood by the email server, or displayed 
properly is NOT the point here, they are, capable of being transmitted 
over the SMTP protocol by virtue of being in 7bit ASCII or having a 7bit 
ASCII representation.

Now much has been said on the topic of it not being possible to 
represent internationalized email addresses in an ACE form due to the 
extreme flexibility of what is considered valid or legal in the local 
part of existing email addresses.  In other words, there is no way to 
encode a local part into an ACE that might not be a valid existing 
address already.

However, encoding just the local part is not really encoding an 
internationalized email address is it? It is only encoding part of one 
part of it, the local part. What happens if we attempt to encode the 
entire address?

The following are examples of internationalized email addresses, same as 
above with UTF8 local parts.
"U+4EF6 . U+5546"@a.b.com
"U+4EF6 . U+5546"@[192.168.44.239]  (non routable IP chosen for example)
"U+4EF6 . U+5546"@[2001:0db8:0000:0000:0000:0000:1428:57ab]  (non zero 
collapsed IPv6)
"U+4EF6 . U+5546"@ U+90F5 U+4EF6 . U+5546 U+52D9  
"U+4EF6 . U+5546"@ xn--5nqv22n.xn--lhr59c  (same domain part as above, 
in ACE form)

Can these be encoded in such a way that the resulting encodings cannot 
possibly be valid email addresses; in order to prevent the accidental 
delivery of an email to the wrong party should a system attempt to use 
them in their encoded form?

Encodings need an identifier to make them readily recognizable as 
encodings, I will choose an identifier that is similar to the one used 
for punycode ACEs, with a twist, it will be a trailing the encoded 
string instead of leading it. Why, because the characteristic of the 
punycode IDN ACE that makes it an invalid non encoded domain name the -- 
characters, will appear on the domain side of the encoded email address 
and will make the resulting encoding invalid as a non encoded email 
address.  Encoding is punycode.

So the list of addresses above with their encoded representations are:
Original: "U+4EF6 . U+5546"@a.b.com
Encoded: "."@a.b.com-x00rr07d--yn
Decoded: "U+4EF6 . U+5546"@a.b.com

Original: "U+4EF6 . U+5546"@[192.168.44.239]  (non routable IP chosen 
for example)
Encoded: "."@[192.168.44.239]-wl85a2z4g--yn
Decoded: "U+4EF6 . U+5546"@[192.168.44.239]

Original: "U+4EF6 . U+5546"@[2001:0db8:0000:0000:0000:0000:1428:57ab]  
(non zero collapsed IPv6)
Encoded: "."@[2001:0db8:0000:0000:0000:0000:1428:57ab]-1i86ed23j--yn
Decoded: "U+4EF6 . U+5546"@[2001:0db8:0000:0000:0000:0000:1428:57ab]

Original: "U+4EF6 . U+5546"@ U+90F5 U+4EF6 . U+5546 U+52D9  (actual UTF8 
characters represented by U+hex)
Encoded: "."@.-mn2hd570ftyfea0819n--yn
Decoded: "U+4EF6 . U+5546"@ U+90F5 U+4EF6 . U+5546 U+52D9  

Original: "U+4EF6 . U+5546"@ xn--5nqv22n.xn--lhr59c  (same domain part 
as above, in ACE form)
Encoded: "."@ xn--5nqv22n.xn--lhr59c -zr98bg78--yn
Decoded: "U+4EF6 . U+5546"@ xn--5nqv22n.xn--lhr59c  

You get out of the encoding exactly what you put into it. In all cases 
internationlized email addresses are input, the resulting encodings have 
no possibility of being existing email addresses, and when decoded, the 
result is identical to the original.

So what does this mean?
a)	It is possible to provide an ASCII Compatible Encoding (ACE) for 
internationalized email addresses.
b)	The resulting ACE is detectable by a single rule, that is it 
ends in "--yn"
c)	The encoded forms cannot be existing email addresses.

At this point, this provides a perfectly valid encoding.  And if any of 
the following is disregarded, this should be considered for inclusion as 
a suggested part of the standard.

What else does the above imply? Well, the two main arguments that were 
used to support UTF8SMTP are that no encoding is possible, and that is 
an encoding were possible, the leaking, that is the encoding being seen 
by users; users would find this, quoting RFC 4952, discomfiting or 
astonishing.

Well, UTF8SMTP provides no lack of discomfiting or astonishing 
features.  The downgraded headers that preserve the original UTF8 email 
address provide their own encoding and ‘leaks’ to the user. I, also 
being a user, find the entire alt address concept, having to have them 
and knowing when and where to use them,  both discomfiting and 
astonishing in the extreme.

The entire UTF8SMTP design, and all of its baggage, serves no purpose 
than to accommodate UTF8 characters of the left side of the @ sign in 
email addresses.

UTF8SMTP fails on both leaking and by being discomfiting and astonishing.

A different EAI design that uses the ACE form above where email 
addresses appear in SMTP headers should be considered. Such a design has 
the potential to eliminate alt addresses and downgrading. Such a design 
would not be transparent to existing email servers, they would need to 
be aware of the encoding, just as they would need to be aware of 
internationalized domain name encoding to resolve names using DNS.

After going down the existing UTF8SMTP design path far enough to see its 
implications, all I ask is that you take some time to reconsider if its 
complexity is worth it consideriing it is lacking on meeting design 
goals.

I am sure the first argument I will hear over this idea has something to 
do with leaking, as I already pointed out, UTF8SMTP already leaks.

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Yangwoo Ko <[email-masked]>
Date: Tue, 20 Nov 2007 10:04:51 +0900

Chris Walker wrote:
> (snip)
> (snip)

Dear Chris Walker,

Please apologize my misunderstanding if (m)any. The most important 
virtue of ACE(-like) approaches is that it is an end-to-end approach. No 
intermediate SMTP server needs to be EAI aware. No protection mechanism 
(such as UTF8SMTP ESMTP extension) or in-transit manipulatio (such as 
downgrading) is required.

If an SMTP server, which is not aware of the proposed encoding rule, 
receives "."@a.b.com-x00rr07d--yn, then it will try to get MX record(s) 
of RHS of @ sign. But, it will certainly fail. What am I missing?

 >(snip)
 > I am sure the first argument I will hear over this idea has
 > something to do with leaking, as I already pointed out,
 > UTF8SMTP already leaks.
 > (snip)

In order to make the proposed encoding to work, all involved SMTP 
servers should be updated to be aware of this new encoding rule. If all 
involved SMTP servers are assumed to fully support current EAI WG's 
approach, then there will be no leak at all because everything will be 
in UTF8.

Regards

---

From: "Chris Walker" <[email-masked]>
Date: Tue, 20 Nov 2007 04:23:21 -0700

-----Original Message-----
From: Yangwoo Ko <[email-masked]>
To: Chris Walker <[email-masked]>
Cc: eai list <[email-masked]>
Date: Tue, 20 Nov 2007 10:04:51 +0900
Subject: Re: [EAI] A collision free ACE for internationlized email 
addresses

> of RHS of @ sign. But, it will certainly fail. What am I missing?

At some point close to the sender an internationalized email address 
would need to be encoded for transit over SMTP and at the recipients 
side it would need to be decoded for display. It is possible that this 
could be done in the mail client software used by the users. These mail 
clients may or may not be directly integrated with the SMTP servers that 
start and finish the actual transport over SMTP.

The address needs to be decoded to access the domain part in order to 
determine the MX record in DNS.

This burden of having to be aware of encodings is not unique to this 
encoding,  if the domain part is in UTF8, a system has to have knowledge 
of the proper methods (INDA ToASCII) in order to convert the UTF8 domain 
part in order to query the DNS. All the records in DNS for 
internationalized domains are stored in punycode form. 

In both scenarios, knowlegde of a encoding format is needed, to 
determine (via DNS MX lookup) which mail server needs to be contacted, 
however after that determination is made, the encoded form will fit 
within the restrictions of SMTP be 7bit ASCII.

---

From: "Chris Walker" <[email-masked]>
Date: Tue, 20 Nov 2007 04:52:25 -0700

-----Original Message-----
From: Yangwoo Ko <[email-masked]>
To: Chris Walker <[email-masked]>
Cc: eai list <[email-masked]>
Date: Tue, 20 Nov 2007 10:04:51 +0900
Subject: Re: [EAI] A collision free ACE for internationlized email 
addresses

> approach, then there will be no leak at all because everything will be 

Im sorry, I musunderstood your question to be something else, please 
disregard my first reply.

The 'downgrading' process, is where a UTF8SMTP email is downgraded to 
SMTP, part of this process as it is currently written is to to preserve 
the orignial UTF8 headers with UTF8 email addresses in downgraded 
headers, these headers are encoded to make the data 7bit compatable.

Chris

---

From: John C Klensin <[email-masked]>
Date: Tue, 20 Nov 2007 12:31:03 -0500

--On Tuesday, 20 November, 2007 04:23 -0700 Chris Walker
<[email-masked]> wrote:

> At some point close to the sender an internationalized email
> address  would need to be encoded for transit over SMTP and at
> the recipients  side it would need to be decoded for display.
> It is possible that this  could be done in the mail client
> software used by the users. These mail  clients may or may not
> be directly integrated with the SMTP servers that  start and
> finish the actual transport over SMTP.

But, Chris, if you include the domain-part in the encoding --
which is what I now understand you want to do-- then even relay
in the path will have to do the decoding in order to know where
to forward the mail.  That also implies some very nasty bounces,
as every (non-submission) SMTP server in the world today knows
that address syntax contains a (public/ non-encoded) "@", which
precedes an FQDN.

Sorry, I just don't see how your ideas work in the real world,
at least as I understand it.

   john

---

From: John C Klensin <[email-masked]>
Date: Tue, 20 Nov 2007 12:59:06 -0500

--On Tuesday, 20 November, 2007 10:04 +0900 Yangwoo Ko
<[email-masked]> wrote:

> If an SMTP server, which is not aware of the proposed encoding
> rule, receives "."@a.b.com-x00rr07d--yn, then it will try to
> get MX record(s) of RHS of @ sign. But, it will certainly
> fail. What am I missing?

That it is worse than that.   We know that, in the real world,
quotes often get lost, even when they shouldn't.   If this is
accidentally turned into 
    [email-masked]
either between relays or in local processing, it is likely to be
rejected as a syntax error or as a broken attempt to specify a
source route.

One can certainly handle these things in a sufficiently upgraded
server, protected by SMTP extension options that keep other
addresses from getting anywhere near it.  That upgrade would
also require (incompatible) header changes to note that
something is an i18n address and not an ASCII one.     But those
forms are not, in practice, going to be reliably transparent to
legacy servers  or legacy MUAs.  It seems to me that was your
main point, with which I agree.  

And, if protection by an SMTP extension option is required, then
there is little advantage over using some flavor of ACE over the
straightforward transmission of UTF-8.  The latter also makes
subaddressing and routing and other actions embedded in the
local-part more feasible, doesn't leave us with an
"ASCII-plus-kludge" header and envelope construct that we have
to live with forever, even when EAI is widely deployed.  And so
on.

Of course, that was the conclusion we reached well over a year
ago, based on almost exactly the same knowledge and reasoning
that we have and are going through today.

Chris, the question about whether Alice (or Aiko) will be able
to find an appropriate domain for a Devanagari-script domain is
(i) there _is_ an ACE for domain name labels  and, more
important, (ii) if there is a market demand for such names on a
global basis, the names will emerge.  If not, then it is not as
much of a problem as you assume.   It is suggestive of one of
the flaws in the model of TLDs that are tied to languages and
presumably administrative unavailable to people who are not
normally associated with that language community, but that isn't
this WG's (or the IETF's) problem.

The bottom line is that we have a choice between:

(1) Keeping the world nice for those with applications that are
all-ASCII and ignorant of, or hostile to, other scripts, even
Latin characters outside the ASCII/ISO-646IRV subset.   Doing
this means that those other scripts will always look like a
clumsy add-on.

(2) Moving to where we would like to end up, in which the email
addressing environment is largely script-agnostic.

Now the latter does imply some transition activities. 

Organizations that have assumed that the whole world is ASCII
and will be ASCII forever will either need to adapt to the new
address formats or will need to reject them as invalid.  For
better or worse, the latter is pretty much what happens today:
we continue to have problems with web sites that accept email
addresses with local parts that contain "strange" characters
such as "+" or "=" and with domain names where the TLD is more
than three or four characters long.   And people who communicate
across language communities will either need to stick with ASCII
addresses (perhaps as aliases) or become clever.   We may open
up some market opportunities for address translation and
forwarding databases --something else we have been discussing
since the beginning-- if sender authentication techniques that
are hostile to forwarding don't get in the way.

It is likely to be bumpy for a while, especially for any
communication that crosses between language communities.  We are
worrying about downgrading precisely to ease the bounce from
some of those bumps.   But it will sort itself out and we will
end up with an internationalized email system.  If it doesn't,
the SMTP option and extended syntax will all fall back out of
use, resulting in less damage than if we had attempted a complex
ACE that turned out to be not-quite-transparent.   

The first option, by contrast, amounts to an assertion that
ASCII is supreme and that every other script is inherently
second-class (or worse) and should stay that way.   I don't find
that acceptable even though I can write most of my native
language in ASCII.   For others...

     john

---

From: Martin Duerst <[email-masked]>
Date: Thu, 22 Nov 2007 11:34:44 +0900

>this WG's (or the IETF's) problem.

Hello John,

I have had difficulties parsing/understanding the last sentence
in your paragraph. Could you explain (probably just giving an
example may be enough).

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 22 Nov 2007 07:55:51 +0100

Martin Duerst wrote:

>> It is suggestive of one of the flaws in the model of TLDs that
>> are tied to languages and presumably administrative unavailable
>> to people who are not normally associated with that language 
>> community, but that isn't this WG's (or the IETF's) problem.
[...]
> in your paragraph. Could you explain (probably just giving an

When I created http://xn--80akhbyknj4f.boldlygoingnowhere.org
I didn't bother to discuss this creation with PIR (TLD org)
or DynDNS (boldlygoingnowhere), possibly I'm not "allowed"
to create Cyril IDNA labels in this TLD.

 Frank

---

From: John C Klensin <[email-masked]>
Date: Fri, 23 Nov 2007 14:44:03 -0500

--On Thursday, November 22, 2007 11:34 AM +0900 Martin Duerst 
<[email-masked]> wrote:

>> It is suggestive of one of
>> normally associated with that language community, but that
>> isn't this WG's (or the IETF's) problem.
>
> Hello John,
>

Sorry.  ICANN problem, not an IETF one.

There have been several suggestions to create language-specific 
TLDs and restrict all registrations and use of those domains 
(all the way down the tree) to labels in the specific language 
represented by the TLD and, in many cases, to speakers of those 
languages authorized by the countries that see themselves as 
controlling them.  There have even been some suggestions that 
the DNS's matching rules should change based on the language (as 
defined by the TLD).

There are, to put it mildly, several issues with those ideas. 
The DNS doesn't know anything about language and most labels are 
not "words" in any language.   In a distributed administrative 
hierarchy, it is not clear how such rules would be enforced if 
one wanted to enforce them.   The "different matching rules" 
idea is just not feasible without radical changes to the DNS 
itself.  The question of who could make a sufficient claim to 
ownership of a language to control the TLD specific to that 
language boggles my mind.

An interpretation of Chris's point about having all of the 
domain name in a the same language (or script) as the local-part 
would require that someone obtain a domain name for a mail 
server in each relevant language.   That isn't really what I was 
thinking about -- I was more concerned about the local-part than 
about the domain name.   But, if one did somehow insist that the 
entire address be in the same script, one might have to 
negotiate with the appointed owners of the language-TLD for a 
registration.   The difficulties that implies strike me as 
another argument against language-specific TLDs, rather than one 
against the EAI model.

But this is getting very far off the WG's mission and charter. 
Speaking of that charter, this is probably the right time to 
remind people that EAI was chartered to explore one very 
specific approach, leading to Experimental, not Standards-track 
documents.  That approach includes a "no ACE in addresses" 
assumption, independent of whether that assumption is logically 
necessary given the base email protocols (as I believe it to 
be).  If Chris (or others) want to explore a different approach, 
e.g., to see how far an ACE approach can be pushed and what 
constraints it would imply, then I believe he or they should be 
producing an I-D and looking for BOFs and/or a WG, not trying to 
refine such a proposal in snippets on this list.    That is just 
opinion, of course.

    john

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 23 Nov 2007 23:22:14 +0100

On Tue, Nov 20, 2007 at 12:59:06PM -0500,
 John C Klensin <[email-masked]> wrote 
 a message of 104 lines which said:

> addresses with local parts that contain "strange" characters such as
> "+" or "="

Unfortunately true. The warning in section 6.3 of RFC 4952 is very
accurate, alas.

Speaking about this issue, two references:

* a new I-D which tries to standardize these "sub-addresses". Not sure
that it is a good idea, I did not read it yet. 
        Title           : Internet Email Subaddressing
        Author(s)       : D. Cridland
        Filename        : draft-newman-email-subaddr-01.txt
        Pages           : 12
        Date            : 2007-11-16

* A very good article of Douglas Lovell in Linux Journal
(http://www.linuxjournal.com/article/9585), "Validate an E-Mail
Address with PHP, the Right Way". To quote it: "There is some danger
that common usage and widespread sloppy coding will establish a de
facto standard for e-mail addresses that is more restrictive than the
recorded formal standard."

---

From: SM <[email-masked]>
Date: Fri, 23 Nov 2007 14:48:12 -0800

>         Filename        : draft-newman-email-subaddr-01.txt

See the Acknowledgements section of that draft. :-)

Regards,
-sm