Subject: [EAI] Additional Clarification re: Need of an ACE for EAI

From: "Chris Walker" <[email-masked]>
Date: Tue, 13 Nov 2007 09:58:08 -0700

Additional Clarification re: Need of an ACE for EAI.

Having read the EAI working groups drafts, the Internet Mail Consortiums 
draft (prior to its dissolution in this process) and the EAI working 
group’s message archive going back to summer of 2006…

There were at several points in time in the past, where there were 
proposals to use ACE representation of the local part as a MEANS for 
implementing internationalized email addresses. That is, encode Unicode 
to ASCII at or very near to the user interface level, and transmit the 
ACE over SMTP then convert the ACE back to Unicode at or near the 
recipients user interface. This method for IMPLEMENTING the transmission 
of email has died, among the earliest and loudest arguments that this is 
not acceptable are:

The ACE representations will leak, as RFC 4952 now states

QUOTE If non-obvious encodings, such as protocol-specific ASCII-
Compatible Encoding (ACE) variants, are used,   the user will 
inevitably, if only occasionally, see them rather than "native" 
characters and will find that discomfiting or astonishing. ENDQUOTE

And the argument that interpreting the local part was only in the domain 
of the system(s) in the domain part of the email address, also from RFC 
4952 referencing RFC 2821 et al.

QUOTE This is the SMTP server that controls the format of the local 
parts of addresses and is permitted to inspect and interpret them. 
ENDQUOTE

So instead of using an ACE, UTF8 Unicode email address are being 
proposed to be sent end to end, STMP obviously does not support that and 
that is the reason for the UTF8SMTP extension.

------------------------------------------------------------------------

I am not arguing that any of the decisions of the EAI working group that 
comprise the transmission of emails over email protocols need to support 
encoded local parts.

------------------------------------------------------------------------

What I am arguing is that as a result of not taking an ASCII encoding 
route, and by taking the UTF8 over the wire route, the following is the 
result of that decision.

There is no STANDARD for systems outside of email protocols that cannot 
support Unicode to use for the encoding of internationalized email 
addresses. 
 
If ACE’ing of the local part was used as the method for email 
transmission, then (as in the case of IDNA) the ACE representation would 
be an obvious choice for use in ASCII systems. However there is no 
ACE’ing of the local part, so there is no obvious standard for ASCII 
systems.

------------------------------------------------------------------------

Consider the following scenario, your company and another company that 
is your business partner exchange information using text files as part 
of some business processes. Let’s say for the sake of this scenario that 
the company that sends the file takes a customer order for some product, 
and your company fills the order and ships it. Part of this data is the 
customer’s email address.

When internationalized domain names came along, if there was an email 
address for a customer that had an internationalized domain name, no 
problem, the punycode ACE version of the domain part is sent in the text 
file.

Now along comes internationalized email addresses, now you have a 
customer with Unicode in the local part of their email address, how do 
you put that into your ASCII text file? Is there a standard for doing 
that? You scratch your head for a while and decide to see what the IETF 
says, well they don’t say anything on the issue.  Hmmm, now what? I 
guess you and your business partner can come up with some encoding of 
your own.  Things are working again.

What is wrong with the above scenario?

Since there is no standard defined, the two companies came up with their 
own encoding method.  Every set of companies that fit into the above 
scenario can do the same, lots and lots of companies all deciding 
different ways to represent internationalized email addresses in ASCII.

Companies A and B decide on using base64, companies C and D decide on 
using punycode, companies E and F decide on using escaped octets. 
Compaies G an H decide on punycode except they use nameprep. Companies I 
and J encode the entire email address, Companies K and L encode only the 
local part and use ACE punycode on the domain part. Some companies have 
different business units that each came up with their own encoding, 
these companies can't even exchange meaningful email address within 
their own company.

There is now NO STANDARD, none of these sets of companies can 
communicate internationalized email addresses effectively, numerous 
encoding methods are applied to email addresses where systems interact, 
such as in databases, with no indication of what encoding system was 
used for any particular email address. 

How on earth did this happen, who is responsible for this mess?

If the IETF EAI working group cannot ‘spend’ a couple paragraphs to 
describe some standard, ANY standard for encoding internationalized 
email addresses in ASCII, they will be.

If you do not think that providing some informational standard ACE for 
the local part is in the domain of the EAI working group, then I ask: 
Which working group do you think is more closely related to this issue 
and should provide the Internet community guidance on this matter?

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 13 Nov 2007 12:11:02 -0500

> support Unicode to use for the encoding of internationalized email 

In my opinion, to the extent that is true (and I note there is in fact
a fallback mechanism proposed -- see below), it's a feature.  In my experience, the
ability of non-IDNA-aware user agents (not to mention users!) to see
the "Punycode" version of the domain turns out to be a giant problem
for the non-geek set.  Some of them would rather the domain _not work_
than that people get a possibly confusing bunch of ASCII.

Explanations of why this backward compatibility was a good thing
have tended to elicit blank stares or else angry denunciations about
how computer geeks don't care about "real usability" in "the real world".

> you put that into your ASCII text file? Is there a standard for doing 

Isn't this what the ALT-ADDRESS is for?

A

-- 
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<[email-masked]>                              M2P 2A8
jabber: [email-masked]                 +1 416 646 3304 x4110

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 13 Nov 2007 19:11:42 +0100

> ACE’ing of the local part, so there is no obvious standard for ASCII 
There's an argument to be made that the choice of the ACE mechanism (if 
any) actually depends strongly on the context of the usage.

For instance, if we ever get around to defining a Mailto:-like IRI, 
which will then support the UTF-8 encoding of EAI's mailbox names, then 
by the rules of converting IRIs to URIs, the logical "ACE" format for 
the corresponding mailto: URI would be the percent-escaped, hex-encoded 
string that corresponds to the UTF-8 byte sequence.

I'll be very surprised if I *ever* encounter a business card with such 
an address upon it, just as I've never (so far) seen a business card 
with the Punycode form of a domain name on it.

I expect people with two-sided business cards to continue to have an 
all-ASCII email address on the side that has their all-ASCII postal address.

Harald

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 13 Nov 2007 17:19:47 -0200

On Tue, Nov 13, 2007 at 09:58:08AM -0700,
 Chris Walker <[email-masked]> wrote 
 a message of 121 lines which said:

> Consider the following scenario, 

Let's use a real case. ICANN just announced the implementation of the
Registrar Data Escrow system:

http://www.icann.org/announcements/announcement-2-09nov07.htm

The technical documentation for the ICANN registrars states:

4.1.1 Escrow records shall be compiled into a single (uncompressed) CSV
      text file or multiple (uncompressed) text files approximately 1 gigabyte
      or one million rows in size, in compliance with RFC 4180                
      <http://tools.ietf.org/html/rfc4180>. In accordance with RFC 4180, the
      character encoding for the CSV file should be US-ASCII, although      
      UTF-8 is also permissible.                                      

If a technical or administrative contact has an internationalized
email address, the only way is to send UTF-8 (which is only
"permissible"). At the present time, we have no standard way to encode
the internationalized address in US-ASCII so the registrar cannot
comply with the "should be" above.

[This is just an example of a real-life case. It is not an opinion.]

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 13 Nov 2007 17:27:08 -0200

On Tue, Nov 13, 2007 at 12:11:02PM -0500,
 Andrew Sullivan <[email-masked]> wrote 

> Isn't this what the ALT-ADDRESS is for?

The way I understand it, ALT-ADDRESS is a human-made alternative, that
has to be configured by the user, not an encoding which can be done
algorithmically.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 13 Nov 2007 21:39:05 +0100

> comply with the "should be" above.
Huh?

RFC 4180 says absolutely nothing about how email addresses are represented.

If you want UTF-8 in email addresses, you have to use an UTF-8 encoding 
of the CSV file. Just as you have to do if you have non-ASCII characters 
in the postal addresses.

What's new here?

Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 14 Nov 2007 11:19:32 -0000

On Tue, 13 Nov 2007 18:11:42 -0000, Harald Alvestrand  
<[email-masked]> wrote:

>> addresses.  If ACE’ing of the local part was used as the method for  
>> email transmission, then (as in the case of IDNA) the ACE  
>> representation would be an obvious choice for use in ASCII systems.  
>> However there is no ACE’ing of the local part, so there is no obvious  
>> standard for ASCII systems.
>>
>>

+1

Current mailto URIs (even if largely percent encoded) are a sufficient  
means of conveying the I18N address within other documents. Or else just  
use charset=UTF-8 when sending those documents.

So the problem does not lie in conveying the global address of the  
customer from the company in Hong Kong to the manufacturing company in the  
USA. The problem arises when the manufacturing company in the USA needs to  
send an email to the customer in Hong Kong. Agreed there is no obvious  
solution to that if the USA company is not willing or able to implenent  
EAI. But one simple and possible solution is for the Hong Kong company to  
set up an automatic forwardng system, so the USA encapsulates the message  
in some agreed format, and the Hong Kong company (which surely _has_  
implemented EAI) unwraps it and sends it on.

But yes, maybe we should be looking at other solutions to this particular  
problem.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 14 Nov 2007 09:31:20 -0200

On Tue, Nov 13, 2007 at 09:39:05PM +0100,
 Harald Alvestrand <[email-masked]> wrote 

> RFC 4180 says absolutely nothing about how email addresses are

I never talked about that. Please forget RFC 4180, the important part
in the ICANN documentation was:

      the

> If you want UTF-8 in email addresses, you have to use an UTF-8 encoding of 
> the CSV file. 

See the sentence above.

---

From: John C Klensin <[email-masked]>
Date: Tue, 13 Nov 2007 19:12:50 -0500

--On Tuesday, 13 November, 2007 09:58 -0700 Chris Walker
<[email-masked]> wrote:

> Having read the EAI working groups drafts, the Internet Mail
> Consortiums  draft (prior to its dissolution in this process)
> and the EAI working  group's message archive going back to
> summer of 2006…
> 
> There were at several points in time in the past, where there
> were  proposals to use ACE representation of the local part as
>...

Chris,

This issue and option have  been explored at great length in the
past.  I don't believe you are saying anything new.  However, to
reprise the reasons that may not be adequately explained...

(1) The "no one screws with the local-part except the delivery
MTA" principle is important to keep all sorts of things working.
Unlike the DNS, where we had a fairly good idea that we wouldn't
find any letter-letter-hyphen-hyphen names and could test that
hypothesis in, at least, all of the gTLDs, several of the
ccTLDs, and the walkable part of the DNS tree, we have no way to
verify that a possible ACE prefix would not be construed as
something else.   Indeed, we know of all sorts of mildly insane,
but perfectly valid, email local part conventions, including
strings with embedded quotes and spaces and strings that embed
command line elements.  Those things might be unwise in the
extreme, but there is no way that an i18n mail activity can blow
them off.

(2) I'm at IGF in Rio, and have gotten several earfulls about
the degree to which the whole ACE idea is part of a broken big
picture because it assumes that ASCII is, and should be the
world's base character set.  IDNA was built around the
assumption that users would never, or almost never, see the
punycode ACE forms.  That assumption turned out to be false and
the user who has been living with a bad transliteration into
ASCII that is at least memorable often doesn't think that
xn--CCCCCC is an improvement (see Andrew Sullivan's note.   I
think the IDNA decision was reasonable, but I've been told this
week that IDNA is a failure from a cultural diversity standpoint
and that we should discard it and dust off and approve my old
"new class" proposal (which has its own set of very significant
technical and operational problems).  I've also been told that,
if there is justification for using an ACE, one needs to justify
it and not, e.g., what I'm now going to name a ECA... an
Arabic-compatible encoding because there is really no rational
reason why ASCII should be the base special-case in an
internationalized system.  

The things that you see as protocol difficulties are actually
useful because they provide additional protection against
misinterpretation of addresses and data in legacy environments.
It remains an important goal for email that we know that it will
either be delivered to the right place or not at all.  ACEs get
very weak on that dimension of the receiving systems have any
sort of algorithmic way of dealing with addresses.

And, as others have pointed out, if you need an ACE, use a URI
mailto and let the characters be turned into hex-encoded UTF-8
octets.  That format is well-defined as part of the EAI spec,
where it is not for raw addresses, it is certainly provides an
ASCII form.

    john
b

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 14 Nov 2007 13:44:45 +0100

>> If you want UTF-8 in email addresses, you have to use an UTF-8 encoding of 
but there's no difference between an email address and a postal address 
in this regard.

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 14 Nov 2007 10:30:55 -0500

>       UTF-8 is also permissible.                                      

This seems merely to be a member of a class of problems that I'll call
"managerial", in the presence of internationalized data.

There are obviously going to be people who want ASCII-only values, and
who think that magic will happen in the computer world that somehow
transmutes 8 bits into 7.  But 7 bits ain't that big: no amount of
asking to make 7 bits bigger will cause non-ASCII to fit in
ASCII-only.

I have been in plenty of meetings in my lifetime where the action
demanded of me by someone was really "to make light go faster".  But
no matter how they insisted, I was unable to comply with this
request.  

The correct reply to ICANN's request is, "You get all the data, or you
get US-ASCII only.  Pick which you prefer."

A

---

From: "Chris Walker" <[email-masked]>
Date: Wed, 14 Nov 2007 11:27:55 -0700

> of the CSV file. Just as you have to do if you have non-ASCII

The hard cold reality on the ground is that there are innumerable ASCII
business processes in existence today. Period. This working group cannot
expect all of them to re-engineer to 8 bit capable systems just to support
new email addresses.

The example above regarding postal addresses is not accurate,
transliterations of every significant non roman character based language
exist for representing those languages in roman characters.

    * ISO 9 — Cyrillic
    * ISO 233 — Arabic
    * ISO 259 — Hebrew
    * ISO 843 — Greek
    * ISO 3602 — Japanese
    * ISO 7098 — Chinese
    * ISO 9984 — Georgian
    * ISO 9985 — Armenian
    * ISO 11940 — Thai
    * ISO 11941 — Korean (different systems for North and South Korea)
    * ISO 15919 — Indic scripts

I have worked in this space for many years and I know of several Fortune
500 companies that have 10s of millions of international customers with
*not a single* postal address stored in Unicode.

________________________________________________________

The conversion of all ASCII systems represents 10s or 100s of millions of
dollars in development costs, just to support internationalized email
addresses.

The companies that have these systems may very likely upgrade their email
server software when internationalized email functionality is available
and can send and receive internationalized emails. That will not magically
allow all the other software that exists in a company that stores,
communicates, or otherwise 'sees' an email address to function properly.

I am positive that the number of systems that come into contact with email
addresses vastly outnumber the number of email servers.

The burden and costs to business and organizations to upgrade email server
software, will be minor compared to other costs. Due to the effort and
expense many of these systems will not be upgraded to being 8bit capable.

Not doing anything, not providing guidance will naturally result in chaos.
There are lots of bright people involved in this effort, there has got to
be some way to complete the following sentence, and include it in one of
RFCs in a prominent location:

The preferred method for representing internationalized email addresses is
in their native UTF8 format, however, if it is not possible to do so and
an internationalized email address must be stored, communicated, or
processed by an ASCII system (all outside of the scope of actually sending
an email) the preferred encoding method is _____________________________.

Chris

---

From: John C Klensin <[email-masked]>
Date: Wed, 14 Nov 2007 14:11:27 -0500

--On Tuesday, 13 November, 2007 17:19 -0200 Stephane Bortzmeyer
<[email-masked]> wrote:

> 4.1.1 Escrow records shall be compiled into a single
> (uncompressed) CSV       text file or multiple (uncompressed)
> text files approximately 1 gigabyte       or one million rows
> in size, in compliance with RFC 4180                
> <http://tools.ietf.org/html/rfc4180>. In accordance with RFC
> 4180, the       character encoding for the CSV file should be
> US-ASCII, although             UTF-8 is also permissible.
> 
> 
> If a technical or administrative contact has an
> internationalized email address, the only way is to send UTF-8
> (which is only "permissible"). At the present time, we have no
> standard way to encode the internationalized address in
> US-ASCII so the registrar cannot comply with the "should be"
> above.

Stephane,

In addition to the comments others have made about this, four
observations:

(1) Even though it is very useful in trying to nail down the
definition of a widely-used but much-varied format, RFC 4180 is
an informational document, not an IETF standard of any type.  If
ICANN wants to "standardize" it to the extent that they write a
spec that requires compliance with it, no one can prevent their
doing so, but whatever decisions they make and rules they adopt
for the use of a non-standard doesn't have much, if anything, to
do with the IETF.

(2) Since ICANN is not a standards body and, in particular, is
not the IETF, we have no real way to real way to interpret what
"should be" or "permissible" mean.  It is almost certainly not
reasonable to assume they correspond to the SHOULD and MAY of
RFC 2119.

(3) The problems with CSV files, including issues of what to do
with too-short or too-long records and the great difficulty in
extending the format with new fields and handling information
with a variable number of subfields, are well known.  Indeed,
they are well-known enough that the choice of CSV, rather than
some variety of tagged data (e.g., XML) for this type of
information should probably be described by such terms of
precise technical evaluation as "plain dumb".   

There is a considerable history of CSV files --even those that
contain header lines -- being hard to handle when revisited
after several years, especially if the data element list has
been changed for more recent versions and software updated to
match.  That history is strong enough that, were someone
actively involved with ICANN reading this discussion, I think he
or she would be doing the community a favor by asking that this
issue be reopened and the format changed before we all come to
regret the CSV files.

(4)  Finally, "... no standard way to encode the
internationalized address in US-ASCII so the registrar cannot
comply with the "should be" above" seems to be to lie somewhere
between "incorrect" and a "red herring".  Even without
Alt-Address, nothing prevents a user from having more than one
address, whether with aliases or alternate labels.   It has been
fairly general knowledge for years that people with email
addresses structured as [email-masked] should find
something else to use in SOA record contact fields because a lot
of tools can't handle "joe\.blogs.example.com" properly.  This
is really no different: if the information is to be in UTF-8,
then an non-ASCII email address might plausibly be included
(although I wouldn't recommend it until the EAI work is _very_
widely supported).  If the information is to be in ASCII, then
anyone with a non-ASCII email address will have to supply an
ASCII alias.  That is certainly not a big deal in the grand
scheme of things.

    john

---

From: Eric Brunner-Williams <[email-masked]>
Date: Wed, 14 Nov 2007 12:06:59 -0800

Guys,

I would have caught this had Mike Zupke showed me the draft. I suggest 
we treat it
as something that needs clarification.

Of course, you're all free to do otherwise.

Eric

---

From: Martin Duerst <[email-masked]>
Date: Thu, 15 Nov 2007 15:43:34 +0900

Hello Stephane,

I'm confused. You haven't made it clear whether you are pointing us
to RFC 4180 to show that in some contexts, there are limitations to
US-ASCII, or to show us that at least in this context, there is no
issue, because it's always possible to use UTF-8.

It seems that other's reading (most explicitly John's reading) is
different, but for me, the text you quoted essentially reads:

Use US-ASCII if you can, if not, use UTF-8.

Translating into what might be the closest normative IETF language,
for what it's worth, would result in something like:

The character encoding for the CSV file SHOULD be US-ASCII, although      
UTF-8 MAY also be used.

Again reading this means the same as the above. A SHOULD means:
Do unless for good reasons, you can't. The next clause gives an
alternative, which makes clear that "the data cannot be represented
in US-ASCII" is a good reason for deviating from the SHOULD.

So if you indeed wanted to use the text cited for saying "see, here's
another case where we an ACE would come in handy", then I don't
think you can make this point.

Regards,    Martin.

>> If you want UTF-8 in email addresses, you have to use an UTF-8 encoding of 

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 15 Nov 2007 09:56:22 +0100

> an email) the preferred encoding method is _____________________________.

...specified as hex. NCR in I-D.klensin-unicode-escapes.  Just an idea,
the Last Call for this draft ends today.

 Frank

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 15 Nov 2007 08:01:40 -0500

> communicates, or otherwise 'sees' an email address to function properly.

Right.  A company that simply upgrades their mail server and says,
"Ok, internationalised mail on now!" gets what they deserve.  What
you appear to be arguing is that we have to arrange for
interoperation with every possible piece of legacy software out there

You're quite right that changing the expected characters in email
addresses is a Big Expensive Deal, and that it will cause pain with a
lot of legacy systems.  Any time you change a fundamental feature of
your systems, you have this problem.  And email is surely a
fundamental feature of the Internet.  Sorry, but making this kind of
fundamental change is going to be expensive.  

That said, see Frank Ellerman's note on a possible way to do
something like what you want.  (But it's not perfect.)

A

-- 
----
                                        +1 416 646 3304 x4110

---

From: John C Klensin <[email-masked]>
Date: Thu, 15 Nov 2007 09:27:13 -0500

--On Wednesday, 14 November, 2007 11:27 -0700 Chris Walker
<[email-masked]> wrote:

> The hard cold reality on the ground is that there are
> innumerable ASCII business processes in existence today.
> Period. This working group cannot expect all of them to
> re-engineer to 8 bit capable systems just to support new email
> addresses.

Chris, it seems to me that you are making up a problem that does
not exist.  We all know that there are innumerable ASCII
business processes.  We also all know that most business
transactions that cross arbitrary national boundaries,
especially multiparty ones, are carried out in English.  This
protocol work changes none of that, any more than the 1992
introduction of capability for multilingual mail required that
all non-English messages be transmitted in a transliterated form.

I would expect business processes that work in ASCII over the
Internet to continue to use all-ASCII addresses... forever or
for as long as those processes last.  To the extent to which
people have native-language addresses, I would expect them to
maintain ASCII aliases or ASCII-addressed mailboxes if they want
to participate in those business processes.  And I would expect
downgrading to be used very little in these cases because the
processes will continue to work as they are working today.

I also expect that, if a multinational business that carries out
its interoffice and internal transactions in ASCII, using ASCII
content and ASCII addresses, wants to enhance its business to
consumer activities in Lower Slobbovia, where few of the Lower
Slobs read or write ASCII characters, I will put a large effort
into localization for email communications, web sites, etc.,
that are used to reach and communicate with those consumers.
That is just what that business does today if it expects to
survive, enhanced only by the ability to advertise a localized
(and national character) email address as well as accepting
email via my already-localized web site.  Throughout history,
trying to carry on business with a population in a particular
language and script that the population doesn't understand has
not been an especially productive or profitable choice and,
again, this work won't change that.

Now, if the real question you are asking is how one transmits an
email address that is defined in non-ASCII characters through
some other protocol, the answers to those questions lie in those
protocols.  In running text in a message, we use whatever
encoding would be used for that message -- reversible encodings
such as Base64 for email.  In something like a MAILTO URL, we
know how to do the encoding because it is part of the base URL
specification. 

What we haven't done to create a standard for re-representing
internationalized addresses independent of protocol.  The reason
is yet another symptom of the reason you are seeing resistance:
[email-masked] is a perfectly valid email
address.  Worse, it is one that example.com might well interpret
as an instruction to try to send mail to the address
W%CF%87%CF%81%CE%B9%CF at host 82 (whatever that means to the
server at example.com).   There is no way to be sure whether the
local part should be interpreted as a single address, as routing
instructions, or as Wχρις (U+0057 U+03C7 U+03C1 U+03B9
U+03C2).  This would be true of any other encoding one tried to
select.

> transliterations of every significant non roman character
> based language exist for representing those languages in roman
> characters.

Unfortunately, as you presumably know, while those
transliterations exist, they are not guaranteed to be
reversible, nor may they be obvious to native users of the
original scripts.  When they are used in postal mail delivery,
they may result in very long delays, in part because the postal
treaties do not require that they be accepted.  They are also
fairly useless to anyone who doesn't actively use Roman
characters.  Why not transliterate everything into Arabic or
Hangul instead?

Again, I'm trying to figure out what problem you are seeing here
... one that is not either solved by existing tools or that
cannot be solved other than by saying "everyone should learn
Latin (and I do mean _Latin_, not 'Roman-derived' here) or,
better yet, English".  That is not what this is about.  It is
about facilitating better communications within language
communities and I just don't think that the issues you are
raising are relevant if that goal is recognized as acceptable
and desirable.

      john

---

From: "Chris Walker" <[email-masked]>
Date: Fri, 16 Nov 2007 02:59:25 -0700

John,

It isn't an imagined problem, it's a real one. That's why I'm here.

Many large corporations do a lot of work with business partners, other 
corporations, and as such have data flowing into their systems that 
originates in numerous locations out of their control.

Example: User Alice fills out a web form on company X website for some 
reason or another, and one of the data fields is her email address, 
company X accepts a raw UTF8 email address e.g. NON ASCII@NON ASCII. Now 
depending on whar Alice entered in the form, company X may send that 
data to company Y (or company Z). The mechanism that companies X and Y 
have is place to do this is ASCII.

In this extraordinarily simple example, how does company X communicate 
the email address to company Y? Should Alice know to have an ASCII email 
address and to use that one? If Alice is an establised customer of 
company Y, and has a saved profile with a non ASCII address, and that is 
used for transmission to company Y, there may not be a data entry field 
for Alice to fill out on her web form.

Again this is a very simple example, in many cases, supply chains for 
example, the data flow is though numerous companies and back again. In 
order fulfillment, the orginal order may be split into several sub 
orders for be fulfilled by numerous companies. Email notification is a 
common feature of order processing.

Ask yourself "Where does data flow?", in these situations, an email 
address is just another piece of data. A piece a data that has the very 
unfortunate characteristic of no standard ASCII encoding.

Modifying every ASCII system that sees email addresses to handle UTF8 
will make the cost and effort of widening date fields to hold four digit 
years look like childs play.

In all probability two dominate solutions will emerge, a multitude of 
incompatable encodings which over time will corrupt the data quality of 
databases that store email addresses, or just throwing the data away when 
it hits an ASCII system. All in all, this burden will be a huge 
impediment to adoption of EAI as it stands now.

Chris

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 16 Nov 2007 11:38:35 +0100

It's a real problem and occurs in many guises. Another variant is that 
for a long time, my postal address was not pure ASCII. More than once, 
someone attempted to verify that the address existed, was correct, or 
that a credit card was OK, and the check failed.

I venture to suggest that the problem is as old as electronic data 
processing, and that if EAI has to solve even just the email address 
part of the problem, EAI won't conclude at all.

Arnt

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 16 Nov 2007 11:14:36 -0000

> the Last Call for this draft ends today.

But you need some notation to introduce that hex, so people know that it  
needs to be unscrambled.

But we already have such a, notation that works _now_, namely the mailto:  
URL (which you can also translate to/from an IRI if you wish when crossing  
a UTF-8/ASCII boundary.

---

From: John C Klensin <[email-masked]>
Date: Fri, 16 Nov 2007 09:29:39 -0500

--On Friday, 16 November, 2007 02:59 -0700 Chris Walker
<[email-masked]> wrote:

>....

Chris, I have lots of experience with the problem you describe.
It is quite real, and I tried to carefully avoid saying anything
else.

The imaginary one is only about assuming that the business
partners, who have been using ASCII all along, will suddenly
insist on using internationalized email addresses.

There is a slightly different problem when those business
partners are dealing with their customers.  But almost no one is
using email any longer to send and receive commercial
transactions directly, nothing prevents a company from insisting
on an ASCII address (or an ASCII ALT-ADDRESS) on a web page
(note that legacy web pages will almost certainly not accept
non-ASCII characters- we have enough trouble getting them to
accept "+", "=", and "/").
 
> Many large corporations do a lot of work with business
> partners, other  corporations, and as such have data flowing
> into their systems that  originates in numerous locations out
> of their control.
> 
> Example: User Alice fills out a web form on company X website
> for some  reason or another, and one of the data fields is her
> email address,  company X accepts a raw UTF8 email address
> e.g. NON ASCII@NON ASCII. Now  depending on whar Alice entered
> in the form, company X may send that  data to company Y (or
> company Z). The mechanism that companies X and Y  have is
> place to do this is ASCII.

If company X permits an non-ASCII address, doesn't get an
Alt-Address, and has no theory about how to communicate that
address to company Y, then it is, in fact, in trouble.   But the
same problem occurs in the general case with postal addresses
that are entirely in non-Latin characters: the postal treaties
_do not_ require that countries accept transliterations on
anything that goes inside a bag to an international reception
point, bags that are labeled using ISO 3166 codes, not ASCII
country names.

However, they do have a way to transmit such email addresses.
They put them into mailto URLs using %HH encoding.  There is,
however, one additional thing they need to do and it is why this
is inappropriate for standardization: company X and Y have to
agree that their mail systems will never accept an address from
a customer with multiple "%" signs in it.   Since the routing
that is normally implied by "%" is now considered a bad and
dangerous practice, that shouldn't be a big deal.   Or they
could adopt some other convention and adjust their procedures so
that a different character that _they_ (specifically Company X
in its web interface) prohibit in mail addresses was used as an
indicator that the local part was encoded and needed to be
decoded.

I'm not suggesting that these addresses can't be encoded in
special circumstances, only that using encoding instead of
directly represented (in UTF-8) addresses won't work for the
most general cases and consequently can't be standardized.

> In this extraordinarily simple example, how does company X
> communicate  the email address to company Y? Should Alice know
> to have an ASCII email  address and to use that one? If Alice
> is an establised customer of  company Y, and has a saved
> profile with a non ASCII address, and that is  used for
> transmission to company Y, there may not be a data entry field 

If company Y has stored a non-ASCII address for Alice and is not
prepared to receive a non-ASCII address for her from company X
which nonetheless accepts that address from their web forms,
then someone is being extraordinarily stupid.   Remember that
company X must choose to accept a non-ASCII address to have an
address to transmit to Company Y.  If it can't, or won't, it
should, indeed, insist that Alice provide an ASCII-only address.
The latter is something I expect to happen regularly for some
years, even if only by legacy web pages generating "not a valid
address" messages when they see non-ASCII characters.

> Again this is a very simple example, in many cases, supply
> chains for  example, the data flow is though numerous
> companies and back again. In  order fulfillment, the orginal
> order may be split into several sub  orders for be fulfilled
> by numerous companies. Email notification is a  common feature
> of order processing.

Same comments.  I think you should look at the EDI standards in
international use and how they handle (or don't handle) selected
non-ASCII data.

For a competent business enterprise, no data field is "just
another piece of data".  Phone numbers are validated to make
sure that, possibly modulo "+" and well-known separators, they
are all-numeric.   Dates are validated to make sure that no one
is born in the 14th months.  And so on.  The long supply chains
you describe have long-ago learned to do those sorts of
validation and insist on it from their partners because,
otherwise, chaos results.   If someone is going to accept a
non-ASCII email address, they will need to obtain upstream
agreements about how to process them, and that is true whether
the address is passed along in UTF-8 or mapped, by mutual
agreement, to an encoded form or alternate representation.

     john

---

From: "Chris Walker" <[email-masked]>
Date: Fri, 16 Nov 2007 14:20:46 -0700

-----Original Message-----
From: John C Klensin <[email-masked]>
To: Chris Walker <[email-masked]>, eai list <[email-masked]>
Date: Fri, 16 Nov 2007 09:29:39 -0500
Subject: Re: [EAI] Additional Clarification re: Need of an ACE for EAI

> insist on using internationalized email addresses.

Yes, you are right, it will probably be the customers of a company 
insisting on the use of internationalized email addresses.

> transactions directly,

What does this have to do with "sending comercial transactions directly" 
over email, this is a strawman argument.

> nothing prevents a company from insisting

If EAI is ever going to enjoy widespread use and popularity, web pages 
will have to accept UTF8 email addresses. People will have to have a 
resonable expectation that if they supply a UTF8 email address and it is 
accepted by some entity that they will actually receive email from that 
entity at that address. EAI has an opportunity to correct the ills of 
the past regarding acceptance of "+", "=", and "/" if the work product 
of this group was more vocal on user interface, interoperability issues, 
and practices that involve EAI beyond the UTF8SMTP wire. The tunnel-
vision view that EAI can be 'solved' by only dealing with the email 
client server protocol is only a subset of wider set of issues.

> point, bags that are labeled using ISO 3166 codes, not ASCII

In the general case, there are alternatives to representing a UTF8 
postal address, namely transliteration, and there is a well established 
history and expectation that the non UTF8 version will transit 
international systems, including ASCII, and will still be useable to 
deliver mail.

Even if a user supplies their postal address to an entity in native 
UTF8, that address can be transliterated by the entity before 
transmission to another party. This transliteration is standardized. 
UTF8 email address enjoy no similar standard.

> indicator that the local part was encoded and needed to be

"> company X and Y have to
> a customer with multiple "%" signs in it."

Well now we are getting to the meat of the problem. Passing the buck. 
Just how many of these 'agreements' will be needed? How many possible 
combinations of interactions are there if only 10 companies existed in 
the entire world? 

If the interaction is only between two parties, then each of the 10 
companies could have an agreement with the other 9, 10 x 9 = 90. The 
number of paths that traverse all 10 companies is 10! = 1 x 2 x 3 x 4 x 
5 x 6 x 7 x 8 x 9 x 10 = 3,628,800. 

>"Or they could adopt some other convention".

The avoidance of this is why standardization exists.

> most general cases and consequently can't be standardized.

What do you consider special and general circumstances? Auguing that an 
international email address encoding can't be standardized is like 
arguing that internatioinalized domain name encoding can't be 
standardized. Except it's a losing argument, since it already has been 
standardized.  

> address" messages when they see non-ASCII characters.

Again more burden and passing the buck, this time to Alice. Poor Alice 
who would love to have just her internationalized email address, but 
can't. She needs two addresses, an ASCII one in addition to her UTF8 
one. She has to check email in multiple places, and since there is so 
little reliability of actually getting any email at her UTF8 address 
from any company, she considers the UTF8 one a novelty that only a few 
close friends (actual humans) use. If Alice grows tired of checking her 
email in two places, which one will she abandon?

> international use and how they handle (or don't handle) selected

I think you should consider that a fraction of actual supply chain like 
systems actually use EDI.

> agreement, to an encoded form or alternate representation.

Validation and encoding are two seperate proccesses and issues, encoding 
does not perform validation, and validation does not perform encoding.

chris

---

From: Martin Duerst <[email-masked]>
Date: Fri, 16 Nov 2007 19:52:29 +0900

Hello Chris,

I think John and Harald and others have already tried, but I'll
try again. I think we understand your scenario, and we know it's
real. But what is at least as real, and more importantly, already
possible/happening, is the same problem with names and addresses.

Please take a moment and rewrite your text below and replace
"email address" with "person's name", or "customer's street address",
or any such. What you will see is that:
1) The problem already exists.
2) If companies haven't handled the problem for name or address,
   they are already having problems. If they have handled it,
   they should be able to handle it for email addresses, too.

BTW, I happen to speak out of experience. My name, in fully correct
spelling, contains an u-umlaut, in ASCII, the closest you can get
is something like Du"rst. For most forms, I just enter Duerst.
When I (re)joined the ACM a couple years back, I entered the fully
correct spelling, because the Web site accepted it and I was in the
mood to give it a try. As you might expect, I have received both
printed and electronic material with various degradations, and there
may be degradations that I haven't seen because they haven't made
it here. There are all kinds of weird effects, such as the fact
that I repeatedly get invitations to join the IEEE, addressed
to the ACM variant of the address, even though I'm already an
IEEE member. I'd assume that if the spelling of my name was more
consistent, the IEEE would weed out the duplicate.

Anyway, as you can see, the problems you mention can already happen.
If a company can handle them, they should be able to figure out how
to address non-ASCII email addresses, too (if only by not handling them).
If a company or organization is currently doing a bad job, then non-ASCII
email addresses might give them an incentive to fix some things overall.

Regards,    Martin.

---

From: John C Klensin <[email-masked]>
Date: Fri, 16 Nov 2007 19:58:50 -0500

Chris,

I don't think this conversation is being particularly productive 
or informative, so am about to drop out of it.  You may 
disagree, of course, and continue it with others.

A few observations that might help clarify where things stand, 
even if they don't move things forward:

(1)  I believe that far more attention has been paid to user 
interface, process, and deployment issues than you seem to 
assume.  The considerations involved many tradeoffs.  The fact 
that the conclusions were not the ones you might have preferred 
does not imply that we were ignorant of the issues or that we 
blew them off.

(2) As Andrew somewhat indirectly pointed out, opinions about 
the success of IDNA differ widely depending on who you talk with 
and what role they occupy.  When they are evaluated as utilized 
by users, the assumption when the protocol was written was that 
those users would never (or almost never) actually see the ACE 
form.  Today, users see the ACE form frequently and some even 
prefer it to the loss of information that might otherwise occur. 
But the ACE form is not a transliteration: few users who used to 
deal with transliterations (even while hating them) consider 
looking at an ACE form that has no mnemonic value at all to be 
an improvement.

(3) The IETF doesn't do UI work.  When exceptions have been 
made, we have usually regretted it because experience indicates 
that, as a community, we aren't very good at it.  In general, 
the only things we attempt to standardize are things that appear 
on the wire.  So, while everyone would probably agree with you 
that the on-the-wire behavior isn't all of the picture, the 
charter of the WG doesn't extend to how email addresses are 
transmitted in back-end processes within a company or between a 
company and its business partners.

(4) I've tried to say this before, but the big problem with an 
ACE model for email local-parts is that all of the plausible ACE 
forms are plausible email addresses.  Consequently, there is no 
way to reliably identify that an encoding has occurred and hence 
to decode it.  This is a much worse problem when email addresses 
are transmitted in running text or as part of formatted data 
files than when they are used in email protocols, although the 
protocol issues are non-trivial.  It is also different for email 
(essentially a collection of peer-to-peer transactions with 
relaying) than it is for the DNS (a client-server environment), 
both because of that difference and because we were able to 
examine the DNS tree carefully to conclude that chances of real 
problems were acceptably small.

Put differently, the big problem with an ACE is telling that it 
is an ACE, rather than a direct form.  Protecting IDNA requires 
that any string with hyphens in the third and fourth character 
positions be prohibited as a DNS label unless it is an IDNA ACE 
("punycode") form with the "xn--" prefix.  There is no practical 
way to apply such a restriction to email and keep it even 
reasonably robust (robustness against known behavior of many 
email systems is, among other things, the reason why making a 
distinction between
     .xyz...
and
     ".xyz...
is pretty much a non-starter.

But the problem is standardizing an auto-detecting ACE form for 
use in the global email system.  Within, e.g., a company 
environment, arranging some sort of flag or tagging to 
distinguish, unambiguously and non-heuristically, between an 
encoded address (whether ACE or something as trivial as Base64 
of %HH encoding) and a non-encoded one, no matter how 
funny-looking, is certainly plausible.  But, in the most general 
case, a ACE that is detected to be an ACE by its spelling or 
syntax alone isn't going to work for email (as well as getting 
us back into the problems that are mentioned above and discussed 
in Andrew's note).

That is what I was trying to say about MAILTO.  The current 
definition prohibits non-ASCII strings.  But assume that either 
a revision were undertaken or a new, MAILTOO URL was defined, in 
either case using ordinary URI rules.  One could then prohibit 
any local-part from appearing in the URI that started with "%" 
(even though it would be valid in a stand-alone email address). 
Or one could say that any %NN construction in a MAILTO[O] URL 
was a character escape and that any actual occurrence of "%" 
would have to be shown as %25, which would be consistent with 
the way the specs (both MAILTO and URI) are now written.  It 
would certainly be sensible to look at that possibility, or the 
possibility of some other tagging or restrictions, for a 
representation format.  Looking at MAILTO _is_ in the WG's 
charter and it would be entirely reasonable to have this 
discussion when we get to it.

Of course, if that change were made to MAILTO, and a company or 
set of companies decided that what was to appear in their email 
address data fields was a MAILTO URL or its data portion, rather 
than a raw email address, your problem, as I understand it, 
would essentially be solved.

(5)  In my experience, when companies or trading partners start 
exchanging data electronically, the process involves reaching 
agreements about formats and data elements.   Sometimes, those 
agreements are "we will all follow standard A, with profile B" 
and sometimes they involve much more specific agreements.  But 
it rarely, if ever, happens that company X will send data to 
company Y and simply assume that company Y will know how to 
figure things out.

That is particularly important in this context because, if a 
series of companies have been exchanging information in 
ASCII-only form, with agreements about elements and formats that 
say "ASCII-only", if someone suddenly starts sending UTF-8 _or_ 
Unicode data using some magic encoding that causes them to look 
like ASCII characters, that would be a breach of the agreements. 
They might be able to get away with that, but only if there was 
no expectation of the encoded data being decoded.  And that 
would take us back to the situation in which users (and systems) 
get to see ugly-looking representational forms that the user who 
actually owns the address may not even recognize... a bad 
situation.

(6) Despite the standards for transliteration that exist for 
_some_ (not all important) scripts, transliteration is a bad 
deal.  People who are concerned about national languages and 
cultural preservation don't like it and often consider it 
insulting.   As I suggested in an earlier note, there is little 
substantive reason for transliterating to ASCII rather than, 
e.g., to Cyrillic or Hangul.   More important, different 
languages and dialects use the same characters to represent 
different phonemes, which can make transliteration models both 
counter-intuitive and ambiguous in application.

(7) Several of your comments imply that you believe that two 
addresses implies two mailboxes and two places to check for 
mail, with no alternative.  That is simply not how the mail 
system works and is used in practice.

    john

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 16 Nov 2007 13:11:12 -0200

On Thu, Nov 15, 2007 at 03:43:34PM +0900,
 Martin Duerst <[email-masked]> wrote 
 a message of 58 lines which said:

> You haven't made it clear whether you are pointing us to RFC 4180

I was pointing to an ICANN document (which itself pointed to RFC 4180,
but it was a detail, the relevant part of the ICANN document was not
about CSV but about UTF-8). My intent was just to provide a real-world
use case for the discussion, not to discuss wether ICANN is right or
wrong.

> So if you indeed wanted to use the text cited for saying "see,
> here's another case where we an ACE would come in handy",

Yes, that was the idea. It does not mean I approve the idea of an ACE,
just that I wanted people to recognize that there may be a problem
(which does not mean it is our responsability to solve it).

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 16 Nov 2007 13:13:28 -0200

On Wed, Nov 14, 2007 at 02:11:27PM -0500,
 John C Klensin <[email-masked]> wrote 
 a message of 86 lines which said:

> definition of a widely-used but much-varied format, RFC 4180 is an
> informational document, not an IETF standard of any type.

My intent was not to use RFC 4180 to prove anything but to show, from
a field where some people here may be familiar with (the ICANN world),
a possible case where the lack of an ACE may be a problem.

> (3) The problems with CSV files, including issues of what to do with
> too-short or too-long records and the great difficulty in extending
> the format with new fields and handling information with a variable
> number of subfields, are well known.  Indeed, they are well-known
> enough that the choice of CSV, rather than some variety of tagged
> data (e.g., XML) for this type of information should probably be
> described by such terms of precise technical evaluation as "plain
> dumb".

Report it to ICANN. XML vs. CSV has nothing to do with this
discussion. The point was the sentence about UTF-8 and US-ASCII.

---

From: Eric Brunner-Williams <[email-masked]>
Date: Sat, 17 Nov 2007 10:26:12 -0800

> (which does not mean it is our responsability to solve it).

I didn't want to comment on the ICANN registrar data escrow format 
question (csv vs
alternative(s)), as that was settled some months ago, my views not 
withstanding.

I also didn't want to comment on the issue of whether the purpose of 
registrar data escrow
is to transfer data, some of which may be broken, to a successor 
registrar operator, after
some predicate condition is determined to exist, e.g., RegisterFly data 
transfer to GoDaddy,
or to ensure registrant email address validity, as that too was 
similarly settled.

What I wanted to point out was just this. A problematic use case, which 
we are not
responsible to solve.

Eric

---

From: "Chris Walker" <[email-masked]>
Date: Sat, 17 Nov 2007 15:12:06 -0700

-----Original Message-----
From: John C Klensin <[email-masked]>
To: Chris Walker <[email-masked]>
Cc: eai list <[email-masked]>
Date: Fri, 16 Nov 2007 19:58:50 -0500
Subject: Re: [EAI] Additional Clarification re: Need of an ACE for EAI

> is pretty much a non-starter.

Actually the problem is defining an encoding that affect only the local 
part while preserving the @ sign and the domain in its original form. 
Encoding an email address vs. encoding local-part provides numerous 
collision free possibilities, any encoding that does not result with an 
@ sign, or locates the encoding identifier at the trailing end of the 
encoding string. non-ascii@non-ascii = yn--someasciiencoding or 
someasciiencoding--yn.

However since one of the fundamental cornerstones of this entire 
UTF8SMTP effort is the argument that an internationaled email address 
CAN'T be encoded in ASCII, I can see how you are not likely to concede 
this point.

> would essentially be solved.

It would only be solved, if there was a statement that indicated that 
this was a preferred encoding for ASCII data processing so it was not 
ambiguous which form should be used, vs. other encoding formats e.g. the 
encoding format used to preserve the downgraded original UTF8 address 
in 'downgrading' document.

> actually owns the address may not even recognize... a bad 

Of course some chages will need to be made, select an ASCII encoding or 
reengineer to 8bit, the only point of selecting a preferred ASCII 
encoding it to prevent the randomness of every interface between two 
entities from deciding on a different encoding.

> system works and is used in practice.

Having a UTF8 and ASCII email address at the same server MAY happen, I 
have not seen anything in the drafts to indicate that it shall happen.

---

From: "Chris Walker" <[email-masked]>
Date: Sat, 17 Nov 2007 16:21:13 -0700

-----Original Message-----
From: Martin Duerst <[email-masked]>
To: "Chris Walker" <[email-masked]>, "John C Klensin" 
<[email-masked]>, "eai list" <[email-masked]>
Date: Fri, 16 Nov 2007 19:52:29 +0900
Subject: Re: [EAI] Additional Clarification re: Need of an ACE for EAI

>    they should be able to handle it for email addresses, too.

Hi Martin,

Are you suggesting that the preferential encoding for an 
internationalized email address should be romanized phonomes? 

The mechanisms of delivery are not the same for names, postal addresses 
and email addresses. Names and postal addresses used in the delivery of 
postal mail involves a human, where the delivery of an email is expected 
to be possible without interpretation by a human.

> consistent, the IEEE would weed out the duplicate.

I certainly understand and sympathize with the complications that you 
have to deal with. Having personally questioned many people who commonly 
use germanic languages on the topic (Germans, Austrians, and Swiss). The 
consensus that I have been able to determine is: It is preferred to 
preserve characters with umlauts, if that is not possible then convert u 
umlaut to ue, o umlaut to oe, etc.  The character that resembles a fancy 
capital B, ess-tzet, a long s, is converted to ss. The practice of 
converting u umlaut to u, and o umlaut to o, is considered ignorant and 
insensitive. 

> email addresses might give them an incentive to fix some things

The issues you notice regarding multiple inclusion or not recognizing 
your existing inclusion in a list is in the CRM match, merge, purge 
problem space. Even if a fully 8bit end to end system accepted and 
preserved UTF8 names and postal addresses, there is a hitch.

Some people will submit their identifying information in transliterated 
or ASCII form. Unless the implementation of the match, merge, purge 
functions is internally aware of transliteration and even Latin1 to 
ASCII conversions, it will not notice that two names or postal addresses 
are the same.

Since many of these conversions are one way streets, i.e. there is no 
way to know that Duerst should be D(u umalaut)rst or that you actually 
prefer Duerst, all data is usually converted to the most reduced form to 
provide the ability to perform matches.

I wish that applying the best practices for dealing with 
internationalized names and postal addresses could be applied to 
internationlized email addresses, however the lack of humans in the 
delivery process, and the issues you pointed out with Latin1 to ASCII 
conversions with umlauts would seem to indicate that a more precise 
solution is needed.

---

From: "Chris Walker" <[email-masked]>
Date: Sun, 18 Nov 2007 18:14:48 -0700

-----Original Message-----
From: John C Klensin <[email-masked]>
To: Chris Walker <[email-masked]>
Cc: eai list <[email-masked]>
Date: Fri, 16 Nov 2007 19:58:50 -0500
Subject: Re: [EAI] Additional Clarification re: Need of an ACE for EAI
> discussion when we get to it.

Seems ICANN has already moved past that since their IDNA email test 
pages have UTF8 in the example "Requires full IDN support" mailto links:

http://idn.icann.org/E-mail_test

Chris

---

From: Martin Duerst <[email-masked]>
Date: Mon, 19 Nov 2007 09:21:59 +0900

>internationalized email address should be romanized phonomes? 

No, I'm suggesting that if they already accept input for non-romanized
names/addresses, they can also accept non-romanized email addresses
(assuming they actually implement EAI). If they don't accept non-romanized
names, then they shouldn't accept non-romanized email addresses.
The email address will then be whatever ASCII-only address the
user will input. This may be a romanized version of an existing
non-romanized address, or something completely different, maybe
even with a different provider.

>to be possible without interpretation by a human.

Yes. That doesn't change the fact that a system should only accept
what it is able to handle. Automatic conversion from non-ASCII to
ASCII only may work in some cases for names and addresses, but in
general isn't going to work, even for names and addresses, so it
shouldn't be relied on.

>ASCII conversions, it will not notice that two names or postal addresses 

Yes. And if somebody in the US registers as Bill Smith and as
William Smith, that might not be purged, either. My guess is
that match/merge/purge will always to some extent be a black
art, i.e. not something that ever will work 100%.

>conversions with umlauts would seem to indicate that a more precise 

Yes. But the availability of a standard ACE for internationalized
email addresses wouldn't really help. Most people, when asked to
input an ASCII-only email address, would certainly prefer to use
something like [email-masked] to something like
[email-masked] (assuming that's a fallback for
d(u umalaut)[email-masked]).

Regards,    Martin.

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Nov 2007 21:24:34 -0500

--On Sunday, 18 November, 2007 18:14 -0700 Chris Walker
<[email-masked]> wrote:

>> definition prohibits non-ASCII strings.  But assume that
>> either  a revision were undertaken or a new, MAILTOO URL was
>> defined, in  either case using ordinary URI rules.  One could
>> then prohibit  any local-part from appearing in the URI that
>> started with "%"  (even though it would be valid in a
>> stand-alone email address).  Or one could say that any %NN
>> construction in a MAILTO[O] URL  was a character escape and
>> that any actual occurrence of "%"  would have to be shown as
>> %25, which would be consistent with  the way the specs (both
>> MAILTO and URI) are now written.  It  would certainly be
>> sensible to look at that possibility, or the  possibility of
>> some other tagging or restrictions, for a  representation
>> format.  Looking at MAILTO _is_ in the WG's  charter and it
>> would be entirely reasonable to have this  discussion when we
>> get to it.
> 
> 
> Seems ICANN has already moved past that since their IDNA email
> test  pages have UTF8 in the example "Requires full IDN
> support" mailto links:
> 

Nope.  If you look at RFC 3490, you will see some words about a
"domain name slot".   An application _may_  choose to accept
[email-masked] (chosen arbitrarily
from the ICANN list) as an address because the Greek strong is
in one of those domain name slots.   However, without EAI and
the UTF8SMTP extension, that application would be very seriously
violating RFC 2821 (and other standards) by putting anything
other than [email-masked] on the wire.

The Mailto link is another issue.   As a URL, the mailto link
for that "full IDN support" column must be represented as 

mailto:mailtest@%CF%80%CE%B1%CF%81%CE%AC%CE%B4%CE%B5%CE%B9%CE%B3%CE%BC%CE%B1.%CE%B4%CE%BF%CE%BA%CE%B9%CE%BC%CE%AE

which more or less demonstrates my point about just how
user-unfriendly automatically-produced ASCII forms that do not
lose information can, and will, get.

    john

---

From: Martin Duerst <[email-masked]>
Date: Mon, 19 Nov 2007 12:50:33 +0900

>lose information can, and will, get.

Well, it's actually a bit different. The current mailto: spec does
not allow percent-escapes in general, and in particular does not
allow them in the domain name 'slot'. Still, several browsers alread
allow this, as well as the corresponding IRI. And ICANN is using the
IRI form in their example page.

I have written a draft to document these changes, but the latest version
is over a year old.
(see http://tools.ietf.org/html/draft-duerst-mailto-bis-03)
I'm planning to update the draft, but that won't happen today, so
it won't happen for the next few weeks, because of the deadline.
Any comments appreciated.

Regards,    Martin.

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Nov 2007 23:22:14 -0500

--On Monday, 19 November, 2007 12:50 +0900 Martin Duerst
<[email-masked]> wrote:

>> mailto:mailtest@%CF%80%CE%B1%CF%81%CE%AC%CE%B4%CE%B5%CE%B9%CE
>> %B3%CE%BC%CE%B1.%CE%B4%CE%BF%CE%BA%CE%B9%CE%BC%CE%AE
>> 
> Well, it's actually a bit different. The current mailto: spec
> does not allow percent-escapes in general, and in particular
> does not allow them in the domain name 'slot'.

You are, of course, correct.  My apologies. 

> Still, several
> browsers alread allow this, as well as the corresponding IRI.
> And ICANN is using the IRI form in their example page.
 
> I have written a draft to document these changes, but the
> latest version is over a year old.
> I'm planning to update the draft, but that won't happen today,
> so it won't happen for the next few weeks, because of the
> deadline. Any comments appreciated.

I'm trying very hard to avoid thinking about mailto until the
EAI work is essentially complete, partially because of the
question of how one might accommodate an alternate address if
that were desired.

regards,
    john

---

From: "Chris Walker" <[email-masked]>
Date: Mon, 19 Nov 2007 09:32:40 -0700

John

If you look at the HTML source for
http://idn.icann.org/E-mail_test
you will see that there are UTF8 mailto links, not escaped.
Which are congruent with the UTF8 http links at
http://idn.icann.org/Main_Page

I'm trying yo understand what the point would be of keeping the mailto 
URI escaped in the future?

Chris

-----Original Message-----
From: John C Klensin <[email-masked]>
To: Chris Walker <[email-masked]>, eai list <[email-masked]>
Date: Sun, 18 Nov 2007 21:24:34 -0500
Subject: Re: [EAI] Additional Clarification re: Need of an ACE for EAI

> mailto:mailtest@%CF%80%CE%B1%CF%81%CE%AC%CE%B4%CE%B5%CE%B9%CE%B3%CE%BC%
> CE%B1.%CE%B4%CE%BF%CE%BA%CE%B9%CE%BC%CE%AE
>