Subject: Re: [EAI] double angle brackets

From: Shawn Steele <[email-masked]>
Date: Tue, 8 Sep 2009 16:51:24 +0000

> To replace double angle brackets for the Sender, John Klensin proposed 
> using multiple From fields. SM amended this with Reply-To to specify the 
> preferred address, as in
>         From: EAI-addr, ASCII-addr
>         Reply-To: EAI-addr
> where the EAI-addr gets dropped in Downgrade.

I'd imagine that "something" isn't going to be happy with 2 addresses.

It seems to me that 99% of the time email can be viewed as primarily "sender side" or "recipient side".  Eg: Mail goes from a sender system to a recipient system.  It is very rare (now) that an in-between system would relay mail from an untrusted server to another 3rd party server.  If there is a relay, it'd have to trust the sender, so it's pretty much "sender side," or recognize the recipient "recipient side."

Assuming that a "sender" has an EAI aware system that is completely conformant and "works" with UTF-8 addresses, then the entire "sender side" would have to be updated to support UTF-8 addresses.  If the sender's relays aren't upgraded, then all of their mail ends up being downgraded anyway.

Similarly a recipient either has an entirely EAI aware system, or they don't.  

So I'm wondering how interesting downgrade really is.  If my sending system trusts the sender, presumably it could request an ASCII address from the trusted sender-side server if it needed to downgrade.  (Through some other mechanism, like asking the trusted server what other aliases were valid for that account using some new, or even proprietary, protocol).

So the proposed solution seems like overkill.

VERY worst case a client (like Outlook) can try to send an EAI email.  If it bounces, the client could resend using my ASCII address (assuming it is properly configured).  It could even do that silently, or with maybe a little warning saying that had been downgraded.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Tue, 08 Sep 2009 14:49:55 -0400

--On Tuesday, September 08, 2009 16:51 +0000 Shawn Steele
<[email-masked]> wrote:

>> To replace double angle brackets for the Sender, John Klensin
>> proposed  using multiple From fields. SM amended this with
>> Reply-To to specify the  preferred address, as in

Right.  In particular, there are a lot of systems that do not
support multiple-address From:, multiple-address Reply-to:, or
both.

> It seems to me that 99% of the time email can be viewed as
> primarily "sender side" or "recipient side".  Eg: Mail goes
> from a sender system to a recipient system.  It is very rare
> (now) that an in-between system would relay mail from an
> untrusted server to another 3rd party server.  If there is a
> relay, it'd have to trust the sender, so it's pretty much
> "sender side," or recognize the recipient "recipient side."
> 
> Assuming that a "sender" has an EAI aware system that is
> completely conformant and "works" with UTF-8 addresses, then
> the entire "sender side" would have to be updated to support
> UTF-8 addresses.  If the sender's relays aren't upgraded, then
> all of their mail ends up being downgraded anyway.
> 
> So I'm wondering how interesting downgrade really is.  If my
> sending system trusts the sender, presumably it could request
> an ASCII address from the trusted sender-side server if it
> needed to downgrade.  (Through some other mechanism, like
> asking the trusted server what other aliases were valid for
> that account using some new, or even proprietary, protocol).

This is the "do we really need downgrade" argument that some of
us have been raising on and off for years.  The answer has never
been "yes" or "no" but "are there enough edge cases (that do not
conform to the analysis) out there to make downgrading useful
anyway, at some cost level".   If anything has changed recently
it has been either

	* a perception that there are fewer relevant cases where
	downgrade is actually needed than was originally
	believed.
	
	* a perception that the costs and risks of having a
	downgrading arrangement (e.g., with double angle
	brackets) make the number of relevant cases needed to
	justify the mechanism much larger than we had originally
	believed.

Of course, both may be true, but would make the case for "no
downgrade" even more open and shut.

> VERY worst case a client (like Outlook) can try to send an EAI
> email.  If it bounces, the client could resend using my ASCII
> address (assuming it is properly configured).  It could even
> do that silently, or with maybe a little warning saying that
> had been downgraded.

That, of course, means that it knows your ASCII address.  Absent
some configuration changes, it might not.

But I think we are in violent agreement, or at least getting
close to it.

    john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 8 Sep 2009 19:26:45 +0000

>	downgrade is actually needed than was originally

For me, this is true
	
>	justify the mechanism much larger than we had originally

And so I think the risk is more interesting now that I think the downgrade is less helpful :)

> some configuration changes, it might not.

If I don't know my ASCII address, I'm not sure how I can send From: EAI, Latn or other syntax :)  So I don't think that's an interesting point.

- Shawn

---

From: Ernie Dainow <[email-masked]>
Date: Tue, 08 Sep 2009 17:29:31 -0400

> support multiple-address From:, multiple-address Reply-to:, or
I don't think it's important if a lot of existing systems do not support 
multiple from addresses.  As non-EAI systems they would only receive 
downgraded messages, which have only a single From address and no Reply-To.

EAI systems, which will receive multiple From addresses, would have to 
support it since it would be part of the EAI spec.

    -Ernie Dainow

---

From: Shawn Steele <[email-masked]>
Date: Tue, 8 Sep 2009 22:04:02 +0000

How do you tell multiple from meaning EAI apart from non-EAI multiple from?

-----Original Message-----
From: Ernie Dainow [mailto:[email-masked]] 
Sent: Tuesday, September 08,  2009 14:30
To: John C Klensin
Cc: Shawn Steele; [email-masked]
Subject: Re: [EAI] double angle brackets

    -Ernie Dainow

---

From: John C Klensin <[email-masked]>
Date: Tue, 08 Sep 2009 19:30:59 -0400

> How do you tell multiple from meaning EAI apart from non-EAI

I don't think you need to.

If I correctly understand the point that Ernie was trying to
make, there are the following cases:

	* non-EAI, support for multiple "From:": will accept
	both and probably discard the i18n email address.
	
	* non-EAI, no support for multiple "From:": Will reject
	either both addresses or the second one.  If one is
	accepted, probably the first one (which should probably
	be the ASCII address) will probably be used.  If both
	are rejected, then nothing much is lost compared to the
	failed downgrade case -- especially if you remember that
	an EAI envelope is unlikely to reach a non-EAI system.
	
	* EAI compliant system: will support multiple "From:"
	because the spec will require it.  

Then the EAI-compliant system will either respond to both
addresses on "reply" or to just the EAI one, but that is
consistent with how much we've historically underspecified what
to do with multiple "From:" situations.

The analysis for Reply-to won't be significantly different, IMO.

   john

---

From: Shawn Steele <[email-masked]>
Date: Tue, 8 Sep 2009 23:49:23 +0000

>	both and probably discard the i18n email address.

That's an interesting assumption.  Presumably I could also reject the entire message since I have no clue how to parse the address.
	
>	an EAI envelope is unlikely to reach a non-EAI system.

Ditto.
	
>	because the spec will require it.  

I still can't tell that they're a pair.  It could be intentionally from 2 different people, one of which has an EAI address, but the other has an ASCII address, and we don't know (or didn't bother to mention) what the EAI address user's downgrade address is.

So an EAI compliant system that "assumed" that "From: Unicode, ASCII" was EAI would not bother replying to the ASCII address, &/or it would 

Besides, if this was reasonable for From, what about To: or CC: or Reply-To:?  Presumably I could mail it to other EAI users as well, so how does a recipient respond?  How would one reliably match up the pairs?

Also, presumably I know my own downgrade address, but am I going to know all of the downgrade addresses for everyone on To: (or hopefully Bcc:) that I'm forwarding a joke to?

I remain unconvinced that this solves enough downgrade problems completely enough to be interesting.

-Shawn

---

From: Ernie Dainow <[email-masked]>
Date: Wed, 09 Sep 2009 08:57:45 -0400

>> 	an EAI envelope is unlikely to reach a non-EAI system.
For non-EAI, downgrade would remove the i18n email address, so From does 
not have multiple addresses.
> So an EAI compliant system that "assumed" that "From: Unicode, ASCII" was EAI would not bother replying to the ASCII address, &/or it would 
This case would have a Reply-To with the EAI address, so the multiple 
 From fields are not used. The reply is composed to the address(es) 
contained in the Reply-To and none of the From headers (see RFC 5322). I 
have tested this with a web mail client and a widely used windows/linux 
email client. From: had two addresses and Reply-To: had a third. When 
replying to such an email, both clients composed mail with To: that 
contained the Reply-To: address only.

Based on this, I expect that most email clients that are upgraded to 
support EAI will actually support the proposal (repeated below) without 
requiring any further work.
    From: EAI-addr, ASCII-addr
    Reply-To: EAI-addr
(where the EAI-addr gets dropped in Downgrade)

> Also, presumably I know my own downgrade address, but am I going to know all of the downgrade addresses for everyone on To: (or hopefully Bcc:) that I'm forwarding a joke to?
As part of the downgrade simplification, we are proposing to drop 
support for downgrade of forward pointing addresses such as To:, Cc:.If 
someone gives you an EAI address, the assumption is that their email 
system can receive EAI email.If their system cannot receive EAI email, 
it is a 'configuration' error on their end and they shouldn't be giving 
out an EAI address until they are ready to receive such mail.

---

From: Shawn Steele <[email-masked]>
Date: Wed, 9 Sep 2009 16:20:45 +0000

> (From: EAI for user 1, ASCII for user 2)
>
>  From fields are not used.

I don't think you understood me correctly.  I'm saying if I wanted the mail to be "From: [email-masked], [email-masked]" when Bill had an EAI address and Steve did not, then it'd be confused with an EAI downgrade format.

I also don't see how Reply-To: helps.  If Reply-To: always has the EAI fallback path, then it can't be used normally.

> out an EAI address until they are ready to receive such mail.

If I send email from an EAI aware account to 10 other EAI aware users, then that's great.  (Like on a list like this).  If someone replies-all, everything works fine.

Now if I add one non-EAI aware user, unless To: & Cc: are fixed, then that person can't reply-all.  In fact I guess all the other addresses have to be dropped for downgrade (or the mail fails)  If they replied-all, they'd get me presumably because of From, but all the other users would fail.  I use Reply-All more than reply.  That's how I replied to this mail in fact.

I'm not sure how much good it does to "fix" from if the others can't be fixed as well.

-Shawn

---

From: Ernie Dainow <[email-masked]>
Date: Wed, 09 Sep 2009 14:26:38 -0400

> I don't think you understood me correctly.  I'm saying if I wanted the mail to be "From: [email-masked], [email-masked]" when Bill had an EAI address and Steve did not, then it'd be confused with a…
Can you actually compose mail like this with your email system? The 
assumption was that multiple From addresses are not widely used and are 
not generally a feature that users can utilize.The proposal was not to 
make the use of multiple From fields available for general use. The From 
headers would be generated automatically specifically to carry the 
downgrade information.
> I also don't see how Reply-To: helps.  If Reply-To: always has the EAI fallback path, then it can't be used normally.
If you don't want the Reply-To to be your EAI address, I don't see any 
reason you couldn't change it, or add additional addresses. This header 
would need to be made available to the user composing the email, with 
its default being the EAI address.
> I'm not sure how much good it does to "fix" from if the others can't be fixed as well.
This is the "triangle" case raised by Harald. As noted, there may have 
to be trade offs in achieving simplification over complexity. This may 
be one of those cases that we have to consider giving up.

---

From: Shawn Steele <[email-masked]>
Date: Wed, 9 Sep 2009 22:31:05 +0000

I'm going to take a couple steps back.  Mostly I'm focusing on the local part of the address, and I think there's a solution to get us unstuck.
 
A lot of background.  At a high level there are really only a few requirements:

* We need Unicode addresses (That's the point of EAI after all :)
* Many people still need a user friendly ASCII address, for the English side of a Japanese business card if nothing else.
* Some sort of "downgrade" must exist for backwards compatibility.  I'm being liberal with the term.  "Downgrade" could be a user trying Unicode and then retrying with ASCII if necessary, or just giving out ASCII because they know EAI won't work in some scenario.

1) We need Unicode addresses.
The Unicode address is solved reasonably well by UTF-8 and the existing RFCs.  UTF-8 also solves a myriad of problems regarding code pages in the rest of the message, but that's not specifically an EAI issue.
 
2) Friendly ASCII addresses.
Friendly ASCII addresses are pretty much a non-issue since mailbox aliases are a common mail feature.  (I get mail to shawnste & Shawn.Steele)  I make the small assumption that aliasing would be extended to Unicode and that aliases will be more common in an EAI world.  This probably primarily impacts mailbox administration tools.

3) Downgrade
So that leaves downgrade.  Downgrade is necessary for a "transition" period, which is probably many years.  Note that there's no technical requirement that downgrade use the friendly ASCII address mechanism.  It is also worth mentioning that downgrade necessarily requires an EAI aware system (even if it's "just" human downgrade, the human has to be aware of the issue) to do the downgrade.

There are also various degrees of downgrade:

* Human-only downgrade with no automated mechanism.  If my mail bounces I try a different address.  Some big problems are that I may not know the other address, I may not know how to send From: an ASCII address, and some systems may accept mail they can't reply to.

* Partially automated downgrade.  This could be something like From:, or the older headers or something else that doesn't provide the same experience in a mixed environment as in a legacy or EAI only environment.  Eg: simple mail may work, but DL's, newsgroups, reply-to, or other cases could get tricky or fail.

Some big problems with partial downgrade are that it is partial, so user's may think mail works, but get unexpected failures in edge cases.  Also (depending on the degree of "partial"), it may fail if the fallback address(es) aren't know or configured correctly.  Also some mail may appear to succeed but may not be able to be replied to.

* Fully automated downgrade.  This would be some magic system that would downgrade all headers so that "everything" worked, including mailing lists, etc.  In addition to the problem of discovering the downgrade address, we've run into numerous technical concerns and edge cases with the solutions investigated so far.

I've been flip-flopping on how much downgrade is necessary.

Human-only downgrade:
I think it's possible that a human-only mechanism may "work", although that may slow EAI adoption significantly.  (If I don't think my EAI address'll work, then I'll only give out my ASCII address, then why bother with EAI?)

Partially automated downgrade:
Personally I don't see much point in partially automated downgrade (folks from Exchange & Outlook agree.)  If my mail to you works, and your replies work, but everything breaks when I send to a DL or newsgroup, then it's almost worse.

Fully automated downgrade:
Fully automated downgrade would be cool, but we've pretty much proved that it's impossible.  Mostly because pairing the addresses breaks down in several cases, or because the syntax to make the pairing explicit breaks downlevel syntax.

A possible solution:

A while back Mark suggested an solution that's been discussed before and I rejected out of hand.  Either I've had a paradigm shift in my thinking or I misunderstood Mark's earlier suggestion, however there is a way that allows fully automated downgrade without breaking any of the other ASCII address alias or UTF8SMTP behavior.  It also only requires updating the EAI aware servers/clients.  Unaware systems wouldn't see anything different, probably even mailing lists.  Upgrade is even supported.

Mark's suggestion (or my modification of it) is something like this:

* Everyone that wants one gets a Unicode address, and a human friendly ASCII alias (if desired).
* When people exchange addresses in written form, they can use Unicode if they think the recipient is EAI aware, or they can use the ASCII address.
* Mail with EAI to EAI and the Unicode address works as spec'd for UTF8SMTP, etc.  No extra fields are sent.
* When a human or form isn't expecting Unicode, the user shares the human-friendly ASCII alias. Then the ASCII to EAI server behaves as normal.  No extra fields are sent.
* When an EAI server (or client) hits an EAI unaware system, the message is downgraded.  Addresses are downgraded by using unmapped punycode, which is algorithmic, so it avoids all the address pairing problems.  An EAI aware client app (like Outlook) could then upgrade the punycode if desired.  Before downgrade a message is identical to the existing practice, just with UTF-8, and is non-breaking.  After downgrade, a message is still identical to existing legacy behavior, so there's no breaking.

By using UNMAPPED punycode (raw ACE encoding), servers can control their own mappings (Turkish I or whatever's interesting to them).  Presumably they'd decode back to Unicode, then do their mapping and routing.  The problems that downgrade had with mailing lists is avoided because all addresses could be downgraded at any point.  Additionally a ASCII-only mailbox user doesn't need an EAI aware server to use EAI because their EAI aware client could do the mapping.

To me, the differences between this and previous punycode proposals (maybe I misunderstood them) are:
a) UTF-8 is preferred, and is the long term form, not punycode.
b) Human-readable ASCII aliases are a preferred method of exchange on the sticky note or whatnot.  (Nobody's going to exchange a punycode address).
c) Punycode is only used for fallback, when UTF8STMP or other EAI protocols aren't recognized.
d) It isn't IDN punycode, but just unmapped Unicode that the server has to decode and map.

The advantages are:
* Users get a Unicode address.
* Users still get a human friendly ASCII address as needed.
* There's a full downgrade capability (with upgrade if desired)
* Interoperability with legacy systems should be very high.
* No problems with pairing of the addresses.
* Works whenever an EAI-aware client/mailbox behave.  Intermediate systems don't impact it, and an EAI aware recipient client can have an EAI experience even if they don’t have an EAI mailbox themselves.
* Only servers/clients upgraded to EAI need to be touched.  Legacy systems in the middle will still behave.  (I'm told by Exchange that it would take decades to upgrade everything).
* Mapping still happens by the mailbox server's rules.
* Algorithmic rules for downgrade mean that downgrade is always possible instead of requiring knowledge of the ASCII alias.
* Senders can downgrade (if the client is EAI aware) even with only a Unicode recipient address, even if their mail server is a legacy server.

The disadvantages are:
* Non-EAI aware recipients would receive punycode addresses instead of human friendly ASCII.  This is mitigated by the fact that most clients display the display name.
* Punycode downgrade requires that EAI aware systems know the punycode mapping.  This is mitigated by reducing the complexity of the other pairing schemes.
* EAI mailbox servers have to unmap the Punycode to Unicode before doing any mapping to find the mailbox.  (Aliases might work in some cases, but you'd still want case mapping probably).
* For systems that don't need EAI (eg: many US mail servers) there's reduced incentive to upgrade to be EAI aware since an EAI aware client is sufficient.  This could delay complete adoption.  (It's also an advantage that could speed up early adoption.)
* Three addresses (Unicode, ASCII, Punycode) instead of 2 (though the Punycode is just an encoded form of the Unicode).

Couple minor comments:
* As I said, the punycode should be unmapped ACE, not IDN form.
* To make that clear, I'd pick a different prefix, or some other signature mechanism.  (My fear is that developers would accidentally call the IDN APIs and get inappropriate mappings).

I know we've discussed Punycode before, but I think the difference is that I'm suggesting it as a last-resort case, not as a replacement for UTF8SMTP, nor a replacement for the human friendly ASCII aliases.  

Please note that I am very much against Punycode as a solution for EAI (eg: replacing UTF-8).  By itself as a long-term solution Punycode has huge problems.  I think it has a place a "hack" to solve downgrade though.

If necessary to further discussion I could put this in draft form.

-Shawn

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 13 Sep 2009 19:02:09 +0200

As I've said several times before, the situation where <<>> has value is 
the situation where an EAI user sends a message to both an EAI user and 
a non-EAI user, and the non-EAI user replies to the message.

Shawn, if you think that in this scenario, it's OK that the recipient 
EAI user does not get the reply from the non-EAI user, please say so 
explicitly.

The one sender / one recipient case works without it.

                       Harald

Shawn Steele wrote:
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>
>

---

From: Andrew Sullivan <[email-masked]>
Date: Sun, 13 Sep 2009 15:54:09 -0400

> EAI user does not get the reply from the non-EAI user, please say so  

As someone who has more or less stopped participating in this WG, I
just want to note that the above amounts to a serious and important
restriction on "backward compatibility".  If the recipient EAI user is
the loser from the non-EAI user, this really means that non-EAI users
(i.e. the non-adopters) lose something, _but also_ that the EAI users
(the adopters) lose something.

I am completely uninformed about the values in play in that trade-off,
and I have no opinion at all on how it ought to go.  But it's a pretty
serious limitation of interoperability between strictly-822-derived
systems and EAI, and it seems to me the WG has to have a strong
statement of the ways that interoperability is expected to succeed and
the ways it is expected to fail.  (Note that I regard "often" as an
acceptable answer.  But clarity is really important.)

A

-- 
Andrew Sullivan
[email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 14 Sep 2009 11:50:22 +0200

The overarching issue found with variants of this proposal previously 
rejected was:

Can we guarantee (for a sufficiently strong version of "guarantee") that 
there are no valid mailboxes that "just happen to look like" the 
Punycoded strings?

If not, we have to stick with the "don't mess with the left-hand side" 
principle and not let the specs define any upgrading, ever. (But then, 
if all home systems for the email recognized the Punycoded form, there 
isn't much reason to upgrade. But (3rd level) then that's a burden 
they'll have to carry around roughly forever..... more levels?)

IDNA made the leap of faith that no existing application would be 
inconvenienced if we assumed that no domain name label starting "xn--" 
existed with any other use than IDNA.

What assumption can we make?

                          Harald (who was around when X.400 LHS encoding 
met Sendmail).

---

From: Shawn Steele <[email-masked]>
Date: Mon, 14 Sep 2009 16:47:22 +0000

> What assumption can we make?

It seems unlikely that mailboxes start with xn--, though other values could maybe be used.  Should such an address already be used, I suggest that it is esoteric enough that such existing usage shouldn't penalize the rest of the Internet moving forward.  Additionally, it'll probably still "work", though some clients might render it funny.

My earlier objections had nothing to do with "does a mailbox looking like that already happen to exist", but rather:
* Punycode is NOT human-friendly.
* UTF-8 on the wire is much cleaner.

My shift in thinking is that UTF-8 on the wire is still strongly preferred, and that a human-friendly address is also required.

-Shawn

---

From: "YAO Jiankang" <[email-masked]>
Date: Tue, 15 Sep 2009 10:23:11 +0800

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: "Harald Alvestrand" <[email-masked]>
Cc: "Mark Davis ⌛" <[email-masked]>; <[email-masked]>
Sent: Tuesday, September 15, 2009 12:47 AM
Subject: Re: [EAI] Thinking about requirements / downgrade

> It seems unlikely that mailboxes start with xn--, though other values could maybe be used.  Should such an address already be used, I suggest that it is esoteric >enough that such existing usage shouldn't penalize the rest of the Internet moving forward.  Additionally, it'll probably still "work", though some clients might render >it funny.

assumption is just a assumption. never comes true. if we move forward without considering the  "esoteric" one,
the protocol might not run properly. if you have a utf-8 address and produce a automaic ACE address which happens to be there already, what will happen?
That means that the different accounts but equalent will  belong to the different users.
 I remember that why this WG does not choose the automaic algorithm to produce the alt-address has been discussed sevral times. It seems that we return again.

> * Punycode is NOT human-friendly.
+1

> * UTF-8 on the wire is much cleaner.

+1

> My shift in thinking is that UTF-8 on the wire is still strongly preferred, and that a human-friendly address is also required.

+1

the problem is that we can not find the human-friendly address easily.

alias way might be a solution. but it seems that alias way might have some other problems. 

Yao Jiankang

---

From: Shawn Steele <[email-masked]>
Date: Tue, 15 Sep 2009 16:42:54 +0000

> the protocol might not run properly. if you have a utf-8 address and produce a automaic ACE address which
> happens to be there already, what will happen?

Worst case I'd imagine that it'd get converted to Unicode for display.  It should still "work"  I've never seen an address that looks anything like xn--, and while that isn't saying much, it seems that it'd be worth the risk to whatever does use xn-- to make progress towards Unicode email.

> That means that the different accounts but equalent will  belong to the different users.

That's doesn't seem possible.  If we used xn--, then it'd only be subject tothe different accounts condition if the legacy xn-- mail system were upgraded to support EAI.  In that case the administrator would necessarily either prohibit new mailboxes that matched xn--, or change the existing xn-- addresses.

-Shawn

---

From: Ernie Dainow <[email-masked]>
Date: Tue, 15 Sep 2009 18:22:01 -0400

Harald Alvestrand wrote:

The
overarching issue found with variants of this proposal previously
rejected was:

Can we guarantee (for a sufficiently strong version of "guarantee")
that there are no valid mailboxes that "just happen to look like" the
Punycoded strings?

I'm not sure it is necessary for a standard to guarantee this. I think
it is just the responsibility of email administration to avoid name
collisions. Currently, when a new email address is requested, the email
administrator for the domain on which the name is requested (or email
admin software) will not grant the email address if

1. the address (local name) is assigned to someone else on the domain.

2. the address is already in use as an alias on another email account
on the domain.

Suppose EAI used automatic punycode email addresses, where punycode
conversion is applied separately to each side of the @. Then email
administrators on EAI systems just have the following additional
verification to do on new email account requests:

3. generate the punycode of the new address (local name) and apply
tests 1 and 2 above to make sure the punycode name is not in use on the
domain.

Since only owners of a domain can create email addresses, name
collisions cannot occur unless administrators fail to check for
uniqueness before granting new email addresses. That requirement is no
different from today.

Another concern about using ACE conversions in EAI was the issue of
punycode addresses "leaking" into the user environment. Just to remind
people how ugly this can be, here is a conversion of one of the EAI
addresses used in one of the downgrade tests.

<[email-masked]>

I think receiving a lot of email with long unrecognizable ACE
conversions might be incentive for people to upgrade to EAI.

Email administrators can avoid local name collisions as outlined above.
This assumption is also important to EAI to avoid collusions on the
domain name.

---

From: "YAO Jiankang" <[email-masked]>
Date: Wed, 16 Sep 2009 11:24:48 +0800

----- Original Message ----- 
From: "Shawn Steele" <[email-masked]>
To: "YAO Jiankang" <[email-masked]>; "Harald Alvestrand" <[email-masked]>
Cc: "Mark Davis ⌛" <[email-masked]>; <[email-masked]>
Sent: Wednesday, September 16, 2009 12:42 AM
Subject: RE: [EAI] Thinking about requirements / downgrade

> Worst case I'd imagine that it'd get converted to Unicode for display.  It should still "work"  I've never seen an address that looks anything like xn--, and while >that isn't saying much, it seems that it'd be worth the risk to whatever does use xn-- to make progress towards Unicode email.

maybe, xn-- isn't saying much for english.  but in Chinese Pingyin, XN can standard a city name "Xi Ning" which is a capital city of province.

If I am from Xi Ning, so I can name my mailbox such as Xn--yao, while Xn standards for the city name, -- as the separator, and yao as my family name.

so [email-masked]  has a lot of meaning. you might not see such name in your life, but you can not prove that it might be there.

> That's doesn't seem possible.  If we used xn--, then it'd only be subject tothe different accounts condition if the legacy xn-- mail system were upgraded to >support EAI.  In that case the administrator would necessarily either prohibit new mailboxes that matched xn--, or change the existing xn-- addresses.
>

yes, the administrator of the email accounts can  prohibit new mailboxes that matched xn--. but you can not prohibit the email user client and other mail servers to do so. The implementor of the EAI protocols might implement the automatic algorithm to produce the alt-address. How does the implementors know the exception case? if you have a utf8 address which happens to map to the other xn-- address instead of  the xn-- address produced by automatic algorithm, how do the MUA and other SMTP servers know it?  these MUA or SMTP servers might still think that  xn-- address produced by automatic algorithm is still a right one for mapping to the uf8 address.

Yao Jiankang
CNNIC

---

From: Shawn Steele <[email-masked]>
Date: Wed, 16 Sep 2009 05:04:56 +0000

> so [email-masked]  has a lot of meaning. you might not see such name in your life, but you can not prove that it might be there.

Of course any string might be used, but the probability seems low.  An administrator could also prohibit it (sure there could be a display glitches for EAI clients).  If we were concerned about xn--, it could be q.-_. or something instead.  (Sure, that could also be in an address, but this is stretching it a bit).  I don't think this need block us.  The risk of breaking a very few addresses is small compared to the functionality that is gained.

Again: I'm only (re)proposing this as a last-resort fallback, not to replace a human friendly alias.

- Shawn

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 16 Sep 2009 08:16:29 +0200

Ernie Dainow wrote:
> collisions. Currently, when a new email address is requested, the 
> email administrator for the domain on which the name is requested (or 
> email admin software) will not grant the email address if
My worry is about what happens if the recipient tries to decode the 
left-hand side as if it was a Punycoded name when it is not. Unless one 
can tell the difference between those domains that use Punycoded LHS and 
those that do not, a recipient will apply the Punycode decoding to 
left-hand sides indiscriminately. This can lead to similar effects as 
the "Bush hid the facts" bug (see 
http://en.wikipedia.org/wiki/Bush_hid_the_facts for details).

                         Harald

---

From: Shawn Steele <[email-masked]>
Date: Wed, 16 Sep 2009 06:25:27 +0000

> http://en.wikipedia.org/wiki/Bush_hid_the_facts for details).

Yup.

But I think it's worth the improvement for behavior in the 99.999% case.  Maybe we could get statistics for how many account names actually look like this.  Also we could pick something really wierd (like the q.-_.z or something).
 
IDN had the same issue, but seems to have succeeded (at least in that respect.)

I don't think we can improve the "other" fallback mechanism sufficiently to be interesting.  We've pretty much proven that pairing of addresses only works in limited scenarios and breaks pretty quickly when deviating from the simple scenarios.

-Shawn

---

From: Bill McQuillan <[email-masked]>
Date: Tue, 15 Sep 2009 23:48:30 -0700

> IDN had the same issue, but seems to have succeeded (at least in that respect.)

Unfortunately, this is a red herring. IDN had a much smaller universe to be
compatible with. Hostnames are generally not allowed to have more than one
hyphen in a row. Thus xn--... should never collide with a hostname, so they
only had to discover if that form was found in the few other DNS usages.

-- 
Bill McQuillan <[email-masked]>

---

From: Ernie Dainow <[email-masked]>
Date: Wed, 16 Sep 2009 08:56:11 -0400

>> http://en.wikipedia.org/wiki/Bush_hid_the_facts for details).
Local names are not decoded until they arrive at the destination domain. 
At that point, if a mailbox exists for the local name, it is delivered. 
On EAI systems, if that mailbox does not exist, then the name is decoded 
and a second delivery is attempted to the UTF8 mailbox. If the checks 
outlined earlier are followed when creating new mailboxes, delivery will 
never go to different users for these two cases.

In fact, email administration on an ACE-based EAI should assign the 
punycode translation of local addresses as an alias for the UTF8 
address. So delivery of either punycode or UTF8 goes to the same mailbox 
without the above two step delivery process.

> But I think it's worth the improvement for behavior in the 99.999% case.  Maybe we could get statistics for how many account names actually look like this.  Also we could pick something really wier…
We have to be careful about about making a statistical decision. If 
there is any chance that mail gets delivered to the wrong place, it is a 
serious breach of privacy. This takes precedence over other goals such 
as "works 99.999% of the time". This is a more serious failure than 
delivery failures in the triangle case, which is what we are basically 
trying to solve with an ACE-based downgrade.

    -Ernie

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 Sep 2009 09:46:04 -0400

--On Wednesday, September 16, 2009 08:16 +0200 Harald Alvestrand
<[email-masked]> wrote:

> Ernie Dainow wrote:
>> 
>> Harald Alvestrand wrote:
>>> The overarching issue found with variants of this proposal
>>> previously  rejected was:
>>> 
>>> Can we guarantee (for a sufficiently strong version of
>>> "guarantee")  that there are no valid mailboxes that "just
>>> happen to look like" the  Punycoded strings?

>> I'm not sure it is necessary for a standard to guarantee
>> this. I think  it is just the responsibility of email
>> administration to avoid name  collisions. Currently, when a
>> new email address is requested, the  email administrator for
>> the domain on which the name is requested (or  email admin
>> software) will not grant the email address if 1. the address
>> (local name) is assigned to someone else on the domain. 2.
>> the address is already in use as an alias on another email
>> account  on the domain.

> My worry is about what happens if the recipient tries to
> decode the left-hand side as if it was a Punycoded name when
> it is not. Unless one can tell the difference between those
> domains that use Punycoded LHS and those that do not, a
> recipient will apply the Punycode decoding to left-hand sides
> indiscriminately. This can lead to similar effects as the
> "Bush hid the facts" bug (see

Let me say what Harald, Bill, and others have said from a
different perspective and a little more strongly.  

Once upon a time, before the web and before we could assume that
almost every node connected to the network was well-connected
(end to end) with just about every other node,  the most common
method for transmitting command instructions from one host to
another (or a device connected to another) involved email
messages that could be received, stored, and forwarded by
intermediaries.  In many cases, local-part addresses were used
to encode information and to secure and/or authenticate the
commands with what amounted to shared-secret encryption and
anti-reply coding.

It may be worth noting that some very similar techniques have
emerged in the "delay-tolerant networking" area.  If one cannot
do end-to-end (or end-to-closely-coordinated-surrogate)
handshakes, there aren't that many options.

Those applications led to some email local parts that were truly
bizarre if one assumed that local parts are some stable
variation on a user name or ID/handle.  Worse, not only are
strings that look remarkably like IDNA ACEs plausible, it is
impossible to do the equivalent of sweeping the DNS tree or
inquiring of top-level registries to give some confidence that
choices are being made that don't conflict with existing uses
(such sweeps and questions were part of the process of selecting
the ACE prefix, and some candidate prefixes were eliminated on
the basis of existing usage).

There is at least anecdotal evidence that methods using local
parts to transmit command and control information (either coded
into the local-part or using the local part to authenticate the
message payload) have not dropped out of use, even though their
presence as a percentage of messages on the Internet has
certainly dropped.

For better or worse, little of this stuff is written down in,
e.g., the RFC series.  At least some of the reason is that many
similar, but certainly not identical, mechanisms have been
invented over and over again in different places and rely on
"security by (often rather extreme) obscurity", rather than
cryptographic integrity... but that doesn't make them less
problematic from our standpoint.

Those mechanisms are among the reasons why the email community
established the very strong "only the receiving system can
interpret a local part and it MUST NOT be messed with in
transit" rules that appear in RFC 5321 and its predecessors and
has been unwilling to change them.  IMO, even if this WG reached
the conclusion that such tag-identified encodings were a good
idea, the proposal would be destroyed on IETF Last Call.

Finally, while such address-encoded information is certainly
still in use, the odds of showing up on, e.g., a search through
Google databases are just about zip --the data aren't archived,
the messages don't refer to each other, many of the addresses
are one-time-use, and natural language processing mechanisms
don't work because these addresses are about as far from natural
language as one can get.  And, finally, anyone who can't
dedicate a host, or at least a domain or primary address and
family of subaddresses, under their control to the particular
control tasks at issue is just not in the game... so the more
relevant and sophisticated methods are not using public/free
email systems of the AOL/ Hotmail/ Gmail/ Yahoo/ etc persuasions.

It just isn't a worthwhile path to try to follow.

      john

---

From: John C Klensin <[email-masked]>
Date: Wed, 16 Sep 2009 15:31:42 -0400

--On Wednesday, September 16, 2009 06:25 +0000 Shawn Steele
<[email-masked]> wrote:

>...
>> left-hand sides indiscriminately. This can lead to similar
>> effects as the "Bush hid the facts" bug (see
> But I think it's worth the improvement for behavior in the
> 99.999% case.  Maybe we could get statistics for how many
> account names actually look like this.  Also we could pick
> something really wierd (like the q.-_.z or something).  

Shawn,

I addressed most of this in a note I sent a few hours ago and
that probably crossed yours in the email.  I won't repeat myself
but, as others have also pointed out, the issues here are much
more complicated than the IDN ones.  In particular, one of the
issues that has generated a lot of traffic on the IDNAbis list
-- the desire to have upper-case letters that are properly
majuscules, not capitals, handled differently depending on
context-- gets a rather different answer here because of the
"local parts get interpreted only by the delivery system"
principle.  For example, while the DNS guarantees
case-insensitive matching for ASCII labels, there are absolutely
no guarantees in email about case sensitivity or insensitivity.
I actually works on a system some years ago in which traffic
for, e.g., [email-masked] was treated as mail traffic for joe
while traffic for [email-masked] was treated as a command
stream for one of joe's processes or daemons.  The idea didn't
last very long for security reasons --and got replaced by
members of the family of the messy strings I described-- but you
get the idea.

I have to agree with what I think was at least the intent of a
recent note of Harald's -- each of these suggestions needs
careful analysis and explanations of what cases exist, which
ones "work" (and what that implies) and which ones fail and in
what ways.   Without that sort of analysis, asserting that a
particular approach will solve some large percentage of cases
does not, IMO, help us very much (and doing so in a large stream
of messages helps, again, IMO, even less).

That said, let me state a question that I think might be
illuminating, even though I'm not sure whether I think what it
implies is acceptable:

Suppose we were willing to start with "the number of cases in
which downgrading is needed will be fairly few if sending
systems are carefully designed and configured and receiving
systems and their MX intermediaries are configured correctly"
(you, Ernie, Jiankang, and others, including myself, have all
said variations on that).  We also assume that the number of
cases in which it is needed will go down over time, as EAI
deploys more broadly and as various actors understand the
configuration issues better.  Could we then say "whatever
downgrading is needed is about getting messages delivered anyway
but, if downgrading is applied, we no longer expect 'replies' to
work reliably"?  On the one hand, that is a fairly big step.
But it has a lot of precedent (some intentional and some not) in
email systems.  And it would sweep away a lot of issues,
including Harald's exemplar triangular case.

Generating a header/marker to warn the recipient that the
headers may not show all recipients and that "reply all" may not
reach everyone to whom the original message was sent is pretty
easy (reexamining the idea of using group syntax might work well
here).  As far as addresses are concerned, it turns the entire
downgrade story into an envelope one, where we don't have syntax
problems (double angle brackets or otherwise).  It minimizes
long-term damage for short-term/ transitional gains and gives
the various humans involved an incentive to upgrade.   And it
makes things completely predictable, which you've been asking
for, as long as one is willing to count "if you reply to this,
it probably won't work as you expect, but no one prevents you
from trying" as predictable behavior (I actually do, YMMD).

I note that there were many examples and bad jokes about both
X.400 and early versions of Exchange whose substance was that
the only reliable "reply" mechanism involved picking up a
telephone, so, in practice (and perhaps even in theory), we've
never had nearly the same level of expectation about replies
working that we've had about deliveries getting through.

But it is a big step.  Is it one we might be willing to consider?

     john

---

From: Shawn Steele <[email-masked]>
Date: Wed, 16 Sep 2009 20:37:09 +0000

Re: casing, etc.  Unlike IDN, I tried to make clear that I'm suggesting no mapping be done prior to encoding, it's merely an encoding.  The server side would still do whatever it wants with casing, which means that it MUST decode it to Unicode, then do whatever mapping it wants.  (Differently cased shäwn and SHÄWN would get different encodings and could not be compared in the encoded form.)  That makes matching of the ACE forms more difficult, but avoids problems with casing rules being interpreted differently.

Re: partial downgrade.  I cannot imagine how allowing mail to be delivered, but not providing for replies to work is helpful.  There’s no way for a sender to know if they were ignored or just had a mail mixup.  I don’t think that the “reply” anecdote you describe is appropriate in the modern world.  We have a much more reliable system now and reply-all is expected to behave.

I think that either we need to have no downgrade at all, or we must have nearly-complete downgrade.  I also think that nearly-complete downgrade would require an algorithmic technique as we’ve pretty much proven that pairing (by <<>> or other means) breaks pretty rapidly.

I don’t think the downgrade technique discussion blocks any of the non-downgrade specs from proceeding.  Obviously EAI doesn’t “need” downgrade, it just might ease the transition.

-Shawn

-----Original Message-----
From: John C Klensin [mailto:[email-masked]]
Sent: Poʻakolu, Kepakemapa 16, 2009 12:32 PM
To: Shawn Steele; Harald Alvestrand; Ernie Dainow
Cc: [email-masked]; [email-masked]
Subject: Re: [EAI] Thinking about requirements / downgrade

> IDN had the same issue, but seems to have succeeded (at least

> in that respect.)

Shawn,

get the idea.

implies is acceptable:

     john

---

From: Shawn Steele <[email-masked]>
Date: Wed, 16 Sep 2009 23:19:15 +0000

> never go to different users for these two cases.

That's where there's confusion.  There is another scenario where an EAI aware system mails to a non-aware system, however the client on that side is EAI aware.  In that case, it is possible that the client would want to upgrade the message for display.

I'm going to describe the thinking of a vendor trying to create an end-to-end solution.  Since beating around the bush didn't work, I will name Microsoft products explicitly, (That doesn't mean we're doing this, although it's pretty obvious we're thinking about it.)  Please replace the Microsoft products with your favorite vendor(s) because other vendors have the same problems and have thought about this as well.
 
As a business, we'd like to provide EAI solutions.  It'd be cool & solve a real customer need.  However people and businesses don't upgrade Exchange and Outlook every day.  Also we have Live mail and web based front ends to consider.

Presumably at least some of our customers want international addresses.  We're pretty much certain that they'll want reliable email service as well.  If an Exchange site is going to upgrade to EAI, they're going to want to make sure that it plays well with other Exchange sites, with Outlook clients, etc.  In some cases an entire enterprise is using Microsoft products; Outlook clients talking to Exchange servers, etc.

If our customers want to upgrade their Tokyo office to support EAI but can't afford to update their US servers right now, what would a Microsoft solution look like?
 
The other downgrade techniques that have been proposed fail in far too many cases for us to have satisfied customers.  It doesn't matter if 90% of their mail succeeds, if 1% fails without any warning.  Even staying on the same vendor's products won't help because of the other downgrade techniques have fundamental flaws.

In that case, a workaround could be to use an algorithmic encoding.  Then customers using Outlook or Live Mail could talk to these improved EAI Exchange servers and interoperate, regardless of the infrastructure in-between.  So this would provide a fairly robust end-to-end solution for customers that used our products.  It would probably also help our customers is a few cases even when some of the infrastructure was on other platforms.

In an EAI-supporting Exchange/Outlook environment it'd be really nice if non-Microsoft products interoperated as well, but even if mail to a different platform failed, at least the boundary would be pretty clear.  Customers could still have solutions that worked within their enterprise, even if external mail ran into the no-downgrade or partial-downgrade experience when talking to non-Microsoft products.

As I mentioned, we're not the only ones thinking along these lines, replace our products with our competitors' above.  If we can provide a more robust solution with our products, then we're inclined to consider it.  Our competitors are.  I'd prefer a solution that interoperates as much as possible though.

When considering a solution that works within our product line, the problem is simpler.  We don't have to worry about strange xn-- codes running scripts because our stuff doesn't do that.  Neither do most of our competitors' products.  I can readily see where an old Unix? box that ran scripts from email addresses wouldn't want to use this mechanism, but they don't have to.  I'm suggesting it as an optional worst-case fallback.

There's nothing in the current EAI RFCs that prohibits algorithmic generation of an ASCII address, address generation is left to the server.  I imagine that fallback would look something like:

1) Use EAI
2) If I can't use EAI, try to use human-readable ASCII (assuming I know enough)
3) If I can't do that, use the worst-case, last-resort Punycode mechanism.  Like for fields that downgrade doesn't address fully.

I get that algorithmic downgrade might not work for all environments, but I think that when vendors look at their offerings they'll try to provide a complete end-to-end solution.  In most cases the algorithmic technique will work.  In other cases it shouldn't be used.

I've heard feedback that xn-- or any other prefix is going to trip up some esoteric part of the system (you don't really need to query every mail server to realize that z.-_., or even xn--, is rarely encountered in email).  I'm not suggesting that this is a perfect solution for every environment, only that it will work in many environments.

Unless downgrade can be solved perfectly some other way, I believe that some software vendors will implement such a last-resort algorithmic solution.

1) Are there any places where algorithmic ASCII addresses that will fail worse than the non-delivery case or partial-support case?  I can't think of any:
* If I support Punycode mailboxes, then I can assign them in a way that aliasing isn't a problem.
* If you don't support Punycode mailboxes, but my mailer tries to send mail that way, then either:
	a) You don't get the mail because there's no such mailbox,
	b) The mail goes somewhere else, like to a different mailbox.  This is similar to a misdirected fax or a typo in an email address today.  I can't trust [email-masked] because I might have got it wrong and it was really [email-masked].  If I've never mailed there before there's no guarantee it'll behave.  If I trust the address as working, then I know it works.
	c) Apparently it could also run a job somewhere or do something else interesting.  In that case it doesn't really matter how I sent mail to that address since I could've done it directly anyway.  I'm not sure how this makes the punycode technique any riskier, that system already has a pretty serious security problem.

2) I realize it's not the right solution for everyone, but is there an objection to me creating an OPTIONAL last-resort draft for applications that think they need it?  Then even if those applications are being stupid, at least they'll all be stupid in a predictable and consistent manner.

-Shawn

---

From: Bill McQuillan <[email-masked]>
Date: Wed, 16 Sep 2009 16:58:27 -0700

> That's where there's confusion.  There is another scenario where an
> EAI aware system mails to a non-aware system, however the client on that
> side is EAI aware.  In that case, it is possible that the client would
> want to upgrade the message for display.

> I'm going to describe the thinking of a vendor trying to create an
> end-to-end solution.  Since beating around the bush didn't work, I will
> name Microsoft products explicitly, (That doesn't mean we're doing this,
> although it's pretty obvious we're thinking about it.)  Please replace
> the Microsoft products with your favorite vendor(s) because other vendors
> have the same problems and have thought about this as well.

<snip>

> 1) Are there any places where algorithmic ASCII addresses that will
> fail worse than the non-delivery case or partial-support case?  I can't
> think of any:
> 	b) The mail goes somewhere else, like to a different mailbox.  This
> is similar to a misdirected fax or a typo in an email address today.  I
> can't trust [email-masked] because I might have got it wrong and it
> was really [email-masked].  If I've never mailed there before
> there's no guarantee it'll behave.  If I trust the address as working,
> then I know it works.
> 	c) Apparently it could also run a job somewhere or do something else
> interesting.  In that case it doesn't really matter how I sent mail to
> that address since I could've done it directly anyway.  I'm not sure how
> this makes the punycode technique any riskier, that system already has a
> pretty serious security problem.

> 2) I realize it's not the right solution for everyone, but is there an
> objection to me creating an OPTIONAL last-resort draft for applications
> that think they need it?  Then even if those applications are being
> stupid, at least they'll all be stupid in a predictable and consistent
> manner.

It seems to me that the potential damage only occurs when an algorthmically
downgraded address is received by an MX in which the algorithmically
modified local part might be interpreted in some fashion other than as a
downgraded address. This is only for addresses local to that domain, since
all others would be treated as opaque by an eai-UNaware server.

I think that this implies that, if a common, algorithmically determined
aliasing system is allowed, then when a domain decides to implement ima, it
must examine such conventions within itself and make sure that it does not
use any that would conflict with the algorithm.

This seems to narrow the area of conflict somewhat.