Subject: [EAI] mail bounce versus punycode conversion

From: Ernie Dainow <[email-masked]>
Date: Mon, 12 Jul 2010 17:17:16 -0400

A message to mailbox <ASCII@non-ASCII> will bounce if the UTF8SMTPbis 
extension is not supported by an MTA. According to rfc5336bis, this will 
not be done by an MTA. This is not obvious behavior, since it is fairly 
trivial to do a punycode conversion of the domain part so that the 
address is all ASCII. Many users are familiar with web browsers that do 
this conversion and may expect similar behavior in email.

To reinforce that this is a failure case, we should add the following 
example to section 4.4 of rfc5335bis-00.

    <ASCII@non-ASCII>
         ; message will bounce if UTF8SMTPbis extension is not 
supported. An MTA will not convert
         ; the domain part to ASCII, but this may be done by an MUA or 
submission server (MSA).

-Ernie

---

From: John C Klensin <[email-masked]>
Date: Tue, 13 Jul 2010 01:53:34 -0400

--On Monday, July 12, 2010 17:17 -0400 Ernie Dainow
<[email-masked]> wrote:

> A message to mailbox <ASCII@non-ASCII> will bounce if the
> UTF8SMTPbis extension is not supported by an MTA. According to
> rfc5336bis, this will not be done by an MTA. This is not
> obvious behavior, since it is fairly trivial to do a punycode
> conversion of the domain part so that the address is all
> ASCII. Many users are familiar with web browsers that do this
> conversion and may expect similar behavior in email.

Speaking for myself only, this is actually one of the difficult
cases I was sort of alluding to in an earlier note today.  

Please review draft-iab-idn-encoding (-03 was posted yesterday).
If one has a system in which any string in a multi-target name
resolution slot is passed to a common name resolution interface,
then it probably will not be possible to prevent an
<ASCII@non-ASCII> string from being resolved, with or without
EAI.  Similarly, if a system identifies the domain part as being
an IDNA-aware domain name slot, <ASCII@non-ASCII> might be
turned into <[email-masked]....> before the mail-specific
software actually gets its hands on it and subsequently treated
as an all-ASCII address.

It is plausible to put <ASCII@non-ASCII> outside the scope of an
EAI standard if the extension isn't specified.  That follows
from the observation that, if the extension is not specified,
everything (ASCII and non-ASCII alike) is outside the scope of
the standard.  But requiring message rejection in one of those
cases is not reasonable: servers who do not announce the
extension are bound only by the rules of RFC 5321 and friends.
Those specifications are specific about what must be accepted
but deliberately not specific about what must be rejected.

> To reinforce that this is a failure case, we should add the
> following example to section 4.4 of rfc5335bis-00.
> 
>     <ASCII@non-ASCII>
>          ; the domain part to ASCII, but this may be done by
> an MUA or submission server (MSA).

The strongest thing that can be said, ideally with a citation of
5321, is something like:

    <ASCII@non-ASCII>
         ; if theUTF8SMTPbis extension is not supported,
         ; the MTA is required to be SMTP [RFC 5321]-compliant
         ; only.  Such an MTA is not required to 
         ; convert the domain part to from U-label form to
         ; A-label form.  If it does not, the message may be 
         ; rejected or simply not delivered.  Similarly,
         ; if theUTF8SMTPbis extension is not supported, an
         ; MUA or submission server (MSA) may perform conversions
         ; to A-label form, but is not required to do so.

Note that the IRI spec makes this even more difficult because 

			mailto:ASCII@non-ASCII

is at least as likely to be converted into

            mailto:ASCII@%-encoded-UTF-8

as into

            mailto:ASCII@A-label...

john

---

From: Ernie Dainow <[email-masked]>
Date: Tue, 13 Jul 2010 08:59:05 -0400

On 7/13/2010 1:53 AM, John C Klensin wrote:
>> A message to mailbox<ASCII@non-ASCII>  will bounce if the
> an IDNA-aware domain name slot,<ASCII@non-ASCII>  might be
> turned into<[email-masked]....>  before the mail-specific
> It is plausible to put<ASCII@non-ASCII>  outside the scope of an

According to section 3.2 in  5336bis-00, <ASCII@non-ASCII> is an 
internationalized address and can't be submitted to an MTA that does not 
support UTF8SMTPbis. An MTA cannot rewrite such a header so the message 
must be rejected. If we want to allow an MTA to do punycode conversion, 
the following paragraph in 5336bis-00 needs to be revised.

1.  If and only if the SMTP client (sender) is a Message Submission
        Server ("MSA") [RFC4409], it MAY, consistent with the general
        provisions for changes by such servers, rewrite the envelope,
        headers, or message material to make them entirely ASCII

---

From: Joseph Yee <[email-masked]>
Date: Tue, 13 Jul 2010 09:51:51 -0400

> software actually gets its hands on it and subsequently treated

Speaking for myself and my past experience on <ASCII@non-ASCII> ...

Some MUAs, with plugin, converts <ASCII@non-ASCII> to <ASCII@A-label>  before submitting.  Some would display <ASCII@A-label> to <ASCII@non-ASCII>.
MTAs 'listen' domain in ASCII and A-label format only.  

So from now to at least in early phase when EAI introduced, there may be some mail addresses in <ASCII@non-ASCII> format but not running under UTF8SMTPbis.  And if EAI aware users with EAI aware MUA and EAI aware MTA send message to those mail addresses, the message would be rejected.  I would guess this group would be early adopters of EAI, but never assume this is the case.

Personally I see the domain convert as 'last resort' hoping to deliver the email.  I don't think it as must-do, but I hope it does not get into must-not-do.  

>         ; to A-label form, but is not required to do so.

I like this text.  Address the domain convert as optional.  We would need to make sure the new text in both 5335bis section 4.4 and 5336bis section 3.2 (last paragraph) be consistent.

Joseph

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Tue, 13 Jul 2010 12:17:11 -0400

--On Tuesday, July 13, 2010 08:59 -0400 Ernie Dainow
<[email-masked]> wrote:

>...
>> It is plausible to put<ASCII@non-ASCII>  outside the scope of
>> an EAI standard if the extension isn't specified.  That
>> follows from the observation that, if the extension is not
>> specified, everything (ASCII and non-ASCII alike) is outside
>> the scope of the standard.  But requiring message rejection
>> in one of those cases is not reasonable: servers who do not
>> announce the extension are bound only by the rules of RFC
>> 5321 and friends. Those specifications are specific about
>> what must be accepted but deliberately not specific about
>> what must be rejected.
>...
>> The strongest thing that can be said, ideally with a citation
>> of 5321, is something like:
>> 
>>      <ASCII@non-ASCII>
>>           ; the MTA is required to be SMTP [RFC
>>           5321]-compliant ; only.  Such an MTA is not
>>           required to
>>           ; MUA or submission server (MSA) may perform
>>           conversions ; to A-label form, but is not required
>>           to do so.
>>    
> 
> According to section 3.2 in  5336bis-00, <ASCII@non-ASCII> is
> an internationalized address and can't be submitted to an MTA
> that does not support UTF8SMTPbis. An MTA cannot rewrite such
> a header so the message must be rejected. If we want to allow
> an MTA to do punycode conversion, the following paragraph in
> 5336bis-00 needs to be revised.
>...

I obviously was not clear; let me try again.  Nothing this WG
does can change basic requirements or flexibilities in 5321 or
5322 except in the context of systems advertising the EAI
extensions.  Doing so is not only out-of-charter, it essentially
would fork the SMTP protocol rather than extending it.  The
Experimental documents may have been sloppy about how things
were worded in this area; the standards-track ones must not be.

To try to break this down a bit...

	(1)  An SMTP server that does not announce UTF8SMTPbis
	cannot be constrained by us.  If it encounters non-ASCII
	domain labels and wants to do conversions into A-label
	form, we cannot prevent it from doing that, nor can we
	prevent it from looking up the UTF-8 string, normalized
	or not, in the DNS.  If it wants to reject the message
	instead, we cannot prevent that either.  We also cannot
	prevent it from simply turning off all of the high-order
	bits and treating the resulting address as ASCII.  That
	would be dumb, but it isn't our problem and there is
	some history of exactly that behavior.  And, finally, if
	it sees a domain name with A-labels in it, it is
	expected to treat those as ordinary LDH-style domain
	names, without any special handling, unless it, or its
	supporting libraries, are IDNA-aware.  If it is
	IDNA-aware, it might make some tests.
	
	(2) We could make a rule that says that a server that
	announces UTF8SMTPbis is required to recognize and
	reject A-labels for either the case that the local-part
	is ASCII, the case where it isn't, or both.  I would
	recommend against doing so because such a rule would
	require that an EAI-conformant server be less flexible
	that one that is not EAI-conformant.   That, in turn,
	could act to discourage people from implementing/
	announcing UTF8SMTPbis or to encourage them to ignore
	us.  Interoperability would not be improved by such a
	prohibition.

The text in 5335bis and 5336bis must be consistent with the
principle above: no restrictions on servers that don't announce
the extension.    It is up to the WG how to handle the second
case but I would urge that we be very careful about prohibitions
-- MUST or SHOULD reject language-- that are not strictly needed
for interoperability.

     john

---

From: Ernie Dainow <[email-masked]>
Date: Tue, 13 Jul 2010 14:16:42 -0400

John C Klensin wrote:
> extensions.  
Understood. The context of the above paragraph from 5336bis-00 section 
3.2, is an MTA that supports UTF8SMTPbis. If such a server is allowed to 
submit <ASCII@non-ASCII> to another MTA that does not support 
UTF8SMTPbis, then we cannot place any new rules on that server, and it 
is bound only by earlier SMTP standards. But my interpretation of 
5336bis-00 is that <ASCII@non-ASCII> cannot be submitted to a 
non-UTF8SMTPbis MTA, so it must make one of the three choices in section 
3.2. These choices do not include doing a conversion of the domain, 
unless it is an MSA.

In the interest of maximum interoperability, I think we should specify 
that a UTF8SMTPbis MTA should do a conversion in this case.

> Doing so is not only out-of-charter, it essentially

---

From: John C Klensin <[email-masked]>
Date: Tue, 13 Jul 2010 14:33:12 -0400

--On Tuesday, July 13, 2010 14:16 -0400 Ernie Dainow
<[email-masked]> wrote:

> Understood. The context of the above paragraph from 5336bis-00
> section 3.2, is an MTA that supports UTF8SMTPbis. If such a
> server is allowed to submit <ASCII@non-ASCII> to another MTA
> that does not support UTF8SMTPbis, then we cannot place any
> new rules on that server, and it is bound only by earlier SMTP
> standards. But my interpretation of 5336bis-00 is that
> <ASCII@non-ASCII> cannot be submitted to a non-UTF8SMTPbis
> MTA, so it must make one of the three choices in section 3.2.
> These choices do not include doing a conversion of the domain,
> unless it is an MSA.
> 
> In the interest of maximum interoperability, I think we should
> specify that a UTF8SMTPbis MTA should do a conversion in this
> case.

Speaking with my WG Co-Chair hat distinctly off, I agree.
Interoperability is good and, if we know how to get messages
through without problems, we should do so.   On the other hand,
if there is lots of UTF-8 stuff in the headers, that might be
harder than just converting domains to A-labels, so we need to
think through this carefully.

   john

---

From: "Jiankang YAO" <[email-masked]>
Date: Wed, 14 Jul 2010 09:55:48 +0800

----- Original Message ----- 
From: "Ernie Dainow" <[email-masked]>
To: "John C Klensin" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, July 14, 2010 2:16 AM
Subject: Re: [EAI] mail bounce versus punycode conversion

> 3.2. These choices do not include doing a conversion of the domain, 

section 3.2 in  5336bis-00 said "If a server advertises UTF8SMTP and the client does not recognize the
   extension, the client may send a regular message [RFC2821] and
   [RFC2822].  In this case, the client may continue to use the
   [RFC3490] or [RFC5890] ToAscii() to encode the domain portion of an
   address. "

so the smtp may send <ASCII@non-ASCII> if the client can do toascii operation to the domain portion of <ASCII@non-ASCII>.

in this case, the smtp will regard <ASCII@ACE domain> as the rfc2821 address.

---

From: Ernie Dainow <[email-masked]>
Date: Wed, 14 Jul 2010 08:28:57 -0400

On 7/13/2010 9:55 PM, Jiankang YAO wrote:
> From: "Ernie Dainow"<[email-masked]>
> To: "John C Klensin"<[email-masked]>
> Cc:<[email-masked]>
>>>> According to section 3.2 in  5336bis-00,<ASCII@non-ASCII>  is
>> submit<ASCII@non-ASCII>  to another MTA that does not support
>> 5336bis-00 is that<ASCII@non-ASCII>  cannot be submitted to a
> so the smtp may send<ASCII@non-ASCII>  if the client can do toascii operation to the domain portion of<ASCII@non-ASCII>.
>
> in this case, the smtp will regard<ASCII@ACE domain>  as the rfc2821 address.
>
>    
This is the case of  SMTP ---> UTF8SMTPbis.
The initial comment was the case of  UTF8SMTPbis ---> SMTP.

Perhaps we can make this section more clear and complete by enumerating 
all the possibilities.

Sender                           Receiver
==========               ==========
SMTP                 --->    SMTP
Legacy systems, not covered by these specifications.

SMTP                 --->    UTF8SMTPbis
Sender will send only ASCII@ASCII or ASCII@ACE. Both must be accepted by 
the receiver.

UTF8SMTPbis    --->    SMTP
Sender can only send ASCII@ASCII or ASCII@ACE.
Anything with non-ASCII in addresses is returned as a delivery failure 
(bounce), except for the case ASCII@non-ASCII which should be converted 
to ASCII@ACE so that it can be sent (if the rest of the message conforms 
to legacy email RFC 5321/5322).

UTF8SMTPbis    --->    UTF8SMTPbis
Sender can send any combination of ASCII, non-ASCII, ACE.

-Ernie