Subject: Re: [EAI] draft review of: draft-ietf-eai-rfc5336bis-07.txt

From: Shawn Steele <[email-masked]>
Date: Wed, 22 Dec 2010 21:43:34 +0000

Some comments:

> Date: Wed, 22 Dec 2010 10:04:48 -0800
> From: Dave CROCKER <[email-masked]>

>     5.  The SMTP extension needs to remove all restrictions it imposes on MIME 
> content-type.  A major reason that MIME was successful was that it was 
> transparent to the transfer infrastructure.  The current SMTP extension 
> specification changes this model, which actually increases the barrier to 
> adoption of Unicode in email.  It needs to be easy for two MUAs that support 
> Unicode in the email header to exchange mail even when the infrastructure does 
> not support UTF-8.

I disagree:)  If a mail agent/client/whatever can handle UTF-8 in the header, then those should certainly be able to handle UTF-8 in the body of the mail.  Yes, MIME has worked because it tunnels other encodings through mechanisms that don't understand those encodings, however any EAI aware client/server/whatever would necessarily be able to understand UTF-8, reducing the need for other content.

In fact, MIME isn't entirely successful.  It works a large part of the time, but there are differences in encoding behaviors between vendors that create very troublesome edge cases and variations that can be difficult to work around.  I've often encountered customers with encoding related problems.  A strong preference for UTF-8 would resolve a large number of interoperability issues.

> However note that <encoded-word> and <A-label> 
> already accomplish this in a way that is transparent to the existing email 
> infrastructure; only the end-systems need to understand it.

Actually a-labels are causing serious pains for infrastructure and applications.  IDN aware applications must be aware of both forms of their name, and a-labels are "leaking" into all sorts of unexpected places.  Even at a simple form, merely encoding Unicode in an a-label to query a server is insufficient for basic web content.  In addition, the http request has to be encoded with a punycode form, and the server needs to be made aware of it's punycode name.  And then there's the mixing of IRIs and URIs, so applications in many layers need to recognize both the punycode and unicode forms.  This is amazingly painful across the entire ecosystem.  

> The second goal is to support Unicode in the more "native" form of UTF-8. 
> (The quotation marks are because UTF-8 is not native Unicode, either; it is
> a highly encoded form of Unicode...)

That is serious quibbling.  By that definition, there is no native form of Unicode.  The Unicode Standard clearly provides for UTF-8, UTF-16, and UTF-32 variations, and makes no preference for which representation is preferred.  UTF-8 is obviously far more compatible with SMTP than either UTF-16 or UTF-32.

> I strongly suggest 
> saying "Unicode" when intending to refer to the richer set of characters that 
> are the goal of this work, and "UTF-8" when referring to the particular binary 
> encoding of Unicode that is the focus of the SMTP extension work.

That makes sense.

> Non-UTF-8 Unicode support -- Can a message support Unicode without 
> UTF-8?  The existence of <encoded-word> and <A-label> constructs makes clear 
> that the answer is yes.  Hence, it should be possible to support Unicode 
> messages, without this SMTP extension.  

This was debated to great extent, primarily as a "fallback" option.  The WG overwhelmingly thinks that <a-labels> are a terrible way to work around these problems:
  * Encoded forms are inherintly non-user friendly
  * Encoded forms "leak".
  * It is impossible to create an EAI aware mail client or server without updating that client or server, even if that update were "merely" to handle the new use of these encodings.  Obviously if an update is required, then UTF-8 could be that update.
  * Implementing IDN has shown how very complicated and annoying leaking encoded forms can be.  The biggest problem is that the encoded form "leaks" into unicode-aware contexts, and suddenly software that wasn't broken by the new standard is broken by needing to decode these things.
  * Clearly the encoded-words were interesting when trying to shove stuff through 7 bit pipes and Unicode was in it's infancy, but that is no longer the case.  Most of the infrastructure supports 8 bit mail, so there is very little benefit to making the standard this complex.

> Instead I recommend the client to 
> signal explicitly when UTF8-encoded addresses are present, such as a 
> <Mail-parameters> option (<esmtp-param>) to the MAIL command.

In my opinion that is more complex.  Currently my client/server have to use an encoding, like ASCII (though some use other encodings).  The UTF8SMTP handshaking clearly indicates that we're talking in UTF-8.  No further handshaking is therefore required.

The only cases where the standard requires anything more than the initial handshaking is when that handshaking failed.

>>         Once isolated by this parsing process, the local part MUST be
>>    treated as opaque unless the SMTP server is the final delivery Mail
>>    Transfer Agent (MTA).

> { The statement about handling of <local-part> is redundant with the base RFC 
> 5321 specification.  The statement, here, should be that the handling of 
> <local-part> is unchanged from the base specification.  Again, it should not 
> repeat the normative language, unless it is changing it.  If it is changing it, 
> the change needs to be essential for support of internationalized addresses. }

The intent here was because some people wanted to explicitly forbid encoding or decoding (specifically punycoding) the local part.  I'm happy if all of the stuff because of that is omitted, defaulting as you say, to the base RFC 5321 specification.

(Full disclosure: I am one of those who wouldn't mind allowing punycode fallback, as a very worst case last resort.  IMO that should only be fore tunneling through non-EAI aware systems though and certainly not a first class way of "solving" EAI.).

>> It MAY send a UTF-8 header
>> [RFC5335bis] (which may also include mailbox names in UTF-8).  It MAY
>> transmit the domain parts of mailbox names within SMTP commands or the
>> message header as A-labels or U-labels

> { I believe that the use of "MAY" is not correct. This would mean that the 
> receiver needs a means of distinguishing whether the data are UTF-8 or not.}

I believe this is trying to say that an ASCII-only message would also work.  Since ASCII is a subset of UTF-8, the encoding doesn't really have to be detected.  In other words:  Just because we exchanged "UTF8SMTP" that doesn't mean that you are required to include characters >= U+0080.  IMO this could probably be change to be MUST?

The intent is very much not to allow switching to non-UTF-8 modes, the handshaking already said that this session is in UTF-8.  (Note that in practice a non-UTF8SMTP session isn't necessarily ASCII.  Existing implementations often use whatever the system code page is.  UTF8SMTP allows a session to actually declare a specific UTF-8 code page, whereas existing implementations just have to get lucky, unless they use MIME).

>> All labels in domain parts of mailbox names which are IDN
>>    forms of A-labels or U-labels MUST be valid.

> { This is strange.  Either it is repeating a normative requirement from SMTP or 
> it is expanding SMTP to require special validation for IDN forms of domain names 
> that is not present for ASCII forms.  Neither interpretation seems like the 
> right thing to be doing here. Such a modification to SMTP seems out of scope.

Seems redundant to me too.  I think this probably comes from the problem that a-labels and u-labels in IDN "leak".  So we can't be sure which one they are.  This clarifies that EAI has to be aware of IDN and make sure it does the right thing?

>>The presence of the UTF8SMTPbis
>>    extension does not change the requirement of RFC 5321 that servers
>>    relaying mail MUST NOT attempt to parse, evaluate, or transform the
>>    local part in any way.

> { This sentence replicates normative language from a different specification. 
> This is a very bad thing to do, in case the original specification changes its 
> language.  I suggest: }

Again, this is a co-chair not wanting me to encode local parts as a-labels to tunnel through non-UTF8SMTP connections, even in a very-worst-case-last-resort sort of way :)

>> If the UTF8SMTPbis SMTP extension is not offered by the server, the SMTP
>> client MUST NOT transmit an internationalized address and MUST NOT transmit

>{ This appears to prohibit the sending of internationalized addresses that are 
>encoded in ASCII, rather than in UTF-8. 

There are no "internationalized addresses that are in ASCII".  There "are" some that use ACE encoding, (but I wouldn't call that strictly ASCII).  Also current practice can send addresses in nearly any code pages, which I've seen in CJK locales. This would require those servers conform to UTF-8, since the current practice just causes a mess when it escapes the local system.

>  If the concern is sending UTF-8, then that's the language that needs to be 
> used.

I believe the main thing is that if I try to connect, and the server doesn't offer UTF8SMTP, then I shouldn't blindly go ahead and send my UTF-8 header anyway.

>  Messages with ASCII-encoding of internationalized addresses need to be 
> permitted to be sent, without first requiring infrastructure support. }

There is no standard for ascii-encoded internationalized addresses.  (Other than domains).

>> There is no ESMTP parameter to assert that a message is an internationalized
>> message.  An SMTP server that requires accurate knowledge of whether a
>> message is internationalized is required to parse all message header fields
>> and MIME header fields [RFC2045] and [RFC2047] in the message body.

> { This is confusing.  There is a rule against sending an internationalized 
> message, unless UTF8SMTPbis is in force, but there is no requirement that the 
> server be told explicitly when a message is internationalized???

It's unnecessary, since the other option would be ASCII, which is, in effect, UTF-8, so the additional handshaking isn't required.  I'm confused what this would gain us.  A "normal" message is, effectively, intenrationalized, because it would still work with (or without) UTF8SMTP.

>> If an SMTP client follows this specification and sends any RCPT commands
>> containing non-ASCII addresses, the SMTP server is permitted to use UTF-8
>> characters in the email address associated with 251 and 551 response codes,
>> and the client MUST be able to accept and process them.

> { I assume that "follows this specification" means that the UTF8SMTPbis option 
> is in force.  If so, then say that, because it is simpler and more precise. But, 
> then, it does require declaring /use/ of the option...

There are two cases:  Either it's UTF8SMTP from the handshaking, so then its this specification, or there was no UTF8SMTP handshaking, so then this RFC doesn't apply.  So if UTF8SMTP was offered, then it is being used.  

Since UTF8SMTP is offered by the server, you can tell if the client is using it because it sent characters > U+007F.  Since the client might not recognize UTF8SMTP, then whether the client supports UTF8 might not be clear, however this section says that if the client sent UTF-8, then the server may respond similarly.

>> 3.6.4.2.  VRFY and EXPN Commands and the UTF-8REPLY Parameter
>>
>> If the VRFY and EXPN commands are transmitted with the optional parameter
>> "UTF-8REPLY", it indicates the client can accept UTF-8 strings in replies to
>> those commands.  This allows the server to use UTF-8 strings in mailbox

> { Is this extension trying to to say that a client might support UTF8SMTPbis but 
> not be able to access UTF-8 replies???  Under what circumstance is this 
> reasonable? }

The server normally has no clue if the client supports UTF-8 until it gets a UTF-8 address from the client.  However these commands could cause UTF-8 replies to an ASCII query.  The option explicitly states that the client is prepared to receive that UTF-8 reply.

- Shawn

Senior Software Design Engineer
Windows Globalization
Microsoft

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 22 Dec 2010 16:42:04 -0800

On 12/22/2010 1:43 PM, Shawn Steele wrote:
> The UTF8SMTP handshaking clearly indicates that we're talking in UTF-8.  No
> further handshaking is therefore required.

Please provide some clarification:

      Where does the text define an explicit protocol handshake, so that both 
sides know that the session is now using UTF-8?

I could not find it (except for VRFY and EXPN...)

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net