Subject: Re: [EAI] Random thought #1 - UTF8SMTP handshaking

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jul 2010 13:55:15 -0400

--On Thursday, July 08, 2010 16:38 +0000 Shawn Steele
<[email-masked]> wrote:

>...
>> There is no ISO8859SMTP extension.  If someone wanted to make
>> one up, they would be obligated to resolve the problem you are
>> concerned about.
> 
> Can you guarantee there's no private XISO8859SMTP extension?

Of course not.  I can't even guarantee that there is no private
BAUDOT or packed ASCII extension, nor an extension that
specifies that Content-types are to be interpreted according to
rules different from those in the IANA registry, nor an
extension that specifies that all DNS name components of email
addresses are to be interpreted with the root to the left.  

The right model for dealing with _any_ private extension is that
it is up to those who are using it to sort out interoperability
with whatever is standardized.  One might reasonably discuss an
exception if an extension were demonstrated to be widely
deployed and important, but I haven't seen anyone come forward
and say "we have an ISO8859SMTP extension, it has been shipping
in products for a decade, and nine million people are depending
on it", only your wild speculations about some extension that
might exist out there somewhere.  Even if such an extension did
exist, the "one encoding on the wire" argument would constitute
a significant tradeoff factor against dealing with it, but we'd
at least have a basis for a meaningful discussion.

What I can guarantee is that trying to design an extension to be
bulletproof against any bad idea that someone might have devised
privately would lead to madness as well as to huge delays as we
try to sort through all of the alternatives, attacks, and real
or imaginary edge cases.   And I can suggest that adopting the
"someone might have done this, therefore we have to support it
forever" mentality is a large part of what got HTML --and the
URI and IRI specs-- into trouble.

>> if one enables bad practices in the name of more
>> interoperability, what one mostly gets is more bad
>> practices...
> 
> I'm not suggesting this to "enable bad practices", I'm
> suggesting it to ensure good practices.  Right now the draft
> has half-way handshaking.  The server says "I'm EAI", and
> ASSUMES that the client may be EAI if it sends data with the
> high bit set.  Previously the server should've rejected that
> data, but now it'll start allowing it because it's making that
> assumption, not because it actually "knows" that the client
> supports EAI.

See procedural note at end.

> To be very clear about what you're alluding, I've been told
> that Exchange is very strict about what it does and does not
> allow, so apparently this isn't something Exchange needs for
> compatibility.  I've tested that myself with the FROM: and
> RCPT:, which fail.  I've also had requests from a Korean
> customer wanting to know how to get local code page email
> addresses to interoperate outside of their intranet.

I'd suggest that the correct answer is "things leak, you really
want to convert to and be using UTF-8 addresses even in the
intranet, and you should not be trying this".  Think about the
design of an MUA that picks up an address from where it is
embedded in a text file.  If the standard is "UTF-8 and UTF-8
only on the Internet" then the conversion is clear.  If it is
"well, you can use UTF-8, or your local code page, or maybe
2022=based table switching", then you have gotten into the worst
parts of the "alternate address" theme because you don't know
which one to try, which order to try them in, etc.  And, worse,
with multiple alternatives, the odds of getting things
configured correctly so that a submission server could assume
that all relays in the chain would support the same options
would go down considerably.

> Obviously they weren't using Exchange SMTP, since we don't
> support that, even on the intranet.  (I don't know what they
> were using).
 
> In any event, I don't think it's interesting enough to make a
> huge deal out of, however it did seem, to me, that we could
> have a chance to confirm that the client was indeed intending
> EAI behavior, for all commands, without assumptions and
> without needing a bunch of UTF8REPLY tokens.  You've been
> around mail longer than me :), so if you think the assumption
> is good enough for interoperability with the huge variety of
> clients out there, then you're likely correct.
> 
> Why is it evil for the server to know what the client supports?

It isn't.  And, if the above is your main point, then let's not
waste time and get distracted by arguments about, e.g., private
ISO8859SMTP extensions or non-conforming DATA content.

I've got two sets of reasons to be hesitant about your proposed
command, one about protocol design (and, if you will, religion)
and the other about procedures.

_Religion_:  I'm extremely reluctant to see another state
introduced into SMTP.  It isn't that we couldn't, but it would
be hard to get right, it would turn developing the text for
5336bis into a task that would require deep understanding of
5321 and the design decisions that have accumulated since 821,
it would require that we sort out the order in which commands
could occur, and it would require that we spend a lot of time
ensuring that such a new state didn't have bad interactions with
extensions already developed and on the standards track.   We'd
have to analyze the implications of additional turnarounds,
specify whether RSET cleared the "UTF8" state, and review
pipelining and figure out whether we needed to update it.   None
of that is insurmountable, but it wouldn't be quick and I don't
think the case has been made yet that it would be worth the
trouble.

_Procedures_:   If we are going to introduce this sort of
facility and remain consistent with the WG's roots, we should
specify this facility, publish an Experimental spec, develop
implementations, and test them against each other and the
installed base to be sure that there are no ill effects.
Perhaps we could bypass those steps, but it would be risky.  My
experience with the IETF suggests that even doing the level of
examination needed to ensure consistency with 5321 and the
Pipelining extension if a new state were added would stretch
things out by at least six months and that going through the
Experimental spec and testing arrangements would stretch that
out to 12 - 18 months.  If I adjust for the level of activity in
this WG (count the number of comments about anything but
editorial issues in 4952bis since it was posted), you can
probably double those numbers. 

There is also the small issue of deployment speed: as with
8BITMIME, the current model requires that a server advertise the
extension (a few lines of code) and disable some tests.
Implementing a new command, advertising its availability, and
maintaining extra state information turns implementation of
email i18n back into a big deal.  To the extent to which we are
depending on rapid deployment (at least within language
communities) as part of our defense against needing much more
complex transition models, making implementation unnecessarily
harder is not in our interest.

If there were compelling need here, I'd think that was
worthwhile.  But, so far, all we've heard about are theoretical
cases what would constitute encouraging bad ideas (e.g.,
anything on the wire other than UTF-8) if were real and we
recognized them.

I note, although you find it distasteful (and, actually, I do
too), that requiring "UTF8 stuff here" parameters on the MAIL,
RCPT, and DATA commands is much less problematic than
introducing a "UTF-8" command, precisely because it doesn't
introduce new state.  Of course, it means we would have to look
at extensions such as BDAT too, but...

     john

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 Jul 2010 18:00:53 +0000

For the record, I can live with the draft with the way it is, it sounds like too much work to do anything else.  I'm still concerned that we effectively have partial handshaking, but I don't see any reason why it can't work as drafted.

-Shawn

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Thursday, July 08, 2010 10:55 AM
To: Shawn Steele; Jiankang YAO; [email-masked]
Subject: RE: [EAI] Random thought #1 - UTF8SMTP handshaking

--On Thursday, July 08, 2010 16:38 +0000 Shawn Steele <[email-masked]> wrote:

>...
>> There is no ISO8859SMTP extension.  If someone wanted to make one up, 
>> they would be obligated to resolve the problem you are concerned 
>> about.
> 

Of course not.  I can't even guarantee that there is no private BAUDOT or packed ASCII extension, nor an extension that specifies that Content-types are to be interpreted according to rules different from those in the IANA registry, nor an extension that specifies that all DNS name components of email addresses are to be interpreted with the root to the left.  

The right model for dealing with _any_ private extension is that it is up to those who are using it to sort out interoperability with whatever is standardized.  One might reasonably discuss an exception if an extension were demonstrated to be widely deployed and important, but I haven't seen anyone come forward and say "we have an ISO8859SMTP extension, it has been shipping in products for a decade, and nine million people are depending on it", only your wild speculations about some extension that might exist out there somewhere.  Even if such an extension did exist, the "one encoding on the wire" argument would constitute a significant tradeoff factor against dealing with it, but we'd at least have a basis for a meaningful discussion.

What I can guarantee is that trying to design an extension to be bulletproof against any bad idea that someone might have devised privately would lead to madness as well as to huge delays as we try to sort through all of the alternatives, attacks, and real
or imaginary edge cases.   And I can suggest that adopting the
"someone might have done this, therefore we have to support it forever" mentality is a large part of what got HTML --and the URI and IRI specs-- into trouble.

>> if one enables bad practices in the name of more interoperability, 
>> what one mostly gets is more bad practices...
> 
> I'm not suggesting this to "enable bad practices", I'm suggesting it 
> to ensure good practices.  Right now the draft has half-way 
> handshaking.  The server says "I'm EAI", and ASSUMES that the client 
> may be EAI if it sends data with the high bit set.  Previously the 
> server should've rejected that data, but now it'll start allowing it 
> because it's making that assumption, not because it actually "knows" 
> that the client supports EAI.

> To be very clear about what you're alluding, I've been told that 
> Exchange is very strict about what it does and does not allow, so 
> apparently this isn't something Exchange needs for compatibility.  
> I've tested that myself with the FROM: and RCPT:, which fail.  I've 
> also had requests from a Korean customer wanting to know how to get 
> local code page email addresses to interoperate outside of their 
> intranet.

I'd suggest that the correct answer is "things leak, you really want to convert to and be using UTF-8 addresses even in the intranet, and you should not be trying this".  Think about the design of an MUA that picks up an address from where it is embedded in a text file.  If the standard is "UTF-8 and UTF-8 only on the Internet" then the conversion is clear.  If it is "well, you can use UTF-8, or your local code page, or maybe 2022=based table switching", then you have gotten into the worst parts of the "alternate address" theme because you don't know which one to try, which order to try them in, etc.  And, worse, with multiple alternatives, the odds of getting things configured correctly so that a submission server could assume that all relays in the chain would support the same options would go down considerably.

> Obviously they weren't using Exchange SMTP, since we don't support 
> that, even on the intranet.  (I don't know what they were using).
 
> In any event, I don't think it's interesting enough to make a huge 
> deal out of, however it did seem, to me, that we could have a chance 
> to confirm that the client was indeed intending EAI behavior, for all 
> commands, without assumptions and without needing a bunch of UTF8REPLY 
> tokens.  You've been around mail longer than me :), so if you think 
> the assumption is good enough for interoperability with the huge 
> variety of clients out there, then you're likely correct.
> 

It isn't.  And, if the above is your main point, then let's not waste time and get distracted by arguments about, e.g., private ISO8859SMTP extensions or non-conforming DATA content.

I've got two sets of reasons to be hesitant about your proposed command, one about protocol design (and, if you will, religion) and the other about procedures.

_Religion_:  I'm extremely reluctant to see another state introduced into SMTP.  It isn't that we couldn't, but it would be hard to get right, it would turn developing the text for 5336bis into a task that would require deep understanding of
5321 and the design decisions that have accumulated since 821, it would require that we sort out the order in which commands could occur, and it would require that we spend a lot of time ensuring that such a new state didn't have bad interactions with
extensions already developed and on the standards track.   We'd
have to analyze the implications of additional turnarounds, specify whether RSET cleared the "UTF8" state, and review
pipelining and figure out whether we needed to update it.   None
of that is insurmountable, but it wouldn't be quick and I don't think the case has been made yet that it would be worth the trouble.

_Procedures_:   If we are going to introduce this sort of
facility and remain consistent with the WG's roots, we should specify this facility, publish an Experimental spec, develop implementations, and test them against each other and the installed base to be sure that there are no ill effects.
Perhaps we could bypass those steps, but it would be risky.  My experience with the IETF suggests that even doing the level of examination needed to ensure consistency with 5321 and the Pipelining extension if a new state were added would stretch things out by at least six months and that going through the Experimental spec and testing arrangements would stretch that out to 12 - 18 months.  If I adjust for the level of activity in this WG (count the number of comments about anything but editorial issues in 4952bis since it was posted), you can probably double those numbers. 

There is also the small issue of deployment speed: as with 8BITMIME, the current model requires that a server advertise the extension (a few lines of code) and disable some tests.
Implementing a new command, advertising its availability, and maintaining extra state information turns implementation of email i18n back into a big deal.  To the extent to which we are depending on rapid deployment (at least within language
communities) as part of our defense against needing much more complex transition models, making implementation unnecessarily harder is not in our interest.

If there were compelling need here, I'd think that was worthwhile.  But, so far, all we've heard about are theoretical cases what would constitute encouraging bad ideas (e.g., anything on the wire other than UTF-8) if were real and we recognized them.

I note, although you find it distasteful (and, actually, I do too), that requiring "UTF8 stuff here" parameters on the MAIL, RCPT, and DATA commands is much less problematic than introducing a "UTF-8" command, precisely because it doesn't introduce new state.  Of course, it means we would have to look at extensions such as BDAT too, but...

     john

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jul 2010 14:08:44 -0400

--On Thursday, July 08, 2010 18:00 +0000 Shawn Steele
<[email-masked]> wrote:

> For the record, I can live with the draft with the way it is,
> it sounds like too much work to do anything else.  I'm still
> concerned that we effectively have partial handshaking, but I
> don't see any reason why it can't work as drafted.

Ack.  

For the record, we did do the analysis the first time around and
concluded that client assertion about what it was doing wouldn't
add enough to be worth the trouble.  I'd be much more nervous if
we hadn't.  And my personal instinct would be to repeat the
analysis and make sure we had covered all of the cases... but I
don't see how to do that unless there is consensus that speed
less important than making sure we get everything as bulletproof
as possible.

     john

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 Jul 2010 18:10:56 +0000

Sorry I missed this on the first go-around then :)  (BTW: Are there "complete" archives?  Not that I can find anything anyway, but it looks like they're available for a certain window?)

-Shawn

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Thursday, July 08, 2010 11:09 AM
To: Shawn Steele; Jiankang YAO; [email-masked]
Subject: RE: [EAI] Random thought #1 - UTF8SMTP handshaking

--On Thursday, July 08, 2010 18:00 +0000 Shawn Steele <[email-masked]> wrote:

> For the record, I can live with the draft with the way it is, it 
> sounds like too much work to do anything else.  I'm still concerned 
> that we effectively have partial handshaking, but I don't see any 
> reason why it can't work as drafted.

Ack.  

For the record, we did do the analysis the first time around and concluded that client assertion about what it was doing wouldn't add enough to be worth the trouble.  I'd be much more nervous if we hadn't.  And my personal instinct would be to repeat the analysis and make sure we had covered all of the cases... but I don't see how to do that unless there is consensus that speed less important than making sure we get everything as bulletproof as possible.

     john

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jul 2010 15:36:52 -0400

--On Thursday, July 08, 2010 18:10 +0000 Shawn Steele
<[email-masked]> wrote:

> Sorry I missed this on the first go-around then :)  (BTW: Are
> there "complete" archives?  Not that I can find anything
> anyway, but it looks like they're available for a certain
> window?)

There should be.  Harald would know. 
But I can't remember whether that analysis was done during the
WG period or in the pre-WG JET work.

    john

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 11 Jul 2010 21:40:58 +0100

Catching up with email.

<hat area-director="off">:

><[email-masked]> wrote:
 [...]

>think the case has been made yet that it would be worth the
I agree that adding a new command and new state is risky and not going 
to work very well for SMTP.

>Perhaps we could bypass those steps, but it would be risky.
>
I think Experimental stage can be bypassed, but yes, it would be risky.

>My experience with the IETF suggests that even doing the level of
Right.

>introduce new state.
Agreed about disadvantages of the new state, but I think a new 
flag/parameter to the MAIL FROM command achieves the same purpose - the 
whole transaction becomes UTF-8-address-aware, much like 8BITMIME.

>Of course, it means we would have to look
Maybe not, if it just a flag/parameter to MAIL FROM.

Having said that, I am on the fence on this. While I prefer explicit 
negotiation as suggested by Shawn, I am also agreeing with your argument 
about UTF-8 emails leaking to non-extended SMTP environments.

---

From: John C Klensin <[email-masked]>
Date: Sun, 11 Jul 2010 18:28:04 -0400

--On Sunday, July 11, 2010 21:40 +0100 Alexey Melnikov
<[email-masked]> wrote:

>> worthwhile.  But, so far, all we've heard about are
>> theoretical cases what would constitute encouraging bad ideas
>> (e.g., anything on the wire other than UTF-8) if were real
>> and we recognized them.
>> 
> flag/parameter to the MAIL FROM command achieves the same
> purpose - the whole transaction becomes UTF-8-address-aware,
> much like 8BITMIME.
> 

> Having said that, I am on the fence on this. While I prefer
> explicit negotiation as suggested by Shawn, I am also agreeing
> with your argument about UTF-8 emails leaking to non-extended
> SMTP environments.

<all hats off ... other than whatever comes with an certain
amount of experience with email protocols, implementations, and
gateways>

If we could do this with a parameter on MAIL and no other
clutter, I'd probably drop my opposition -- having the client
just tell the server that it is about to send UTF-8 traffic is
pretty harmless at worst.

But part of Shawn's argument for a new verb is that we've
already got issues with VRFY and EXPN because the replies from
those commands are not "errors" even though they use the
response code mechanism.  Another issue is that signaling the
server that the client is UTF8-capable opens the door to a whole
series of issues that we decided to not touch for
interoperability reasons, include non-ASCII response messages
(with or without non-ASCII digits in the codes), language
negotiation for response codes, etc.  All of that is very
tempting, but would vastly complicate SMTP compared to "take the
codes and translate in the MUA that handles the response",
especially in relay environments in which the language(s) of the
relay operator may have no relationship to those of either the
sender or the receiver.

best,
    john