Subject: Re: [EAI] Random thought #1 - UTF8SMTP handshaking

From: "Jiankang YAO" <[email-masked]>
Date: Thu, 8 Jul 2010 11:57:14 +0800

“UTF8REPLY” is to inform the server that the client can support utf8. without “UTF8REPLY”, the server will issue the ascii message only.

on the other hand,

if the client can use mail from or rcpt to  with the utf8@utf8, it means that the client can support utf8.
The serve can know the client's ability through the message the server recieves from the client.

so your two way handsaking is theoretical possible, but not practicable or necessary.
  ----- Original Message ----- 
  From: Shawn Steele 
  To: [email-masked] 
  Sent: Thursday, July 08, 2010 3:27 AM
  Subject: [EAI] Random thought #1 - UTF8SMTP handshaking

  I don’t want to derail our time frame, but I was wondering if this was worth considering:

  Currently the UTF8SMTP(bis) extension is a one-way handshaking.  Eg: the client knows the server understands UTF8SMTP, however there’s no way for the server to generally know if the client can accept UTF-8.  That leads us to things like “UTF8REPLY”.  Additionally, it is not necessarily guaranteed that a MAIL or RCPT command is UTF-8 just because it’s non ASCII (a misbehaving client could be using their local 8-bit code page or something).  It’s also not guaranteed that DATA be UTF-8 (I think historically data has sometimes used local code pages, though now it’s probably more likely to be correctly tagged).

  To me, stamping “UTF8xxxx” on every command seems pretty awkward.  Should we consider a “UTF8” command that a client can use to confirm that it handles UTF-8?  Then the handshaking doesn’t have to happen with every command?  Eg:

  UTF8

  250 Ok

  MAIL FROM:<[email-masked]>

  250 Ok

  …

  DATA

  … utf8 data …

  .

  As mentioned, I don’t want to cause a lot of delay discussing this, but is it worth considering?  Would it be too much churn?

  -Shawn

   

  http://blogs.msdn.com/shawnste

------------------------------------------------------------------------------

  _______________________________________________
  IMA mailing list
  [email-masked]
  https://www.ietf.org/mailman/listinfo/ima

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 Jul 2010 07:28:20 +0000

> if the client can use mail from or rcpt to  with the utf8@utf8<mailto:utf8@utf8>,
> it means that the client can support utf8.

Clearly, but how do we know it's utf8@utf8<mailto:utf8@utf8>?  It "might" be iso8859-x@iso8859-x<mailto:iso8859-x@iso8859-x>.  (Yea, I get that it shouldn't be, but if we don't tell the server, how do we know for sure?  That client could be using the ISO8859SMTP extension)  UTF8SMTP says "the server know (& will probably obey) the EAI rules.  How do we know what the client's going to do?  Just because we see data with the high bit set doesn't mean its UTF-8 aware.

It's probably more interesting for DATA, where "simple" connections are more likely to accidentally (or intentionally) end up in a non-UTF-8 code page.

Eg: if I open a windows cmd window and telnet to a SMTP server and type non-ASCII data after "DATA", what happens?  I'll be sending whatever the machines current ACP is.  While that isn't strictly legal, it probably "works" in some environments.

-Shawn

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jul 2010 12:18:33 -0400

--On Thursday, July 08, 2010 07:28 +0000 Shawn Steele
<[email-masked]> wrote:

>> if the client can use mail from or rcpt to  with the
>> utf8@utf8<mailto:utf8@utf8>, it means that the client can
>> support utf8.
> 
> Clearly, but how do we know it's utf8@utf8<mailto:utf8@utf8>?
> It "might" be iso8859-x@iso8859-x<mailto:iso8859-x@iso8859-x>.
> (Yea, I get that it shouldn't be, but if we don't tell the
> server, how do we know for sure?  That client could be using
> the ISO8859SMTP extension)  UTF8SMTP says "the server know (&
> will probably obey) the EAI rules.  How do we know what the
> client's going to do?  Just because we see data with the high
> bit set doesn't mean its UTF-8 aware.

There is no ISO8859SMTP extension.  If someone wanted to make
one up, they would be obligated to resolve the problem you are
concerned about.  Without EAI, nothing prevents a client from
sending ISO 8859-x or, for that matter EBCDIC, to an
unsuspecting server.  The server will either notice that the
high bit is set and reject the message, or it will invoke its
"treat as nonsense" procedure, or it will get into a greater or
lesser amount of trouble depending on what is in the material
being sent.  With EAI, we get a statement from the server that,
if non-ASCII is being sent, UTF-8 is expected and, implicitly,
that anything else may be treated with hostility.  Whatever your
"announce that UTF-8 is being used" command might accomplish,
solving this problem isn't on the list because a client could
omit that command and send UTF-8 (or ISO-8859-x or EBCDIC)
anyway.  In each case, the server has to be careful enough to
avoid being damaged (however that is defined).  And, in each
case, it would be fairly stupid for the server to accept/
encourage anything but UTF-8.

I mention EBCDIC above partially to point out that the potential
problem has been with us since the dawn of the Internet, rather
than this being a new, i18n-specific, problem.  The conclusion
35-odd years ago was "one form on the wire and anyone who tries
anything else deserves whatever happens to them".  No change
with this: we specify UTF-8, we make no provisions for anything
other than UTF-8, and anyone who decides to send something else
is probably not going to interoperate.  We could, if you think
it useful, add text to "Framework" (4952bis) to explain this,
but, if you want that, say so soon and please send text.

In a way, this is a corollary to the message of
draft-iab-i18n-encoding: we are much better off with a single
on-the-wire encoding than with a multiple-choice situation.  If
someone wants to use ISO 8859-666 internally to an operating
system that is fine... and really no different from using UTF-16
internally.  A well-defined conversion model is needed to get
the internal form to and from Unicode in UTF-8.  But the Unicode
folks claim that is a solved problem for all known and
widely-used CCSs.  If they are wrong and it isn't, it is their
problem, not one that we should make worse by creating
alternatives.

> It's probably more interesting for DATA, where "simple"
> connections are more likely to accidentally (or intentionally)
> end up in a non-UTF-8 code page.
> 
> Eg: if I open a windows cmd window and telnet to a SMTP server
> and type non-ASCII data after "DATA", what happens?  I'll be
> sending whatever the machines current ACP is.  While that
> isn't strictly legal, it probably "works" in some environments.

It is not only "not strictly legal", it is a serious violation
of the spec to send anything other than ASCII text without MIME
structure and a Content-type that specifies the relevant
"charset".  Whether it will "work" or not depends on what the
server does to protect itself.  Note first that, if 8BITMIME is
not specified, the server may reasonably reject anything
(envelope, headers, message body) that contains octets with the
high bit on.  Even if it doesn't, it may reasonably reject
anything whose headers do not contain a Content-type header
field (and MIME-Version, etc.).

Will things "work" if both client and server are 8-bit-clean,
permissive, and use the same character coding internally (or
have and apply good heuristics)?  Yes.  But that is one of the
main things we've learned from many years of experience with
email (and possibly applications protocols in general):  if one
enables bad practices in the name of more interoperability, what
one mostly gets is more bad practices... and indignation by
those who are using them that their non-conforming systems don't
interoperate better with everything else.    As a handy example,
to this day, when we discover a case of MIME not working as
intended, one of the main reasons usually turns out to be a
long-ago decision by a particular vendor to favor the "suffix"
of a file name over the content-type in interpreting the content
of a message body.   I'll leave the name of that particular
vendor as an exercise.

     john

---

From: Shawn Steele <[email-masked]>
Date: Thu, 8 Jul 2010 16:38:20 +0000

> than this being a new, i18n-specific, problem. 

I think that's my point.  > 0x7f has worked in some environments in the past.  I'm not saying it was good or bad, or wise or stupid or whatever, but, in some cases, I believe that there have been systems that blindly exchange such data and rely on the fact that everyone in the org used the same ACP to make it "work".

I believe I first encountered this on a Unix box running sendmail in the early 90's.

I *think* that modern clients/servers are stricter, so that, unlike html, there is less of that strangeness going on now.

> concerned about.

Can you guarantee there's no private XISO8859SMTP extension?

> if one enables bad practices in the name of more interoperability, what
> one mostly gets is more bad practices...

I'm not suggesting this to "enable bad practices", I'm suggesting it to ensure good practices.  Right now the draft has half-way handshaking.  The server says "I'm EAI", and ASSUMES that the client may be EAI if it sends data with the high bit set.  Previously the server should've rejected that data, but now it'll start allowing it because it's making that assumption, not because it actually "knows" that the client supports EAI.

To be very clear about what you're alluding, I've been told that Exchange is very strict about what it does and does not allow, so apparently this isn't something Exchange needs for compatibility.  I've tested that myself with the FROM: and RCPT:, which fail.  I've also had requests from a Korean customer wanting to know how to get local code page email addresses to interoperate outside of their intranet.  Obviously they weren't using Exchange SMTP, since we don't support that, even on the intranet.  (I don't know what they were using).

In any event, I don't think it's interesting enough to make a huge deal out of, however it did seem, to me, that we could have a chance to confirm that the client was indeed intending EAI behavior, for all commands, without assumptions and without needing a bunch of UTF8REPLY tokens.  You've been around mail longer than me :), so if you think the assumption is good enough for interoperability with the huge variety of clients out there, then you're likely correct.

Why is it evil for the server to know what the client supports?

-Shawn