Subject: Re: Is it good to publish a new RFC on SMTP without internationalisation?

From: Dan Oscarsson <[email-masked]>
Date: Thu, 16 Mar 2000 12:30:18 +0100 (MET)

>Bottom line is that internationalization of email is going to
>require strategies and mechanisms that are compatible with slow
>and incremental rollout, rather than flag days or the assumption
>that newly non-compliant systems can just fall by the wayside.
>I think that means either use of ESMTP extension mechanisms to
>negotiate extended character set usage or insertion of a
>directory layer that would present internationalized materials
>to users but keep SMTP operating on ASCII-subset protocol
>elements.  Neither strategy should be incompatible with the
>draft 821bis document.

I think you, Matti Aarnio and Perry Metzger all think that I want to
start using UTF-8 without any compatibility to the current
ASCII only world. I do not!

The way I can see (and could define) is to use ESMTP with a new 
option UTF8, that identifies that the server can use UTF-8, just
like it now can do 8BITMIME.
While the current draft says that the 7-bit only mode can be switched
using 8BITMIME to 8-bit in body, I get the impression that it forbinds
an ESMTP mode to  allow 8-bits in MAIL FROM, RCPR TO and  headers.
Is this so?
If there is no problem that we define a UTF-8 option in ESMTP that
will allow 8-bit UTF-8 to be used everywhere, them I have nothing against
the draft.

Doing it this way with ESMTP there is no problem with interoperability.
If a MTA using UTF-8 talks to an old MTA not supporting the
UTF8 option of ESMTP, it has to encoded domain names, headers etc
using ASCII encodings. Just like is done today when a MTA with
a n 8 bit body connects to a MTA without 8BITMIME, it has to
convert into a 7-bit MIME body.

   Dan

---

From: Matti Aarnio <[email-masked]>
Date: Thu, 16 Mar 2000 13:59:50 +0200

> ASCII only world. I do not!

	You do.  You CAN'T introduce UTF-8 in ADDRESSES.
	You can (with similar means to 8BITMIME) do it in
	message BODY (and perhaps in headers).

	SMTP transport on  MAIL FROM / RCPT TO handles about
	ADDRESSES, those you can't modify to use UTF-8 encoded
	characters.

> convert into a 7-bit MIME body.

	Sure, but what is the point in having UTF-8 encoded DOMAINS in
	use at all, if you have to support both those, and compatibly
	encoded ones ?  They would mean DIFFERENT lookup keys on the DNS,
	after all..

	For local-parts we could allow this type of transformations
	to happen -- they are local matter, after all, but DNS...

	Say latin-1 domain:        åttabittar.se
	would encode in UTF-8 in:  XXttabittar.se
	and "safe-7" encoded as:   YYYYYttabittar.se

	There are already several propositions of encoding UNICODE
	chars in DNS, dig up RFC archive.  Including:
				   +U+88776+ttabittar.se
	Hmm.. No, I don't remember how it was exactly, but '+' isn't
	kosher char in domain name, as I seem to have coded in my MTA's
	RFC 821 scanner..

-- 
/Matti Aarnio	<[email-masked]>

---

From: Keith Moore <[email-masked]>
Date: Thu, 16 Mar 2000 10:07:28 -0500

> like it now can do 8BITMIME.

no, that's not even close to a solution.  first of all, SMTP is not
the problem - 8-bit SMTP has no problems whatsoever carrying UTF-8.  
second of all, we looked at going down a similar path (content-negotiation 
in SMTP) when the MIME and SMTPEXT work was done, and it was so dangeroulsy
complex and failure-prone that we eventually abandoned it. 

The problem is that putting UTF-8 in email addresses breaks essentially 
every single internet mail user agent in the world.  and this doesn't 
even begin to consider the interactions between mail programs and other 
programs.  

internationalization of applications will be extermely difficult,
and will require a great deal of careful analysis, design, and planning.
this is true for electronic mail even more than for other applications.
attempts at quick fixes are likely to be extremely disruptive.

also, drums should not be anticipating what the output of idn will look like.

Keith

---

From: Robert Elz <[email-masked]>
Date: Thu, 16 Mar 2000 22:59:10 +1100

Date:        Thu, 16 Mar 2000 12:30:18 +0100 (MET)
    From:        Dan Oscarsson <[email-masked]>
    Message-ID:  <[email-masked]>

  | The way I can see (and could define) is to use ESMTP with a new 
  | option UTF8, that identifies that the server can use UTF-8, just
  | like it now can do 8BITMIME.

In that case, it cannot possibly go in the drafts from this WG, as
the purpose here is simply to clean up the existing documents, and
make them match current practices (current desireable practices).

The inventing of anything new is explicitly excluded.

kre

---

From: Dave Crocker <[email-masked]>
Date: Thu, 16 Mar 2000 08:36:05 -0800

>ASCII only world. I do not!

Dan,

There are two lines of thinking in response to your note.  I fear you have 
noticed only one of them.

The one that has dominated this thread pertains to the details of an 
approach to adding internationalization.  The most significant point about 
the responses to your suggestion, in that regard, is that a number of 
experienced email protocol designers (and implementors) believe
your approach won't work.  That makes for a nice, healthy debate, out of 
which might come a solid proposal.  As noted, DRUMS is not the proper venue 
for that debate but, yes, it would be a good one to pursue, somewhere.

The point that has been missed was Klensin's main point, concerning 
process:  Whatever conventions are pursued for internationalization in 
email, they won't happen quickly.  John used "months" as an optimistic 
choice for developing the specification.  As he and the other respondents 
well know -- and as the content of the various respondents should make 
clear by demonstration -- MANY months is more likely.  And then there is 
deployment.

For global standardization, nothing is ever simple or quick.  Hence, it 
really is not feasible to "slip a change in" at a late stage like this.

d/

=-=-=-=-=
Dave Crocker  <[email-masked]>
Brandenburg Consulting  <www.brandenburg.com>
Tel: +1.408.246.8253,  Fax: +1.408.273.6464
675 Spruce Drive,  Sunnyvale, CA 94086 USA

---

From: Maynard Kang <[email-masked]>
Date: Fri, 17 Mar 2000 18:17:01 +0800

Hi,

> even begin to consider the interactions between mail programs and other

Agreed, Keith. The problem is not just with the transport layer.

However, I think Dan's suggestion of an ESMTP extension to indicate
8-bit envelope capability does have some merit. Perhaps it could be
taken further with extensions introduced at other stages of the Internet
mail system to preserve interoperability- maybe even a POP3 extension to
ensure that non-8bit capable MUAs don't break (Then again, not all MUAs
use POP, but let's just use this as an example).

> attempts at quick fixes are likely to be extremely disruptive.

Definitely. However, internationalization is still nonetheless important
and its impact on e-mail, being one of the fundamental applications used
on the Internet, should be an important priority and dealt with
appropriately.

I'm not advocating a quick fix, but hey, we should talk.

> also, drums should not be anticipating what the output of idn will look like.

Perhaps another forum to deal specifically with these problems should be
established.

Maynard

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 Mar 2000 09:20:00 -0500

> use POP, but let's just use this as an example).

that's exactly the point.  you cannot add internationalization to 
electronic mail by making a simple change to a single protocol.or
application.  you need to chnage MTAs, MUAs, message stores, gateways,
archivers, filters, firewalls, mailing list managers, etc.  and 
the result has to be incrementally deployable - meaning that each
component has to be able to change separately from the others without
having an adverse effect.  furthermore, this constraint has to be
satisfied even though for many of these components (that talk to each
other) there is no good way for them to be aware whether their peers 
support the internationalized extensions.  (SMTP is the rare exception
to this rule, but you can't solve the problem entirely with SMTP)

one inescapable conclusion of the incremental deployment constraint is 
that there absolutely must be a pure-ascii encoded form of i18n address 
which is as equally valid (as a sender or recipient address) as the 
utf-8 format.  

(and given that constraint, it's difficult to imagine why the recipient 
address should *ever* appear as utf-8 on the wire - trying to do things
this way drastically increases the complexity, and decreases the 
reliability, of the solution.)

> on the Internet, should be an important priority and dealt with

I completely agree that internationalization is very important.  But 
given that email is the most widely used, implemented, and deployed 
applications on the internet, and one or the more complex applications
on the Internet, and that it is extremely important to maintain its
reliability, I suspect it will be the last application to acheive
widespread address internationalization support - even if such work 
takes priority over other applications.

> Perhaps another forum to deal specifically with these problems should be

idn already exists, and that group clearly needs to understand the 
potential impact of its proposals on applications - trying to define
dns internationalization without understanding its effect on applications
would be a disaster.  that and there is a need for cross-application
compatibility rather than having piecemeal solutions for each application.
so for the time being, the discussions need to take place on idn.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 17 Mar 2000 15:37:46 -0000

> that there absolutely must be a pure-ascii encoded form of i18n address 

Have you learned nothing from your past mistakes, Keith?

---Dan

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 Mar 2000 12:54:04 -0500

> Have you learned nothing from your past mistakes, Keith?

I'm constantly learning from my mistakes - and yours.

Keith