Subject: Re: Is it good to publish a new RFC on SMTP without internationalisation?

From: Charles Lindsey <[email-masked]>
Date: Fri, 17 Mar 2000 10:31:05 +0000 (GMT)

On Thu, 16 Mar 2000 14:51:38 -0500
	Keith Moore <[email-masked]> said...

> no it wouldn't.  because it implies having two forms of each email 
> address (one ascii and one utf-8) and some sort of conversion at 
> the boundaries, with the usual bitrot due to implementation errors.
> and when addresses leak  across the boundaries (which they will)
> chances are they won't be usable from the other side. 

Eh? Why would that be. Ascii is a subset of UTF-8. If you look closely,
you will see that all the headers of this message are already written in
UTF-8.

So if people want to move to domain names or local-parts using exotic
characters, then UTF-8 will, I am sure, be the way we eventually go.
But if someone wanted such things to pass through present transports,
then said transports might try to squeeze them into RFC 2047 format,
which might or might not work, but would sure lead to an unholy mess.
So I hope no-one will try to do such things until transports are 8-bit
clean as a matter of course. That sure is not going to happen overnight,
but it seems we ought to be preparing the ground by starting to move in
that direction, even though it might be 10 years before we can reap the
benefit.

FWIW, Netnews is already planning to go that way.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 Mar 2000 09:49:37 -0500

> you will see that all the headers of this message are already written in

yes, but if the headers of your message had used characters with
values > 127, my mail user agent would have (correctly) refused 
to generate a reply to your message.

> characters, then UTF-8 will, I am sure, be the way we eventually go.

you might be sure, but to me it's clear that using utf-8 on the wire 
for email (at least, any time in the next ~20 years) would be an 
incredibly stupid and disruptive thing to do.  so I hope you're wrong.

> which might or might not work, but would sure lead to an unholy mess.

yes.  2047 is another naive approach that is often suggested.
it would indeed lead to an unholy mess.  

given the transition issues, some sort of encoding scheme is a given.
but for a variety of reasons it cannot be 2047.

> that direction, even though it might be 10 years before we can reap the

let me get this straight: we don't know where we're going, but we 
ought to start moving in that direction anyway?

> FWIW, Netnews is already planning to go that way.

Netnews != email, and despite superficial similarities, what makes sense
for netnews doesn't necessarily make sense for email.

Keith

---

From: Philip Hazel <[email-masked]>
Date: Fri, 17 Mar 2000 15:22:42 +0000 (GMT)

I was going to keep my head down and my mouth shut, but my resolution 
has failed.

> to generate a reply to your message.

<dons flame-proof clothing>

Plenty of MUAs would have dealt with it quite happily (at least if the 
characters were, e.g. in the "phrase" part of a header line or in the 
Subject: line). Which approach provides the better service to the end
users?

Since I'm here, I might as well add a general view:

People are going to do this stuff anyway, whatever anybody says. If 
nobody says anything, various different ad hoceries will arise. It might 
be better if one approach were sanctioned, but doing nothing is not 
going to persuade people to stick to 7-bit mail. They are already past 
that. Use of utf-8 will grow willy-nilly and spread slowly. When there
are problems, they will get fixed. There is no way that any kind of
Master Plan can ever be feasible, not least in that it can't be made
quickly enough. All IMHO, of course.

A little story:

Back in 1990 or thereabouts, UK academia was still using its old, 
X.25-based network, when some radicals suggested we join the Internet. 
This was eventually agreed, but there was a large amount of concern 
about preserving interoperability for email, and worries that those 
who connected to the Internet early would cut themselves off from those
who converted last. We were forbidden to use SMTP - I'm not making this 
up - and all mail was supposed to go via the old network, including mail
to the rest of the Internet via a heavily congested national gateway 
that suffered from serious delays. Luckily, "experiments" were
sanctioned, and under this umbrella the rules were quietly ignored.
Nobody got isolated. People figured out how to send the right mail to
the right places over the right network. It was in nobody's interests to 
move to a situation where interoperability got broken. Of course there 
were problems from time to time, but they got fixed. I believe a lot of 
the original paranoia was needless. I suggest there is a parallel with 
the topic of this thread.

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 Mar 2000 10:51:40 -0500

> Subject: line). Which approach provides the better service to the end

it's not clear.  I have no problem with MUAs that attempt to display
such things, even realizing that any attempt to display untagged
characters in headers amounts to guesswork.  but there are a number 
of MUAs that propagate such errors in replies or even make them worse.
I do have a problem with those.

also, assuming that we will eventually standardize on utf-8 
as the representation of human-readable text in message headers,
developers of MUAs that are assuming other things may find themselves
in a bind.  fortunately, you can usually distinguish utf-8 from
other charsets by using heuristics.

> going to persuade people to stick to 7-bit mail. 

I agree with this.

> They are already past 

I don't agree with this.  Absent guidance, people will do ad hoc extensions
willy nilly.  The problem is that such extensions, once they are deployed,
may not easily be fixed.  I also believe that it is possible to develop
a reasonable plan for transition to i18n in a sane fashion, and to do this
within a relatively short timeframe (say one year).  

What I don't know is whether enough people will be willing to listen to 
such a plan - there are a lot of people trying to promote naive approaches 
(and some who think they can make a lot of money by doing so).

> A little story:

yes, I remember janet.  I remember that big-endian addresses leaked
over to the little-endian side and vice versa.  I remember that
its host addresses could be abbreviated on a per-component basis.
I remember that whenver I sent a message to a .uk address (and say, 
needing to give that person someone else's email address in the message
body) I would try to guess whether that person's mailer understood 
big-endian and little-endian - and based on that guess I would either 
reverse the domain or not.  I remember how heuristics to overcome 
address leakage from within janet would cause mail for [email-masked] 
to go to [email-masked], which became a problem when .cs became a valid 
TLD.  I remember writing code for my mailing list manager to check for 
addresses of the form user@uk.* in subscription requests and if the domain
of the requested address did not appear valid, to reverse it.  
(I think that code is still in there...it was still needed only a few 
years ago.)

and all of this experience is from someone on the *outside*.

yes, clueful users did know how to work around these problems, but
not with 3 nines of reliability.  it involved some guesswork and
sometimes the guesses were wrong.  on the other hand, users with
less insight into how the mail system worked generally suffered.  

the problem got "fixed" only because folks eventually got native
IP access and could completely bypass all of the Janet stuff.

but I do think that the internet/janet experience is an instructive
example of the kinds of things we can expect when trying to 
add i18n support to email - so for example we will have two
different forms of an address with leakage across enclave boundaries
and folks on one side of the boundary having to guess what kinds
of addresses are used by the other side, and do conversions between
the two on an ad hoc basis.  and so on.

but instead of one enclave where things are different, that is
distinguished by a domain suffix, we will have an arbitrary 
number of them, and they may not be distinguished
by any kind of naming convention. 

and whereas in the case of Janet there was a clear migration path
to native SMTP/TCP/IP (for which the necessary tools were already 
well proven and widely available and a large user community 
for those tools - speaking the same language (more or less) as
the Janet users - was already in place), this will not be the case 
when migrating to i18N email.

Keith

---

From: Philip Hazel <[email-masked]>
Date: Fri, 17 Mar 2000 16:18:28 +0000 (GMT)

On Fri, 17 Mar 2000, Keith Moore wrote:

> Absent guidance, people will do ad hoc extensions

I agree with the first two sentences, but I'm skeptical about the last 
one. It may be possible, but I'm too cynical to believe it will happen.
At least, not by, say June 2001. (And we actually needed it several
years ago.)

> over to the little-endian side and vice versa. 

Fun, wasn't it? One fun one was uk.tele.nokia.fi. There was a conspiracy
to make one that was meaningful both ways round, but I don't think
anybody actually did that.

> less insight into how the mail system worked generally suffered.  

I don't disagree, and I think the same story will happen again with i8n.
My point is that most of the time email flowed pretty well during the 
X.25-Janet/Internet transition. [Just for the record, the UK academic 
network is still called JANET - only now it's an IP network.]

> IP access and could completely bypass all of the Janet stuff.

Took 5 years, more or less. Extrapolate for i18N, anybody????