Subject: RE: [idn] Re: 7 bits forever!

From: "Dan Kohn" <[email-masked]>
Date: Mon, 01 Apr 2002 18:27:48 -0800

I believe that Einar could be most easily satisfied with something along
the lines of a UTF8HEADERS ESMTP extension, which would specify both
that 8 bit character are permitted in the header and that those
characters MUST be interpreted as UTF-8.  It would be expected that this
extension would normally be used in conjunction with 8BITMIME [1] or
BINARYMIME [2].  The challenge is that any mail relay implementing this
extension MUST be able to downconvert headers to 7-bit RFC 2047 [3]
compliant encoded words if the next hop does not support UTF8HEADERS.

This would seem to be a middle ground between Dan's desire for
just-send-8 and other's demand that existing RFC 2821 [4] compliant
relays not be broken by adding support for UTF-8 headers.

This should be an easy RFC to write up.  If there is interest from
implementers, I would be happy to do so.  In any event, I would suggest
taking follow-up to ietf-smtp [5].

[1] http://www.ietf.org/rfc/rfc1652.txt
[2] http://www.ietf.org/rfc/rfc3030.txt
[3] http://www.ietf.org/rfc/rfc2047.txt
[4] http://www.ietf.org/rfc/rfc2821.txt
[5] http://www.imc.org/ietf-smtp/

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>
Essays announced on <mailto:[email-masked]>=20
=20

-----Original Message-----
From: Einar Stefferud [mailto:[email-masked]]=20
Sent: Monday, April 01, 2002 11:47
To: [email-masked]; [email-masked]
Subject: Re: [idn] Re: 7 bits forever!

Hi John -- My thesis is that the problem lies now in the SMTP
standards, where the current "Send 8-bits" problem has always
resided.  The MIME working group could not solve that problem, and so
did not, but it made mail work since its deployment, which was the
objective in the MIME WG.  inside the RFC822 Body there is only one
way to handle 8-bit data, and that is to tag it, and bag it.  Even if
you can transmit 8-bit data without encoding it in the body of an
SMTP message, it needs to be tagged and delimited in the body.

MIME Quoted-Printable was designed to be a work-around-damage
solution at the time, and was not design to remove the need for
handling 8-bit data in SMTP.

So, my only point is that it appears that some work needs to be done
in the SMTP standards, and beating up on the MIME WG does not compute
given this observation.

That is all...  Just trying to shift the focus of this thread to
where someone might think of a way to solve the current problem.

In the meantime, Thankfully, Mail still works, which was the MIME
objective for Quoted-Printable.

Cheers;-)....\Stef

At 12:41 PM -0500 4/1/02, John C Klensin wrote:
>Stef,
>
>Either I misunderstand your message, or we have a bit of a
>disconnect.
>
>The SMTP extension "8BITMIME" was specified fairly early in the
>game.  My impression is that it is fairly widely deployed and
>used, possibly to the extent that most non-ASCII traffic is
>moving that way; someone on this list (or the SMTP extensions
>one) may have some numbers.  Where it is deployed, MIME is still
>required to label ("tag") the data -- one can still not tell one
>sort of 8bit character from another, and there is, I think,
>still much more traffic on the network in ISO 8859-N variants,
>or IS 2022-based systems, or local character sets than there is
>UTF-8 encoding of Unicode -- but quoted-printable (or base64)
>are not, unless the mail store or mail retrieval implementation
>cannot deal with 8bit materials.
>
>       john
>
>
>--On Monday, 01 April, 2002 08:59 -0800 Einar Stefferud
><[email-masked]> wrote:
>
>  > Not to pick on Dan's particular message.
>  > It was just the last one to provide me with a handy response
>  > point.
>  >
>  > Looking back, the original 1990-1991 argument was between
>  > "Just send 8 bits" with no tagging of the content, vs "tagging
>  > and bagging" to deal with the fact that 8-bit at that time
>  > would break too many systems, and the problem of needing to
>  > tag the types of text characters that were contained in RFC822
>  > message bodies.
>  >
>  > But, as there were two problems to solve (8 bit SMTP ) and
>  > (tagging RFC822 content), the decision was made (after a very
>  > long and very heated argument) to split that problem according
>  > to the boundaries of SMTP and RFC822.
>  >
>  > In fact, the two problems were and still are totally distinct.
>  >
>  > The RFC822 solution group produced MIME with Quoted-Printable
>  > as its charter specified, to assure interworkability as soon
>  > as possible.
>  >
>  > The 8-bit folks did not do anything then or since to require
>  > 8-bit carriage.
>  >
>  > So, it is time for a working group to solve this problem.
>  >
>  > It is not going to help anyone to argue about the past, as in
>  > fact, it was very rational at the time, and one side of the
>  > problem was ignored as soon as quoted printable relieved the
>  > immediate pressure for an SMTP solution.
>  >
>  > So, now is a good time for some constructive discussion of
>  > ways to solve the charset problems, hopefully with some useful
>  > charset tagging tools for the non-RFC-822 parts of the SMTP
>  > envelope.  (But I do not mean to specify either the work or
>  > the results at this point.)
>  >
>  > It is indeed unfortunate the the DRUMS WG did not find a way
>  > to solve the 8-bit SMTP problem, but now is the time to get to
>  > it.  There should not be too much 7-bit SMTP code to fix by
>  > now.
>  >
>  > Cheers...\Stef
>  >
>  >
>  >
>  >
>  > At 8:35 AM +0000 4/1/02, D. J. Bernstein wrote:
>  >> Claus Faerber writes:
>  >>  > D. J. Bernstein <[email-masked]> schrieb/wrote:
>  >>  > > I'm not saying that Quoted-Printable had no short-term
>  >>  > > benefits for
>  > its
>  >>  > > short-term costs. I'm saying that, viewed from our
>  >>  > > long-term
>  > perspective
>  >>  > > eleven years later, the failure to require 8-bit
>  >>  > > transparency was an amazingly stupid decision.
>  >>  > From our present perspective, that's true. Back then, it
>  >>  > might have been the best solution.
>  >>
>  >> No. The failure to require 8-bit transparency was
>  >> inexcusable. Every approach that failed to require 8-bit
>  >> transparency could have been dramatically improved. Consider,
>  >> for example, these three approaches:
>  >>
>  >>    (1) Quoted-Printable;
>  >>    (2) Quoted-Printable plus ``you must handle unencoded
>  >>    8-bit data too''; (3) pure 8-bit without this 7-bit
>  >>    garbage.
>  >>
>  >> Whether or not you understand that #3 would have been better
>  >> than #2, surely you understand that #2 would have been vastly
>  >> better than #1.
>  >>
>  >>  > Further, remember that the first MIME standards date back
>  >>  > to 1992. Back then, Unicode was brand-new and UTF-8 only
>  >>  > came with the 2.0 version in 1996. Without UTF-8, you just
>  >>  > could not even think about using Unicode in message
>  >>  > headers; and without Unicode, you could not solve the
>  >>  > charset-labelling problem.
>  >>
>  >> Get your facts straight. First, UTF-8 was introduced years
>  >> before 1996, although under another name. Second, even
>  >> without knowing about UTF-8, people _were_ thinking about
>  >> Unicode in headers, and proposed several workable approaches.
>  >> Third, even without Unicode, there were several solutions to
>  >> the character-set labelling problem.
>  >>
>  >> Anyway, none of this is relevant to IETF's acceptance of
>  >> 7-bit garbage in 1991, and none of it is relevant to IETF's
>  >> acceptance of 7-bit garbage today.
>  >>
>  >>  > The movement towards UTF-8 everywhere is quite new.
>  >>
>  >> Again, get your facts straight. The ``movement'' started with
>  >> X-Open, Rob Pike, and Ken Thompson a decade ago. RFC 2277,
>  >> requiring UTF-8 support for all text on the Internet, is four
>  >> years old.
>  >>
>  >> ---D. J. Bernstein, Associate Professor, Department of
>  >> Mathematics, Statistics, and Computer Science, University of
>  >> Illinois at Chicago
>  >
>  >
>  >
>  >
>  >
>  >

-
This message was passed through [email-masked],
which
is a sublist of [email-masked]. Not all messages are passed.
Decisions on what to pass are made solely by Raffaele D'Albenzio.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 2 Apr 2002 05:14:22 -0000

Dan Kohn writes:
> a UTF8HEADERS ESMTP extension

No! 8-bit support is not an option. 8-bit support is not something to be
negotiated. 8-bit support is _required_. Do you want implementors in
twenty years to be dealing with 7-bit MTAs?

> The challenge is that any mail relay implementing this

Sorry to bother you with the facts, but the 8BITMIME requirement that
you're mindlessly imitating is flagrantly violated in practice by many
MTAs, including most versions of sendmail. (All versions from 6.57 in
1993 until 8.7 in 1995; at least a few years after that if the popular 8
flag is set; I haven't checked the most recent versions.)

Furthermore, converting an 8-bit email address to 7 bits would destroy
interoperability. This isn't just a Q-P display failure; this is mail
delivered to the wrong address. Reliably undoing the conversion means
upgrading every mail program on the Internet, rather than just the 7-bit
programs that are actually at fault.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Florian Weimer <[email-masked]>
Date: Tue, 02 Apr 2002 08:48:14 +0200

"D. J. Bernstein" <[email-masked]> writes:

> negotiated. 8-bit support is _required_.

I agree.  8BITMIME has largely failed (strictly speaking, announcing
8BITMIME requires body rewrite features in mail relay software).  We
should not introduce similar functionality, as an RFC for that will
be ignored, implemented incorrectly, or not implemented due to the
involved complexity.

> MTAs, including most versions of sendmail.

qmail announces 8BITMIME support without actually implementing it,
too.  (This defect only affects mail relays, though.)

---

From: Keith Moore <[email-masked]>
Date: Tue, 02 Apr 2002 02:30:05 -0500

> be ignored, implemented incorrectly, or not implemented due to the

It's a different world now.  At the time 8BITMIME was defined few MUAs 
supported MIME, but many could display 8bit text in message bodies
as long as it wasn't MIME encoded, and sender and receiver happened 
to use the same character set.  And given the demographics of the 
Internet at that time, there was at least a fair chance that both
sender and receiver used iso-8859-1 (though many others were in use).

At the present time most MUAs support MIME encoding.  Essentially all
MUAs that support utf-8 also support MIME; thus they are able to decode 
2047 headers and use the proper character encoding for display. OTOH 
unlabeled 8bit text in message headers is not treated consistently
across MUAs, and there are many more character sets in wide use than when
8BITMIME was defined.  Though it is generally possible to distinguish
between several of the popular encodings even without tagging, it's 
probably the case that header text encoded in 2047 is more likely to 
work with existing MUAs than unencoded utf-8.

The real problem with 8BITMIME was that it essentially forced the
MTA to make a guess about whether the recipient's MUA supported MIME.
I agree that we don't want to do that again.  MTAs should be encouraged 
to handle relayed messages as transparently as possible.

However, the benefit in doing so is mostly for the long term.  It still 
won't permit us to use unencoded utf-8 in message headers in the near term. 

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 2 Apr 2002 08:08:40 -0000

By the way, in case anyone thinks that my sarcastic ``7 bits forever!''
is an exaggeration, here's an old quote from Mark Crispin: ``7-bit MTA's
will never go away. Ever. Mail goes on many paths besides IP, some of
which are forever 7-bits. You will always have to solve the problem of
getting your 8-bit mail into 7-bits for some 7-bit MTA.''

It is reasonable to demand time to upgrade, so that there are no
interoperability problems. It is not reasonable to demand that this
7-bit garbage be allowed to remain on the network forever, at the
expense of all future implementors and users.

> qmail announces 8BITMIME support without actually implementing it, too.

Of course. Announcing 8BITMIME in the SMTP server benefits the users.
Deliberately ignoring the 8BITMIME requirements in the SMTP client also
benefits the users.

I thought it was clear that I was criticizing 8BITMIME, not sendmail's
violation of it. Essentially everything that sendmail has done wrong
with 8-bit support has complied with the relevant IETF specifications,
and essentially everything that sendmail has done right with 8-bit
support has violated the relevant IETF specifications.

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Tue, 2 Apr 2002 10:19:45 +0200

> expense of all future implementors and users.

Why change? It works as designed. If we change this, we risk
fragmenting the net. Yes, it costs some more bits when 7-bit
downgrading is used, but that is not a general cost for
normal email use. And it costs some maintenance and some
space in MTAs, but that is not much in total.

keld

---

From: Dave Crocker <[email-masked]>
Date: Tue, 02 Apr 2002 00:18:42 -0800

>which are forever 7-bits. You will always have to solve the problem of

the word sarcasm is about tone, not truth.

in fact, there is quite a bit of experience that shows just how long 
established systems and services do stay around.

the difference between 20 years and forever is not worth measuring.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: Keith Moore <[email-masked]>
Date: Tue, 02 Apr 2002 08:53:56 -0500

> expense of all future implementors and users.

forever is a long time.  and yet, even if all use of quoted-printable
were to cease today, there would still be a need for new MUAs to be
able to read old messages stored on servers for many more years, perhaps
decades.

using raw utf-8 in message headers wouldn't replace 2047. instead,
it would add yet another format that MUAs are expected to parse
and display. it would not reduce complexity, it would add to it.

Keith

---

From: "David Leung \(Neteka Inc.\)" <[email-masked]>
Date: Tue, 2 Apr 2002 13:42:00 -0500

> and display. it would not reduce complexity, it would add to it.

If we add new MIME headers and ESMTP commands, MUA and MTA should be able to
have a smooth transition from 7bit encoded 8bits to UTF8. If there is a
requirement of change anyway, why not change it to something that is more
simple and well support by the Unicode Consortium.