Subject: Re: [idn] Re: 7 bits forever!

From: Einar Stefferud <[email-masked]>
Date: Mon, 1 Apr 2002 08:59:19 -0800

Not to pick on Dan's particular message.
It was just the last one to provide me with a handy response point.

Looking back, the original 1990-1991 argument was between "Just send 8 bits"
with no tagging of the content, vs "tagging and bagging" to deal with
the fact that 8-bit at that time would break too many systems, and
the problem of needing to tag the types of text characters that were
contained in RFC822 message bodies.

But, as there were two problems to solve (8 bit SMTP ) and (tagging
RFC822 content), the decision was made (after a very long and very
heated argument) to split that problem according to the boundaries of
SMTP and RFC822.

In fact, the two problems were and still are totally distinct.

The RFC822 solution group produced MIME with Quoted-Printable as its
charter specified, to assure interworkability as soon as possible.

The 8-bit folks did not do anything then or since to require 8-bit carriage.

So, it is time for a working group to solve this problem.

It is not going to help anyone to argue about the past, as in fact,
it was very rational at the time, and one side of the problem was
ignored as soon as quoted printable relieved the immediate pressure
for an SMTP solution.

So, now is a good time for some constructive discussion of ways to
solve the charset problems, hopefully with some useful charset
tagging tools for the non-RFC-822 parts of the SMTP envelope.  (But I
do not mean to specify either the work or the results at this point.)

It is indeed unfortunate the the DRUMS WG did not find a way to solve
the 8-bit SMTP problem, but now is the time to get to it.  There
should not be too much 7-bit SMTP code to fix by now.

Cheers...\Stef

At 8:35 AM +0000 4/1/02, D. J. Bernstein wrote:
>Claus Faerber writes:
>  > D. J. Bernstein <[email-masked]> schrieb/wrote:
>  > > I'm not saying that Quoted-Printable had no short-term benefits for
its
>  > > short-term costs. I'm saying that, viewed from our long-term
perspective
>  > > eleven years later, the failure to require 8-bit transparency was an
>  > > amazingly stupid decision.
>  > From our present perspective, that's true. Back then, it might
>  > have been the best solution.
>
>No. The failure to require 8-bit transparency was inexcusable. Every
>approach that failed to require 8-bit transparency could have been
>dramatically improved. Consider, for example, these three approaches:
>
>    (1) Quoted-Printable;
>    (2) Quoted-Printable plus ``you must handle unencoded 8-bit data too'';
>    (3) pure 8-bit without this 7-bit garbage.
>
>Whether or not you understand that #3 would have been better than #2,
>surely you understand that #2 would have been vastly better than #1.
>
>  > Further, remember that the first MIME standards date back to 1992.
>  > Back then, Unicode was brand-new and UTF-8 only came with the 2.0
>  > version in 1996. Without UTF-8, you just could not even think
>  > about using Unicode in message headers; and without Unicode, you
>  > could not solve the charset-labelling problem.
>
>Get your facts straight. First, UTF-8 was introduced years before 1996,
>although under another name. Second, even without knowing about UTF-8,
>people _were_ thinking about Unicode in headers, and proposed several
>workable approaches. Third, even without Unicode, there were several
>solutions to the character-set labelling problem.
>
>Anyway, none of this is relevant to IETF's acceptance of 7-bit garbage
>in 1991, and none of it is relevant to IETF's acceptance of 7-bit
>garbage today.
>
>  > The movement towards UTF-8 everywhere is quite new.
>
>Again, get your facts straight. The ``movement'' started with X-Open,
>Rob Pike, and Ken Thompson a decade ago. RFC 2277, requiring UTF-8
>support for all text on the Internet, is four years old.
>
>---D. J. Bernstein, Associate Professor, Department of Mathematics,
>Statistics, and Computer Science, University of Illinois at Chicago

---

From: Dave Crocker <[email-masked]>
Date: Mon, 01 Apr 2002 09:34:57 -0800

>should not be too much 7-bit SMTP code to fix by now.

DRUMS was not permitted to create new technology.

Its chartered task was to eliminate known protocol bugs and clean up 
existing documentation.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Apr 2002 12:41:57 -0500

Stef,

Either I misunderstand your message, or we have a bit of a
disconnect.

The SMTP extension "8BITMIME" was specified fairly early in the
game.  My impression is that it is fairly widely deployed and
used, possibly to the extent that most non-ASCII traffic is
moving that way; someone on this list (or the SMTP extensions
one) may have some numbers.  Where it is deployed, MIME is still
required to label ("tag") the data -- one can still not tell one
sort of 8bit character from another, and there is, I think,
still much more traffic on the network in ISO 8859-N variants,
or IS 2022-based systems, or local character sets than there is
UTF-8 encoding of Unicode -- but quoted-printable (or base64)
are not, unless the mail store or mail retrieval implementation
cannot deal with 8bit materials.

      john

--On Monday, 01 April, 2002 08:59 -0800 Einar Stefferud
<[email-masked]> wrote:

> Looking back, the original 1990-1991 argument was between
> "Just send 8 bits" with no tagging of the content, vs "tagging
> and bagging" to deal with the fact that 8-bit at that time
> would break too many systems, and the problem of needing to
> tag the types of text characters that were contained in RFC822
> message bodies.
> 
> But, as there were two problems to solve (8 bit SMTP ) and
> (tagging RFC822 content), the decision was made (after a very
> long and very heated argument) to split that problem according
> to the boundaries of SMTP and RFC822.
> 
> The RFC822 solution group produced MIME with Quoted-Printable
> as its charter specified, to assure interworkability as soon
> as possible.
> 
> It is not going to help anyone to argue about the past, as in
> fact, it was very rational at the time, and one side of the
> problem was ignored as soon as quoted printable relieved the
> immediate pressure for an SMTP solution.
> 
> So, now is a good time for some constructive discussion of
> ways to solve the charset problems, hopefully with some useful
> charset tagging tools for the non-RFC-822 parts of the SMTP
> envelope.  (But I do not mean to specify either the work or
> the results at this point.)
> 
> It is indeed unfortunate the the DRUMS WG did not find a way
> to solve the 8-bit SMTP problem, but now is the time to get to
> it.  There should not be too much 7-bit SMTP code to fix by
> now.
> 
> Cheers...\Stef
> 
> 
> 
> 
>>  > > eleven years later, the failure to require 8-bit
>>  > > transparency was an amazingly stupid decision.
>>  > From our present perspective, that's true. Back then, it
>>  > might have been the best solution.
>> 
>> No. The failure to require 8-bit transparency was
>> inexcusable. Every approach that failed to require 8-bit
>> transparency could have been dramatically improved. Consider,
>> for example, these three approaches:
>> 
>>    (1) Quoted-Printable;
>>    (2) Quoted-Printable plus ``you must handle unencoded
>>    8-bit data too''; (3) pure 8-bit without this 7-bit
>>    garbage.
>> 
>> Whether or not you understand that #3 would have been better
>> than #2, surely you understand that #2 would have been vastly
>> better than #1.
>> 
>>  > Further, remember that the first MIME standards date back
>>  > to 1992. Back then, Unicode was brand-new and UTF-8 only
>>  > came with the 2.0 version in 1996. Without UTF-8, you just
>>  > could not even think about using Unicode in message
>>  > headers; and without Unicode, you could not solve the
>>  > charset-labelling problem.
>> 
>> Get your facts straight. First, UTF-8 was introduced years
>> before 1996, although under another name. Second, even
>> without knowing about UTF-8, people _were_ thinking about
>> Unicode in headers, and proposed several workable approaches.
>> Third, even without Unicode, there were several solutions to
>> the character-set labelling problem.
>> 
>> Anyway, none of this is relevant to IETF's acceptance of
>> 7-bit garbage in 1991, and none of it is relevant to IETF's
>> acceptance of 7-bit garbage today.
>> 
>> Again, get your facts straight. The ``movement'' started with
>> X-Open, Rob Pike, and Ken Thompson a decade ago. RFC 2277,
>> requiring UTF-8 support for all text on the Internet, is four
>> years old.
>> 
>> ---D. J. Bernstein, Associate Professor, Department of
>> Mathematics, Statistics, and Computer Science, University of
>> Illinois at Chicago
> 
> 
> 
> 
> 
>

---

From: Einar Stefferud <[email-masked]>
Date: Mon, 1 Apr 2002 11:47:14 -0800

Hi John -- My thesis is that the problem lies now in the SMTP 
standards, where the current "Send 8-bits" problem has always 
resided.  The MIME working group could not solve that problem, and so 
did not, but it made mail work since its deployment, which was the 
objective in the MIME WG.  inside the RFC822 Body there is only one 
way to handle 8-bit data, and that is to tag it, and bag it.  Even if 
you can transmit 8-bit data without encoding it in the body of an 
SMTP message, it needs to be tagged and delimited in the body.

MIME Quoted-Printable was designed to be a work-around-damage 
solution at the time, and was not design to remove the need for 
handling 8-bit data in SMTP.

So, my only point is that it appears that some work needs to be done 
in the SMTP standards, and beating up on the MIME WG does not compute 
given this observation.

That is all...  Just trying to shift the focus of this thread to 
where someone might think of a way to solve the current problem.

In the meantime, Thankfully, Mail still works, which was the MIME 
objective for Quoted-Printable.

Cheers;-)....\Stef

---

From: Keith Moore <[email-masked]>
Date: Mon, 01 Apr 2002 18:26:39 -0500

> handling 8-bit data in SMTP.

I'd put this somewhat differently - QP was designed to be deployable
in the near-term without first requiring MTAs, etc. to upgrade.
If MIME had required that all SMTP servers be upgraded before sending
MIME messages, then MIME would have failed.

> in the SMTP standards, and beating up on the MIME WG does not compute

I don't think it would be controversial to write a BCP declaring that 
SMTP MTAs should support transparent relaying of all 8bit non-control
characters in the message body.  Neither would it change much - because
most existing MTAs already provide this.

Putting UTF-8 text in the message header is a different problem entirely.

Keith