Subject: Re: gzip-8bit
From: Bruce Lilly <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Sat, 01 Mar 2003 00:22:09 -0500
List-Id: <ietf-822.imc.org>

[email-masked] wrote:

> Very unlikely, actually. CTE has always been extensible and has always allowed
> X- tokens. The rules for handling unknown CTEs are also well defined.

Actually, there is a problem; the rule seems to be the following
from RFC 2045, section 6.4:

    Any entity with an unrecognized Content-Transfer-Encoding must be
    treated as if it has a Content-Type of "application/octet-stream",
    regardless of what the Content-Type header field actually says.

That basically means that the content is treated as opaque. Obviously,
one cannot decode an unknown encoding, so that part of the CTE is
moot.  But the other part, viz. the content domain (7bit, 8bit, or
binary) is unspecified.  So an MTA presented with a message with
unknown CTE cannot tell (based on MIME header fields) whether or
not the next stage of transfer requires 8BITMIME or binary transport
or plain old 7bit transport.

I suppose the MTA in question could peek inside the message body to
determine the domain. Unless of course it's not all available (e.g.
it's in a stream too large to be held in memory) -- oops.

If compression is to be considered part of the CTE, then it is
certainly conceivable that some CTE may have binary domain, and
lacking explicit information an assumption of binary domain is
safe (it won't result in foisting incompatible content on the
next hop).  OTOH if compression is considered a separate
attribute, then there doesn't seem to be much point in a CTE with
a binary domain other than "binary", and one could assume 8bit
domain for an unknown CTE.

The danger is that as both of the 2045-specified encodings (as
opposed to the identity CTE values) have 7bit domains, there may
be implementations that presume a 7bit domain for any unrecognized
CTE (if I were a gambler, I'd bet on it).

Date: Fri, 28 Feb 2003 21:40:07 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

>   [ rejecting bytes 128-255 as an anti-spam technique ]
> > zero false positives
> 
> Wrong. A huge number of non-spam messages use bytes 128-255.

I have yet to see a single one.  of course, since I only speak English,
there's a very high probability that any message that I see that contains
those bytes in a message header is spam.  at any rate, they're all in 
violation of the standards, and as far as I'm concerned, it's reasonable to
outright reject any message that is so flagrantly broken.

Date: Fri, 28 Feb 2003 16:54:07 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: Marc Mutz <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> There's a problem hiding in the claims that deflate-8bit and friends can
> be sent over 8BITMIME-supporting channels. Consider a MIA sending a
> message with deflate-8bit body to a SMTP server announcing 8BITMIME.

> If the server isn't prepared to reject unknown charsets in the body of
> messages (and which server is?), it will accept the message and thus
> the responsibility for delivery.

> The next hop doesn't support 8BITMIME. No problem. This is an
> intelligent server that acts by the gateway model of rfc1652 and tries
> to recode the offending body parts. Whooops... bounce. :-(

> Am I missing something?

This is nothing new; nothing requires that servers in possession of an 8bitMIME
message currently be capable of downgrading. Per RFC 1652 section 3, they
already have the option of returning a bounce if the next hop doesn't support
8bitMIME.

It is entirely possible that this encoding will fail to deploy if 8bitMIME
support isn't sufficiently widespread. But it is even more likely that this
will fail to catch on if clients don't add the necessary support.

But we won't know until we try.

				Ned

Date: 1 Mar 2003 00:30:38 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Keith Moore writes:
  [ rejecting bytes 128-255 as an anti-spam technique ]
> zero false positives

Wrong. A huge number of non-spam messages use bytes 128-255.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Fri, 28 Feb 2003 19:06:01 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> Rejecting bytes 128-255 as an anti-spam technique is idiotic. If any
> noticeable percentage of the Internet started doing this, spammers would
> simply stop using those bytes. (Most of them never even started.)

the number of spammers doing this is not a concern.  a significant percentage
of spam messages are doing this.  and since the messages themselves are
illegally formatted and unlikely to be correctly displayed, it seems perfectly
reasonable to treat them as trash.  it's one of the few effective filtering
techniques with zero false positives.

From: Marc Mutz <[email-masked]>
Organization: "Old Europe" - and proud
To: [email-masked]
Subject: Re: gzip-8bit
Date: Sat, 1 Mar 2003 00:42:16 +0100
Sender: [email-masked]

--Boundary-02=_sP/X+/WbhFdBg2N
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Saturday 01 March 2003 01:00, Claus F=E4rber wrote:
<snip>
> It also includes a draft for a quoted-binary Extended CTE that allows
> efficient transfer of binary data over 8BITMIME environments.
<snip>

There's a problem hiding in the claims that deflate-8bit and friends can=20
be sent over 8BITMIME-supporting channels. Consider a MIA sending a=20
message with deflate-8bit body to a SMTP server announcing 8BITMIME.

If the server isn't prepared to reject unknown charsets in the body of=20
messages (and which server is?), it will accept the message and thus=20
the responsibility for delivery.

The next hop doesn't support 8BITMIME. No problem. This is an=20
intelligent server that acts by the gateway model of rfc1652 and tries=20
to recode the offending body parts. Whooops... bounce. :-(

Am I missing something?

Marc

=2D-=20
The illegal we do immediately.
The unconstitutional takes a bit longer.            -- Henry Kissinger

--Boundary-02=_sP/X+/WbhFdBg2N
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_sP/X+/WbhFdBg2N--

Date: Fri, 28 Feb 2003 15:33:31 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: [email-masked]
Cc: [email-masked]
Sender: [email-masked]

> Keith Moore <[email-masked]> schrieb/wrote:
> > bullfeathers.  the way MIME defines it, it's a transfer encoding.
> > it's the means used to convert from canonical form to on-the-wire
> > form.  putting compression anywhere else would break MIME.

> It's very likely that a new CTE would break MIME, too.

Very unlikely, actually. CTE has always been extensible and has always allowed
X- tokens. The rules for handling unknown CTEs are also well defined. And such
tokens have been seen fairly wide use: X-UUENCODE, for example. To the extent
there have been problems the problems have been with the use of nonstandard and
often as not poorly defined stuff.

> Another idea would be to create an Extended MIME standard (EMIME) that
> may be used only within an confined environment. This would allow to add
> new ``Extended'' CTEs, new headers such as Content-Encoding (from HTTP),
> and an extension mechanism that allows further extensions.

I really don't think the risks are sufficient to justify this sort of thing.

				Ned

Date: Fri, 28 Feb 2003 23:12:52 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> The amplification factor that Marc Mutz identified could be used as a
> DoS attacxk against an MTA as well as against an end user.
>
> Summary: I think the ideal situation would include having
> deflate-8bit, deflate-base64, and the escaping/line-breaking method
> (separate from compression),

One way to address the both the concern in the first paragraph and
the desire in the second would be to expand the MIME architecture to
incorporate the Content-Encoding field from HTTP.

The Content-Transfer-Encoding would continue to be for the express
purpose of making the message body conform to the transport
requirements.  In addition to the quoted-printable and base64
transformations, we would add escaped-8bit (or whatever people decide to
call it).

Compression would be handled separately by Content-Encoding, same as in
HTTP (which already defines gzip and deflate encodings).

The semantics of Content-Encoding are different from
Content-Transfer-Encoding.  The latter describes the message body,
not the object it represents.  But the Content-Encoding describes the
object itself.  With compression handled by Content-Encoding rather
than Content-Transfer-Encoding, MTAs would never need to decompress the
body, because transport agents at most reformat the message, but never
transcode the object itself.

AMC

Date: 01 Mar 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Keith Moore <[email-masked]> schrieb/wrote:
> bullfeathers.  the way MIME defines it, it's a transfer encoding.
> it's the means used to convert from canonical form to on-the-wire
> form.  putting compression anywhere else would break MIME.

It's very likely that a new CTE would break MIME, too.

Another idea would be to create an Extended MIME standard (EMIME) that
may be used only within an confined environment. This would allow to add
new ``Extended'' CTEs, new headers such as Content-Encoding (from HTTP),  
and an extension mechanism that allows further extensions.

It could go with an application/message-partial type (similar to  
message/partial but with EMIME content and without the restriction that  
it can't be encoded with base64 or qp), which can be used to encapsulate  
EMIME entities within MIME environments.

Some rough drafts (very rough, and including some material that I don't
consider to be a good idea any longer) can be found at:
http://www.faerber.muc.de/temp/20020404-binary-posts.html

It also includes a draft for a quoted-binary Extended CTE that allows
efficient transfer of binary data over 8BITMIME environments.

Claus
-- 
http://www.faerber.muc.de/

Date: 28 Feb 2003 22:18:08 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Rejecting bytes 128-255 as an anti-spam technique is idiotic. If any
noticeable percentage of the Internet started doing this, spammers would
simply stop using those bytes. (Most of them never even started.)

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Fri, 28 Feb 2003 16:59:21 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

> The compression isn't a transfer encoding issue per se. 

bullfeathers.  the way MIME defines it, it's a transfer encoding.
it's the means used to convert from canonical form to on-the-wire
form.  putting compression anywhere else would break MIME.

Date: Fri, 28 Feb 2003 16:57:26 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> There's nothing gained from rejecting bytes 128-255 by default.

False.  doing so filters messages that are illegally formatted,
unlikely to be presented correctly, and likely to be spam.

From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
To: Lyndon Nerenberg <[email-masked]>, Charles Lindsey <[email-masked]>
Cc: [email-masked]
Subject: RE: gzip-8bit 
Date: Fri, 28 Feb 2003 15:57:15 -0600
Sender: [email-masked]

To date, MIME is specified such that it can be generated (and sent) in a stream, without requiring the entire contents be kept in temporary memory while the length is checked and recorded in the header.  I would consider that any new CTE keep true to this valuable design principle.  This is an important characteristic for low-memory devices that record and send large stream-oriented message files (voicemail / mobile videomail).  These applications would be great candidates to use the compression/8-bit CTE's being discussed.

Greg V.

-----Original Message-----
From: Lyndon Nerenberg [mailto:[email-masked]]
Sent: Friday, February 28, 2003 3:39 PM
To: Charles Lindsey
Cc: [email-masked]
Subject: Re: gzip-8bit 

>One of the features of YEnc IIRC (and it had many "features") was an
>indication near the beginning of the total length in bytes. In the case of
>a compressed stream, of course, that should be the uncompressed length so
>that implementations can detect early on that the length exceeds the "house
>limit" and abandon the decoding.

Does yenc provide integrity protection of the length value? Or can
I lie and say the 20GB uncompress data is really just 1KB?

--lyndon

To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: gzip-8bit 
Date: Fri, 28 Feb 2003 14:38:34 -0700
From: Lyndon Nerenberg <[email-masked]>
Sender: [email-masked]

>One of the features of YEnc IIRC (and it had many "features") was an
>indication near the beginning of the total length in bytes. In the case of
>a compressed stream, of course, that should be the uncompressed length so
>that implementations can detect early on that the length exceeds the "house
>limit" and abandon the decoding.

Does yenc provide integrity protection of the length value? Or can
I lie and say the 20GB uncompress data is really just 1KB?

--lyndon

Date: 28 Feb 2003 19:23:09 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Philip Hazel writes:
> My inclination will be to add such a facility to Exim as optional,
> because, sure as eggs are eggs, somebody won't want it.

You already have a pattern-matching facility that lets people reject
sender addresses containing the letter Q if they want. There's nothing
gained from rejecting bytes 128-255 by default.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Fri, 28 Feb 2003 14:11:53 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
CC: Dan Kohn <[email-masked]>, [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Bruce Lilly wrote:

> Content-Transfer-Encoding actually specifies two nearly
> independent items; an encoding and the domain of the
> (possibly-encoded) content.  Both of the currently
> standardized encodings have a 7bit domain.

An alternative way of looking at deflate-8bit is that it
specifies *three* things; a compression method in addition
to the encoding and domain.

Other possible approaches would be to specify the compression
separately, either as a parameter (to Content-Disposition
perhaps) or via a separate "Content-Compression" header field.

The compression isn't a transfer encoding issue per se. It's
more of a disposition issue (the content could be saved either
in compressed or uncompressed form by the the recipient's UA).

Date: Fri, 28 Feb 2003 14:03:15 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Dan Kohn <[email-masked]>
CC: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Dan Kohn wrote:
> However, can you (or anyone else) comment on the interest in a 7bit
> compressed CTE.  I'm currently leaning toward doing only a deflate-8bit,
> and not also doing a deflate-base64. If an 8BitMIME MTA needs to
> downconvert for next hop delivery, it would need to implement both
> unescaping of the 8bit octets and decompression.  Do you envision this
> as a problem?
> 
> The reason is to try to minimize the number of CTEs.

A UA issue:
Currently, with QP and Base64, binary content can simply
be Base64 encoded (QP isn't terribly effective for binary
content, nor is Base64 terribly useful for text which has
a few 8-bit octets).  So the UA can pretty much do the
right thing without user interaction or user
sophistication.  Of course, a knowledgable user could
apply compression before attaching.  Unfortunately in
practice there exists a large variety of compression
and/or packaging mechanisms (e.g. just the other day I
recieved an attachment in "Stuffit" format, and I had
to track down a decoder for that).

With a larger choice for binary attachment encodings, things
get a bit more complicated.  The UA can't necessarily
determine whether the MTA will support 8BITMIME (the user
may be off-line). Yet a choice must be made among the
available encodings. Negotiation along the lines
of RFC 3297 might be useful.  Simply asking the user to
choose probably isn't practical, as the vast majority of
users have little technical sophistication.

However, the UA could presumably choose between a straight
base64 and deflate-base64 based on how well the binary
content compresses.  Likewise it could choose between
deflate-8bit and some encoding/line-breaking method to
convert binary content to 8bit-compatible form if such
a method were available.

The amplification factor that Marc Mutz identified could
be used as a DoS attacxk against an MTA as well as against
an end user.  E.g. if somebody intended to attack an MTA
which does not support 8BITMIME, the same issues that
Marc identified will come into play, and could be further
amplified by the number of envelope recipients. If
deflate-base64 is used, that issue does not arise for the
MTAs, and users benefit by having a standard compression
method rather than a hodgepodge.

Marc's other point about possibly separating the translation
and line-breaking from deflate is also interesting. In the
case of binary content which does not compress well, that
could be used to encode the content for transfer over an
8BITMIME link, just as base64 alone (w/o the deflate
compression) could be used if 8BITMIME is known to be
unavailable.

Content-Transfer-Encoding actually specifies two nearly
independent items; an encoding and the domain of the
(possibly-encoded) content.  Both of the currently
standardized encodings have a 7bit domain.

Summary: I think the ideal situation would include having
deflate-8bit, deflate-base64, and the
escaping/line-breaking method (separate from compression),
combined with support for RFC 3297 negotiation.  That way
it is possible for UAs and MTAs to handle transfer
compatibly and transparently for unsophisticated users,
and the negotiation puts the onus on the sender to provide
an uncompressed alternative if the reciever or intermediate
MTA determines that deflation would exceed some size
limit.

From: Marc Mutz <[email-masked]>
Organization: "Old Europe" - and proud
To: [email-masked]
Subject: Re: gzip-8bit
Date: Fri, 28 Feb 2003 17:21:38 +0100
Sender: [email-masked]

--Boundary-03=_Ty4X+/Qnqy4Zcb9
Content-Type: multipart/mixed;
  boundary="Boundary-01=_Sy4X+en5gyUgZ/C"
Content-Transfer-Encoding: 7bit
Content-Description: signed data
Content-Disposition: inline

--Boundary-01=_Sy4X+en5gyUgZ/C
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: body text
Content-Disposition: inline

On Monday 24 February 2003 13:07, [email-masked] wrote:
<snip>
> So who wants to do  the merge?
<snip>

I've implemented a deflate-8bit encoder for KMail using zlib.
I've used a mixture of the two drafts, actually:
1. I used the deflate algorithm (zlib's deflate()), not gzip (mainly b/c
   zlib's interface to gzip files is - well - file-based, while the
   deflate interface is stream based).
2. I didn't shift the octets other when escaping (I think it's not
   necessary, yEnc does this to get around the many-NULs problem, which
   isn't present in deflate-8bit).
3. I escaped the following octet values:
     0x00, 0x09, 0x0A, 0x0D, 0x20, 0x3D (NUL, HT, LF, CR, SP, '=3D')
   by prepending '=3D' and shifting their octet value by 64, ie.
     NUL becomes '=3D@'
     HT  becomes '=3DI'
     LF  becomes '=3DJ'
     CR  becomes '=3DM'
     SP  becomes '=3D`'
     '=3D' becomes '=3D}'
   I admit that 42 would probably create visually more pleasing escape
   sequences:
    '=3D*', '=3D3', '=3D4', '=3D7', '=3DJ', '=3Dg'
4. I've set the maximum line length to 78 (mostly to have more than one
   line with small test vectors so CRLF injection can be tested).

ad line lengths:
I think that the line length should be free. Make it that a line MUST=20
NOT be generated longer than 998, but MAY be generate with as few as 78=20
octets (counting the escape characters). Also, implementations MUST=20
accept any line length <=3D 998 and SHOULD accept arbitrary line lengths.

Injection of CRLF MUST NOT occur between the escape character and the=20
escaped octet, but robust implementations MAY accept a
  "=3D" CRLF escaped-char
sequence as being equivalent to
  "=3D" escaped-char.

ad empty bodies:
All other CTEs generate empty output if the input was emtpy. Plain=20
deflate() creates it's 8 octet header and the 4 octet trailer/checksum=20
in this case.

I suggest that implementations SHOULD NOT encode empty input with=20
deflate-*, but MUST accept both the 12 octet and the zero octet form of=20
deflate-8bit-encoded content as meaning empty decoded content.=20
analogously for deflate-base64.

Attached is the result of applying my implementation of deflate-8bit to=20
input consisting of the octet values 0x00..0xFF, in order. The CRLF is=20
present as LF, ie. not in MIME-canonical form. The foo-42 example is has=20
been created with 42 as escape shift, the other one with 64.

And two nitpicks about the naming: I think deflate-8bit is misleading,=20
since it's not the 8bit (identity) CTE that's applied to the deflate=20
output, but a shifting and/or escaping algorithm, resp. So what about=20
deflate-shifted? Or shifted-deflate (and thus base64-deflate)? Then=20
shifted could even conceivably be used separately, e.g. for already=20
compressed files. The second nitpick is the use of '-', which in=20
quoted-printable doesn't have any special meaning, but here separates=20
two independent layers. So how about making this deflate+shifted (or=20
shifted+deflate) and deflate+base64. This could be a naming convention=20
in case someone wants to use another algorithm on either side of the +=20
in the future, e.g. deflate+base85 or bzip2+shifted...

I knew you wouldn't like this :-)

Marc

=2D-=20
memAlloc() Amnesia Error: Out of Memory

--Boundary-01=_Sy4X+en5gyUgZ/C
Content-Type: application/octet-stream;
  name="deflate-8bit-example"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="deflate-8bit-example"

eNoBPUAB//49QAECAwQFBgcIPUk9SgsMPU0ODxAREhMUFRYXGBkaGxwdHh89YCEiIyQlJicoKSor
LC0uLzAxMjM0NTY3ODk6Ozw9fT4/CkBBQkNERUZHSElKS0xNTk9QUVJTVFVWV1hZWltcXV5fYGFi
Y2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6e3x9fn+AgYKDhIWGh4iJiouMjQqOj5CRkpOUlZaXmJma
m5ydnp+goaKjpKWmp6ipqqusra6vsLGys7S1tre4ubq7vL2+v8DBwsPExcbHyMnKy8zNzs/Q0dLT
1NXW19jZ2tsK3N3e3+Dh4uPk5ebn6Onq6+zt7u/w8fLz9PX29/j5+vv8/f7/rfZ/gQ==

--Boundary-01=_Sy4X+en5gyUgZ/C
Content-Type: application/octet-stream;
  name="deflate-8bit-example-42"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="deflate-8bit-example-42"

eNoBPSoB//49KgECAwQFBgcIPTM9NAsMPTcODxAREhMUFRYXGBkaGxwdHh89SiEiIyQlJicoKSor
LC0uLzAxMjM0NTY3ODk6Ozw9Zz4/CkBBQkNERUZHSElKS0xNTk9QUVJTVFVWV1hZWltcXV5fYGFi
Y2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6e3x9fn+AgYKDhIWGh4iJiouMjQqOj5CRkpOUlZaXmJma
m5ydnp+goaKjpKWmp6ipqqusra6vsLGys7S1tre4ubq7vL2+v8DBwsPExcbHyMnKy8zNzs/Q0dLT
1NXW19jZ2tsK3N3e3+Dh4uPk5ebn6Onq6+zt7u/w8fLz9PX29/j5+vv8/f7/rfZ/gQ==

--Boundary-01=_Sy4X+en5gyUgZ/C--

--Boundary-03=_Ty4X+/Qnqy4Zcb9
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-03=_Ty4X+/Qnqy4Zcb9--

From: Marc Mutz <[email-masked]>
Organization: "Old Europe" - and proud
To: [email-masked]
Subject: Re: gzip-8bit
Date: Fri, 28 Feb 2003 01:27:23 +0100
Sender: [email-masked]

--Boundary-02=_rzqX+BYey6lVX0h
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Friday 28 February 2003 00:30, Adam M. Costello wrote:
<snip>
> It's not just content-encodings, but content-types too.  For example,
> I once created a 300kB PNG that represents a 300MB image.
<snip>

The difference being that you can still save the png to disk w/o a=20
problem while the "a" x 1e9 attachment doesn't let you...

Marc

=2D-=20
It is truly ironic that the United States, once the beacon for
promoting the principles of freedom of expression, is now
systematically infecting other countries with this dangerous public
policy choice [the DMCA] that will restrict more speech than any law
before it.    -- EFF FTAA Alert:
                 Stop Hollywood Forcing Technology Ban on 34 Countries

--Boundary-02=_rzqX+BYey6lVX0h
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_rzqX+BYey6lVX0h--

content-class: urn:content-classes:message
Subject: RE: gzip-8bit
Date: Fri, 28 Feb 2003 08:00:13 -0800
Importance: normal
From: "Dan Kohn" <[email-masked]>
To: "Steve Dorner" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Steve Dorner wrote:

> My real point is just to say "go team".  I find a non-binary
> compressed CTE to be a FAR better thing than a binary one.  Yes,
> there are some icky issues here and there, but they pale in
> comparison to the fundamental restructuring that binary CTE's
> require.  I won't support a binary CTE anytime soon, but an 8-bit
> compressed one, I will do.

Steve, just to be crystal clear, I don't think anyone is advocating a
binary compressed CTE, since an 8bit compressed one (simply encoding
NUL, CRL, LF, an escape character (=), and maybe SP and TAB), adds only
2 to 4% overhead.

However, can you (or anyone else) comment on the interest in a 7bit
compressed CTE.  I'm currently leaning toward doing only a deflate-8bit,
and not also doing a deflate-base64. If an 8BitMIME MTA needs to
downconvert for next hop delivery, it would need to implement both
unescaping of the 8bit octets and decompression.  Do you envision this
as a problem?

The reason is to try to minimize the number of CTEs.

> So count us as a commercial vendor "on board".

Glad to hear it.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Fri, 28 Feb 2003 10:20:35 -0500
From: "Simon St.Laurent" <[email-masked]>
Subject: Re: draft-freed-mime-p4-00.txt
To: [email-masked]
Sender: [email-masked]

Joseph Reagle writes:
>Regardless, this isn't a biggie, it just seems simpler to me to not
>look for "endorsements" in the tree in any case, but instead look at
>the status/level and quality of the spec, and if the IETF and W3C are
>comfortable sitting next to each other on a branch, that's that.
><smile/>

I have some concerns about the "IETF and W3C... sitting next to each
other on a branch", particularly regarding architectural issues.  

It seems fairly clear that the two organizations have different
perspectives on a regular basis - a notable recent case was www-tag
discussion of RFC 3205, where the W3C appears to be making an effort to
wave off application of 3205 to Web Services architecture.  Larry
Masinter masterfully skewered this at:

http://lists.w3.org/Archives/Public/www-tag/2003Feb/0208.html

but HTTPSubstrate-16 still lurks out there, and has for months.

While the IETF and W3C appear to agree about the value of MIME types,
there are a number of issues where registrations could in fact have a
significant impact on other aspects of Internet architecture, notably
fragment identifers because of their content-type-sensitive connections
to URI processing.

Roy Fielding's comments on XPointer seem to raise these questions quite
strongly:

http://lists.w3.org/Archives/Public/www-xml-linking-comments/2002OctDec/
0039.html

While I disagree with some of his points, these are discussions that I
would expect to see emerge within the RFC process on ietf-xml-mime or
elsewhere, and not necessarily discussions I think the W3C particularly
encourages.

It is possible that sections 3.3.1 and 3.3.2 of the draft address these
concerns:
----------------------------
3.3.1 Preliminary Community Review

   In all cases notice of a potential media type registration may be
   sent to the "[email-masked]" mailing list for review.  This
   mailing list has been established for the purpose of reviewing
   proposed media and access types.

   The intent of the public posting is to solicit comments and feedback
   on the choice of type/subtype name, the unambiguity of the references
   with respect to versions and external profiling information, and a
   review of any interoperability or security considerations.  The
   submitter may submit a revised registration, or abandon the
   registration completely, at any time.

3.3.2 IESG Approval

   Media types registered in the standards tree MUST be approved by the
   IESG prior to registration.
----------------------------

but I fear that discussion here has made these steps seem rather pro
forma.

It's up to the IETF to decide how much trust it has for the W3C and
other submitters, but I would ask that this discussion pause to consider
that some MIME registrations involve a good deal more than choosing a
name, filling out a form, and passing it along.  "A review" is good; "a
substantive discussion" is better.

-- 
Simon St.Laurent
Ring around the content, a pocket full of brackets
Errors, errors, all fall down!
http://simonstl.com -- http://monasticxml.org

Date: Thu, 27 Feb 2003 20:22:02 -0600
To: [email-masked]
From: Steve Dorner <[email-masked]>
Subject: Re: gzip-8bit
Sender: [email-masked]

At 1:17 AM -0500 2/27/03, Bruce Lilly wrote:
>there could be some pathological POP3 or IMAP implementations.

Ahem.  I can assure you there are plenty of pathological 
implementations of POP3 and IMAP.  I might go so far as to say that 
ANY implementation of IMAP is by definition ... but I digress.

My real point is just to say "go team".  I find a non-binary 
compressed CTE to be a FAR better thing than a binary one.  Yes, 
there are some icky issues here and there, but they pale in 
comparison to the fundamental restructuring that binary CTE's 
require.  I won't support a binary CTE anytime soon, but an 8-bit 
compressed one, I will do.

So count us as a commercial vendor "on board".

To: IETF-Announce: ;
CC: [email-masked]
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-freed-mime-newenc-00.txt
Date: Fri, 28 Feb 2003 07:24:01 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: Deflate-8bit and Deflate-base64: Compression 
                          Content-Transfer-Encodings for MIME
	Author(s)	: N. Freed
	Filename	: draft-freed-mime-newenc-00.txt
	Pages		: 12
	Date		: 2003-2-27
	
This document defines two additional MIME content-transfer-encodings,
deflate-8bit and deflate-base64.  Adding these CTEs to MIME that
provide facilities for loss-less, adaptive, general-purpose
compression.  The first of these, deflate-8bit, produces 8bit output,
while the second, deflate-base64, produces the same sort of output as
the base64 content-transfer-encoding defined in RFC 2045.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-freed-mime-newenc-00.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-freed-mime-newenc-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-freed-mime-newenc-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-freed-mime-newenc-00.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-freed-mime-newenc-00.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

To: [email-masked]
Xref: clerew local.mime:2477
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: gzip-8bit
Date: Fri, 28 Feb 2003 11:22:17 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Marc Mutz <[email-masked]> writes:

>So there you have a nice DoS attack: send a 1M message with all "a"s and=20
>deflate-8bit to someone you don't like and see his box grind to a halt=20
>under the memory pressure of a 1G attachment....

One of the features of YEnc IIRC (and it had many "features") was an
indication near the beginning of the total length in bytes. In the case of
a compressed stream, of course, that should be the uncompressed length so
that implementations can detect early on that the length exceeds the "house
limit" and abandon the decoding.

The Encoding we are discussing is intended for use with seriously large
files. It would be nice also if some machinery could be brought in at the
same time to enable advance computation of the total length of a
message/partial (or something in lieu of message/partial).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Arnt Gulbrandsen <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Unicode newsgroup name options
Date: Fri, 28 Feb 2003 11:06:37 +0100
Sender: [email-masked]

Several people have told me, in the past month or two, about configuring 
their MTAs to reject untagged 8-bit mail. Looking at my personal mail 
archive, it certainly seems that untagged 8-bit mail tends to be spam. 
So maybe the right question is: Which MTAs can be configure to reject 
e.g. untagged 8-bit mail?

On a similar vein, I run zmailer at home. It rejects mail containing 
various syntax errors, and in my experience 100% of that mail is spam.

--Arnt

Date: Fri, 28 Feb 2003 09:24:36 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
cc: [email-masked], <[email-masked]>, <[email-masked]>
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

On 27 Feb 2003, D. J. Bernstein wrote:

> Philip Hazel writes:
> > as a non-combatant
>
> My notes on http://pi.cr.yp.to say that Exim chokes on 8-bit bytes. Has
> that been fixed? If not, why not?

Evidence, please. Exim has always been 8-bit clean by design. (But I am
mortal; bugs are not unknown.)

Ah. Perhaps you mean 8-bit bytes in the envelope, not in the message.
Indeed, Exim refuses to handle those if they occur in the domain because
it enforces the RFC 2821 rules (letters, digits, hyphens, dots only).

The reason this has not been "fixed" is that I have not been keeping up
with this issue (joining the discussion is clearly a sink of time).

Now that you have brought to my attention the existence of some
facilities whereby UTF-8 envelopes can be tested, I will do some
experiments when I get time. My inclination will be to add such a
facility to Exim as optional, because, sure as eggs are eggs, somebody
won't want it. This is, after all, a religious war...

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Thu, 27 Feb 2003 18:05:45 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: "Adam M. Costello" <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > > So there you have a nice DoS attack: send a 1M message with all "a"s
> > > and deflate-8bit to someone you don't like and see his box grind to
> > > a halt under the memory pressure of a 1G attachment....
> >
> > Excellent point, and one that certainly needs to be mentioned in the
> > security considerations section. In fact this probably is something
> > that should be mentioned in the security considerations for the
> > similar content-encodings for HTTP, if it isn't already.

> It's not just content-encodings, but content-types too.  For example,
> I once created a 300kB PNG that represents a 300MB image.  I tried
> viewing the image with mozilla on a 256MB machine, and quickly the X
> server became unresponsive, and I had to ssh from another machine and
> kill mozilla.  Presumably this attack would be similarly effective when
> delivered as a message attachment.

> MNG can do the same thing with a file on the order of 100 bytes (it
> has a way to create a uniform image of any size just by specifying its
> color).

Thanks for pointing this out; I'll add a note to this effect to the
revised media type registration procedure.

				Ned

Date: Thu, 27 Feb 2003 23:30:59 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

[email-masked] wrote:

> > So there you have a nice DoS attack: send a 1M message with all "a"s
> > and deflate-8bit to someone you don't like and see his box grind to
> > a halt under the memory pressure of a 1G attachment....
>
> Excellent point, and one that certainly needs to be mentioned in the
> security considerations section. In fact this probably is something
> that should be mentioned in the security considerations for the
> similar content-encodings for HTTP, if it isn't already.

It's not just content-encodings, but content-types too.  For example,
I once created a 300kB PNG that represents a 300MB image.  I tried
viewing the image with mozilla on a 256MB machine, and quickly the X
server became unresponsive, and I had to ssh from another machine and
kill mozilla.  Presumably this attack would be similarly effective when
delivered as a message attachment.

MNG can do the same thing with a file on the order of 100 bytes (it
has a way to create a uniform image of any size just by specifying its
color).

AMC

Date: Thu, 27 Feb 2003 14:52:16 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: Marc Mutz <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> > 7. Security Considerations
> >
> >    The deflate algorithm is complex and hence prone to implementation
> >    errors.  In particular, certain inflate implementations are known
> > to not perform sufficient checking of their input stream and hence
> > may be vulnerable to certain forms of attack.  Aside from this, the
> > new content-transfer-encodings specified in this document are believe
> > not to raise any security considerations not already present in MIME
> > itself.
> <snip>

> Huuu.... ;-)

> I'm currently hacking up experimental support for this for KMail and
> I've came across the fact that I can't give a maximum decoded size for
> a given encoded size, although the following experiments suggest that
> the decoded size converges to about ( 1000 * encoded size ) as encoded
> size tends to infinity:

> ...

> So there you have a nice DoS attack: send a 1M message with all "a"s and
> deflate-8bit to someone you don't like and see his box grind to a halt
> under the memory pressure of a 1G attachment....

Excellent point, and one that certainly needs to be mentioned in the
security considerations section. In fact this probably is something
that should be mentioned in the security considerations for the
similar content-encodings for HTTP, if it isn't already.

				Ned

From: Marc Mutz <[email-masked]>
Organization: "Old Europe" - and proud
To: [email-masked]
Subject: Re: gzip-8bit
Date: Thu, 27 Feb 2003 23:26:42 +0100
Sender: [email-masked]

--Boundary-02=_sCpX+2pRldpJil/
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Monday 24 February 2003 13:07, [email-masked] wrote:
<snip>
> 7. Security Considerations
>
>    The deflate algorithm is complex and hence prone to implementation
>    errors.  In particular, certain inflate implementations are known
> to not perform sufficient checking of their input stream and hence
> may be vulnerable to certain forms of attack.  Aside from this, the
> new content-transfer-encodings specified in this document are believe
> not to raise any security considerations not already present in MIME
> itself.
<snip>

Huuu.... ;-)

I'm currently hacking up experimental support for this for KMail and=20
I've came across the fact that I can't give a maximum decoded size for=20
a given encoded size, although the following experiments suggest that=20
the decoded size converges to about ( 1000 * encoded size ) as encoded=20
size tends to infinity:

$ perl -e 'print "a" x 10;'|gzip -9 |wc -c
     23
$ perl -e 'print "a" x 100;'|gzip -9 |wc -c
     24
$ perl -e 'print "a" x 1000;'|gzip -9 |wc -c
     29
$ perl -e 'print "a" x 10000;'|gzip -9 |wc -c
     46
$ perl -e 'print "a" x 100000;'|gzip -9 |wc -c
    133
$ perl -e 'print "a" x 1000000;'|gzip -9 |wc -c
   1004
$ perl -e 'print "a" x 10000000;'|gzip -9 |wc -c
   9738
$ perl -e 'print "a" x 100000000;'|gzip -9 |wc -c
  97072
$ perl -e 'for (1..10) { print "a" x 100000000;}'|gzip -9 |wc -c
 970502

So there you have a nice DoS attack: send a 1M message with all "a"s and=20
deflate-8bit to someone you don't like and see his box grind to a halt=20
under the memory pressure of a 1G attachment....

=46iltering on message size suddenly fails. Decoding *must* be done in=20
chunks even for seemingly small messages or implementations risk a DoS=20
vulnerability. Likewise, applications need an emergency break that=20
stops the decoding process if the decoded content gets too large.

Granted, the problem of overly large attachments already exists in the=20
normal MIME CTEs, but there, you still have the guarantee that the=20
encoded and decoded size is of the same order of magnitude. The new=20
thing here is that the maximum decoded size is three orders of=20
magnitude greater than for any other CTE. I'd think that many DoS=20
protections in current MUAs are built with the same-order assumption=20
implicitly used all over the place...

To summarize: The security considerations section should warn about this=20
fact and if there is a guaranteed upper limit on the decoded size as a=20
function of encoded size, that function should be included in the=20
document. Usage of the rfc2183 Content-Disposition with a size=20
parameter should be made a SHOULD, even though that parameter can be=20
most easily faked, of course (I'm a bit disappointed that rfc2183=20
doesn't mention that one shouldn't rely on the size parameter and=20
probably stop processing if the decoding starts producing more than=20
double the size announced...)

Marc

=2D-=20
'When you see the ping of death, duck and cover.'
                        -- Bruce Schneier, Crypto-Gram Oct 2002

--Boundary-02=_sCpX+2pRldpJil/
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_sCpX+2pRldpJil/--

Date: 27 Feb 2003 22:31:54 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Philip Hazel writes:
> as a non-combatant

My notes on http://pi.cr.yp.to say that Exim chokes on 8-bit bytes. Has
that been fixed? If not, why not?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Thu, 27 Feb 2003 21:16:47 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip vs. deflate (was RE: gzip-8bit)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Dan Kohn <[email-masked]> wrote:

> The main advantage of gzip that I can see is the CRC32 and

Ah, I hadn't realized how much weaker the Adler32 checksum is compared
to CRC32, but I just poked around the web, and it looks like there is
mounting evidence that the difference is significant, especially for
small inputs (less than several kB).

In defense of zlib (not that it's mine to defend--I had nothing to do
with it), I'd like to point out that zlib was designed at the same
time as its first application, PNG, which already provides a CRC32
on the compressed data.  The zlib checksum on the uncompressed data
was therefore serving only to detect errors in the deflate/inflate
implementations, not to detect corruption of the PNG stream.  The zlib
designers may have expected that to be the typical case: the zlib
checksum would be in addition to a higher-layer checksum.

I now see why Content-MD5 is relevant.  If Content-MD5 is present, it
is already plenty strong enough, so it doesn't matter whether there is
another checksum on the same data, or what it is.  But if Content-MD5 is
absent, and the transfer-encoding checksum is the only end-to-end check,
then its strength matters.

> the ease of piping things through the gzip program.

Yes.  I had thought that there was a flip side of that coin, that the
lack of a libgzip was a disadvantage for implementations that want to
avoid the performance cost of a pipe, but I just learned that zlib
supports gzip streams too.

AMC

From: "Shmuel (Seymour J.) Metz" <[email-masked]>
Date: Thu, 27 Feb 2003 12:37:05 -0500
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Atid/2
Sender: [email-masked]

In <[email-masked]>,
on 02/25/2003
   at 01:33 PM, Mark Crispin <[email-masked]> said:

>The choice is simple.  Go with UTF-8 in headers, and face substantial
>opposition; or go with something that stays compliant with RFC 2822
>(such as the punycode method suggested by the Kohn draft) and not be
>opposed.

No. The choice is go with UTF-8 in headers and be opposed or try to
make news comply with RFC 2822 and be opposed. There is no option that
won't have opposition. And UTF-8 is not the only area of contention.

Instead of asking whether each possibility will be opposed, we need to
ask whether the reasons given for the opposition make sense in the
context of Usenet and whether the IESG and IETF will consider the
opposition to be reasonable.
 
-- 
-----------------------------------------------------------
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2
     Team OS/2
     Team PL/I
-----------------------------------------------------------

From: Arnt Gulbrandsen <[email-masked]>
To: Dan Kohn <[email-masked]>
Subject: Re: gzip vs. deflate (was RE: gzip-8bit)
Cc: IETF RFC-822 list <[email-masked]>
Date: Thu, 27 Feb 2003 18:38:29 +0100
Sender: [email-masked]

Dan Kohn writes:
> If we do gzip, the FNAME parameter will definitely be a SHOULD NOT 
> set, MUST ignore, since Content-Disposition with RFC 2231 can do the 
> same thing with i18n.

You also need some similar rules for MTIME vs. modification-date-parm. 
There may be more, although I can't think of anything right now.

> The main advantage of gzip that I can see is the CRC32

And the main disadvantage is an ugly paragraph of SHOULD NOT, MUST, 
SHOULD NOT, MUST. And the attendant extra code in careful readers.

> and the ease of piping things through the gzip program.

Isn't that the argument for shar over tar?

Sadly, I feel sure that the programmers that are swayed by the ease of 
system("gzip") vs. calling zlib are the programmers who don't implement 
new RFCs (correctly) anyway.

--Arnt

content-class: urn:content-classes:message
Subject: gzip vs. deflate (was RE: gzip-8bit)
Date: Thu, 27 Feb 2003 07:19:07 -0800
Importance: normal
From: "Dan Kohn" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>
Cc: "IETF RFC-822 list" <[email-masked]>
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Deflate fits better architecturally, IMO. gzip has an uncomfortable
> overlap with Content-Disposition.

> Having to look for filenames both in content-type and
> content-disposition is bad enough. That it might be in the gzip
> meta-information as well is No Fun At All.

If we do gzip, the FNAME parameter will definitely be a SHOULD NOT set,
MUST ignore, since Content-Disposition with RFC 2231 can do the same
thing with i18n.

The main advantage of gzip that I can see is the CRC32 and the ease of
piping things through the gzip program.

Ned and I are still very open to being convinced either way.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

From: Arnt Gulbrandsen <[email-masked]>
To: Dan Kohn <[email-masked]>
Subject: Re: gzip-8bit
Cc: IETF RFC-822 list <[email-masked]>
Date: Thu, 27 Feb 2003 11:02:41 +0100
Sender: [email-masked]

Dan Kohn writes:
> Thanks for the HTTP reference; that's quite helpful. Any view on 
> whether it makes more sense to mandate gzip versus deflate combined 
> with a Content-MD5?

Deflate fits better architecturally, IMO. gzip has an uncomfortable 
overlap with Content-Disposition.

Having to look for filenames both in content-type and 
content-disposition is bad enough. That it might be in the gzip 
meta-information as well is No Fun At All.

--Arnt

Date: Thu, 27 Feb 2003 09:03:28 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: [email-masked], <[email-masked]>, <[email-masked]>
cc: "D. J. Bernstein" <[email-masked]>
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

On Wed, 26 Feb 2003, Bruce Lilly wrote:

>
> D. J. Bernstein wrote:
>
> >    * We can use GoofyCode in various places.
>
> Is the "Goofy" in "GoofyCode" supposed to be longhand for "q"
> (as in "Goofymail") or for "djb" (as in "Goofydns")?  If
> not please do try to use the correct term for whatever it is
> you're unsucessfully attempting to refer. Otherwise your
> rantings are likely to be dismissed as nothing more than
> ranting.

I am avoiding entering this discussion in general. However, as a
non-combatant reading it, it is perfectly obvious to me that what Dan
means by "GoofyCode" is "one of the many non-UTF-8 encodings that has
been proposed", such as Punycode or whatever.

Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Thu, 27 Feb 2003 01:17:33 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Dan Kohn <[email-masked]>
CC: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Dan Kohn wrote:

> I may have missed something obvious, but my quick read of
> RFC 1939 says that POP3 only supports RFC 822 messages.  Can someone
> comment on POP3's capabilities to deal with 8BitMIME and binary content?

POP3 message transfer (header, empty line, body) is much like SMTP;
after a RETR command, the server sends a response code followed by
byte-stuffed lines, ending the transfer with a lone dot on a line.
And as such it is possible for an implementation to be nearly
binary compatible (the data ends with CRLF.CRLF, and the optional
TOP command is expected to return some number of message _lines_).
But the interesting question is whether or not some implementor
has coded in a hard line limit below 998 octets. RFC 1939 mentions
a 512 character (incl. CRLF) limit on response lines; if a careless
implementor used that for message lines, one could have a 510 octet
limit. With few exceptions, POP doesn't really concern itself with
message content, it just moves the data from point A to point B.

> Can conforming IMAP4 implementations only support 7bit bodies?  If so,
> what the capabilities negotiation to indicate 8BitMIME support?

IMAP can transfer 8-bit body content, or can encode binary content
for transmission between server and client.  IMAP delves
heavily into the message structure so as to be able to provide
information derived from message and MIME part header fields, etc.
and explicitly only allows standard 7-bit header content. I don't
think there's any line length limit mentioned in the IMAP RFCs;
data transmission is generally a byte count followed by the data,
so I wouldn't expect a line length limit per se -- but I suppose
it's possible that some implementation has a limit.

 From a practical point of view, all of my mail goes through a couple
of POP3 transfers, one of which is actually implemented by an IMAP
server.  I have no trouble with 8-bit body content, though 8-bit
header content can be mangled (all 8-bit-set octets replaced with
ASCII 'X') by the IMAP implementation (and that's easy to patch).
And I've seen lines longer than 998 octets.  But I may be lucky;
there could be some pathological POP3 or IMAP implementations.

Date: 27 Feb 2003 05:19:24 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Chris Newman writes:
> I know of systems which will: strip some of the 8-bit characters,
> downconvert to 2047 with "UNKNOWN" charset label, convert all 8-bit
> characters to '?', and bounce the message before it gets in the
> message store.

Please identify the systems that you're talking about.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Thu, 27 Feb 2003 04:40:46 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Dan Kohn <[email-masked]> wrote:

> All true and useful background.  But do you have a view on the subject?

I think zlib/deflate is technically more appropriate for a
transfer-coding than gzip/deflate, because the latter includes superfluous
meta-info (like original filename and operating system) and uses a more
expensive checksum.

gzip is generally available as a standalone program, which makes it easy
to whip up a quick-and-dirty implementation of gzip-8bit that involves
forking an external process.

zlib is generally available as a library, which is good for writing a
high-performance implementation of deflate-8bit inside the main program,
but not as easy as the quick-and-dirty implementation of gzip-8bit.

I don't think consideration of Content-MD5 adds any arguments on either
side.

AMC

Date: Wed, 26 Feb 2003 23:01:13 -0500
From: Lawrence Greenfield <[email-masked]>
To: [email-masked], Dan Kohn <[email-masked]>
Subject: Re: gzip-8bit
Sender: [email-masked]

   From: "Dan Kohn" <[email-masked]>
[...]
   Good point.  I may have missed something obvious, but my quick read of
   RFC 1939 says that POP3 only supports RFC 822 messages.  Can someone
   comment on POP3's capabilities to deal with 8BitMIME and binary content?
   Presumably, at least 8BitMIME is supported, or else POP3 servers would
   have been corrupting all sorts of European charsets for years.

"RFC 822 messages" include messages with 8-bit bodies, as RFC 822 has
been updated by RFC 2045. (Ok, you can split hairs between "Standard"
and "Draft Standard" but it isn't useful.) Any reasonable POP
implementation won't corrupt such messages, but there are a lot of
wacky POP implementations.

It's not worth worrying about.

   Can conforming IMAP4 implementations only support 7bit bodies?  If so,
   what the capabilities negotiation to indicate 8BitMIME support?

I am not aware of any IMAP implementations that have a problem with
8-bit messages. IMAP has specific protocol syntax to indicate 8-bit
transfer encoding, and is in fact extensible:

body_fld_enc    ::= (<"> ("7BIT" / "8BIT" / "BINARY" / "BASE64"/
                    "QUOTED-PRINTABLE") <">) / string

Larry

content-class: urn:content-classes:message
Subject: RE: gzip-8bit
Date: Wed, 26 Feb 2003 19:47:09 -0800
Importance: normal
From: "Dan Kohn" <[email-masked]>
To: "IETF RFC-822 list" <[email-masked]>
Sender: [email-masked]

All true and useful background.  But do you have a view on the subject?
;-)

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

-----Original Message-----
From: Adam M. Costello
[mailto:[email-masked]] 
Sent: Wednesday, February 26, 2003 19:28
To: IETF RFC-822 list
Subject: Re: gzip-8bit

Dan Kohn <[email-masked]> wrote:

> Any view on whether it makes more sense to mandate gzip versus deflate
> combined with a Content-MD5?

As far as I can imagine, the issues of interactions between gzip/deflate
and Content-MD5 would be the same as the issues of interactions between
zlib/deflate and Content-MD5.  In either case, the gzip or zlib trailer
includes a 32-bit hash of the uncompressed data, so about one in 4e9
random corruptions will go undetected.  The Content-MD5 provides an
independent 128-bit hash on the very same data.  Having both is thus
equivalent to a 160-bit hash on the uncompressed data, which would allow
about one in 1e48 random corruptions to go undetected.

The 32-bit hashes in gzip and zlib (CRC32 for gzip, Adler32 for zlib)
are not cryptographic; they are designed to catch random corruption,
not malicious corruption.  MD5 is a cryptographic hash, designed to
catch malicious corruption, but that only works if you can protect the
MD5 hash itself from malicious corruption, which is not true for the
Content-MD5 header field.  A cryptographic hash is no better for this
purpose than a non-cryptographic hash.  I assume MD5 was chosen not
because it is a cryptographic hash, but because it was the most widely
implemented greater-than-32-bit hash at the time.

CRC was designed to be fast and simple in hardware.  It can be made
reasonably fast in software using lookup tables.  Adler32 was designed
to be fast and simple in software (it's about 4 times faster).

AMC

Date: Thu, 27 Feb 2003 03:28:02 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Dan Kohn <[email-masked]> wrote:

> Any view on whether it makes more sense to mandate gzip versus deflate
> combined with a Content-MD5?

As far as I can imagine, the issues of interactions between gzip/deflate
and Content-MD5 would be the same as the issues of interactions between
zlib/deflate and Content-MD5.  In either case, the gzip or zlib trailer
includes a 32-bit hash of the uncompressed data, so about one in 4e9
random corruptions will go undetected.  The Content-MD5 provides an
independent 128-bit hash on the very same data.  Having both is thus
equivalent to a 160-bit hash on the uncompressed data, which would allow
about one in 1e48 random corruptions to go undetected.

The 32-bit hashes in gzip and zlib (CRC32 for gzip, Adler32 for zlib)
are not cryptographic; they are designed to catch random corruption,
not malicious corruption.  MD5 is a cryptographic hash, designed to
catch malicious corruption, but that only works if you can protect the
MD5 hash itself from malicious corruption, which is not true for the
Content-MD5 header field.  A cryptographic hash is no better for this
purpose than a non-cryptographic hash.  I assume MD5 was chosen not
because it is a cryptographic hash, but because it was the most widely
implemented greater-than-32-bit hash at the time.

CRC was designed to be fast and simple in hardware.  It can be made
reasonably fast in software using lookup tables.  Adler32 was designed
to be fast and simple in software (it's about 4 times faster).

AMC

To: "Dan Kohn" <[email-masked]>
cc: [email-masked]
Subject: Re: gzip-8bit 
Date: Wed, 26 Feb 2003 20:09:09 -0700
From: Lyndon Nerenberg <[email-masked]>
Sender: [email-masked]

>Can conforming IMAP4 implementations only support 7bit bodies?  If so,
>what the capabilities negotiation to indicate 8BitMIME support?

Unencoded 8bit textual (i.e. CRLF delimited text lines) body content is
supported by the base protocol. Non-textual 8bit and binary body content
gets transferred using the base64 encoding.

--lyndon

Date: Wed, 26 Feb 2003 19:07:23 -0800
From: Chris Newman <[email-masked]>
Subject: RE: gzip-8bit
To: Dan Kohn <[email-masked]>, [email-masked]
Sender: [email-masked]

While the POP3 spec doesn't require support for 8 bit MIME, I'm not aware 
of any implementations which have problems with 8 bit MIME.

IMAP permits 8 bit in APPEND as long as the charset is explicitly labelled, 
so IMAP standards compliant clients have to deal with 8 bit MIME and I'm 
not aware of any problems.

Now unlabelled 8 bit in the headers is a completely different story.  I 
know of systems which will: strip some of the 8-bit characters, downconvert 
to 2047 with "UNKNOWN" charset label, convert all 8-bit characters to '?', 
and bounce the message before it gets in the message store.  For any given 
message, the vast majority of clients will display the headers in a charset 
other than the sender's intention.  So it just wouldn't interoperate.

                - Chris

begin  quotation by Dan Kohn on 2003/2/26 17:32 -0800:

>
> Bruce Lilly wrote:
>
>>> BTW, does anyone know of NNTP or 8BitMIME ESMTP transport
>>> implementations that do not support 998 octets per line?
>
>> Don't forget POP and IMAP!
>
> Good point.  I may have missed something obvious, but my quick read of
> RFC 1939 says that POP3 only supports RFC 822 messages.  Can someone
> comment on POP3's capabilities to deal with 8BitMIME and binary content?
> Presumably, at least 8BitMIME is supported, or else POP3 servers would
> have been corrupting all sorts of European charsets for years.
>
> Can conforming IMAP4 implementations only support 7bit bodies?  If so,
> what the capabilities negotiation to indicate 8BitMIME support?
>
>           - dan
> --
> Dan Kohn <mailto:[email-masked]>
> <http://www.dankohn.com/>  <tel:+1-650-327-2600>

content-class: urn:content-classes:message
Subject: RE: gzip-8bit
Date: Wed, 26 Feb 2003 17:54:26 -0800
Importance: normal
From: "Dan Kohn" <[email-masked]>
To: "IETF RFC-822 list" <[email-masked]>
Sender: [email-masked]

Thanks for the HTTP reference; that's quite helpful.  Any view on
whether it makes more sense to mandate gzip versus deflate combined with
a Content-MD5?

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600> 

-----Original Message-----
From: Adam M. Costello
[mailto:[email-masked]] 
Sent: Wednesday, February 26, 2003 17:38
To: [email-masked]
Subject: Re: gzip-8bit

Lawrence Greenfield <[email-masked]> wrote:

> There's absolutely no reason to bet on probabilities here when
> counting the output octets is so easy and guarantees correct behavior
> even when faced with a smart adversary.

I agree.  Furthermore, I see no need to require any particular algorithm
for choosing the line breaks.  Simply require that lines not exceed
998 octets (plus the CRLF).  One implementation might count the output
octets in order to wait as long as possible before inserting a line
break.  Another implementation might break the input into lines of 499
octets, then escape the octets that need it, which can at worst double
the length to 998.  The simpler implementation will have 0.4% overhead
for line breaks instead of 0.2%.  Is that enough to warrant disallowing
the simpler implementation?  Should we require gzip -9 rather than the
default gzip -6?  No, any compression algorithm that produces a valid
stream should be allowed, and likewise any line breaking algorithm that
produces valid lines should be allowed.

By the way, HTTP defines both "gzip" and "deflate" content codings,
where "gzip" is deflate with a gzip header/trailer (RFC-1952), and
"deflate" is deflate with a zlib header/checksum (RFC-1951).  The
gzip header contains various meta-info that is not useful in a
transfer-encoding, while the zlib header contains just the minimal
compression parameters needed to interpret the compressed data.  You
might consider whether you prefer to define gzip-8bit or deflate-8bit.

AMC

Date: Thu, 27 Feb 2003 01:37:31 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Lawrence Greenfield <[email-masked]> wrote:

> There's absolutely no reason to bet on probabilities here when
> counting the output octets is so easy and guarantees correct behavior
> even when faced with a smart adversary.

I agree.  Furthermore, I see no need to require any particular algorithm
for choosing the line breaks.  Simply require that lines not exceed
998 octets (plus the CRLF).  One implementation might count the output
octets in order to wait as long as possible before inserting a line
break.  Another implementation might break the input into lines of 499
octets, then escape the octets that need it, which can at worst double
the length to 998.  The simpler implementation will have 0.4% overhead
for line breaks instead of 0.2%.  Is that enough to warrant disallowing
the simpler implementation?  Should we require gzip -9 rather than the
default gzip -6?  No, any compression algorithm that produces a valid
stream should be allowed, and likewise any line breaking algorithm that
produces valid lines should be allowed.

By the way, HTTP defines both "gzip" and "deflate" content codings,
where "gzip" is deflate with a gzip header/trailer (RFC-1952), and
"deflate" is deflate with a zlib header/checksum (RFC-1951).  The
gzip header contains various meta-info that is not useful in a
transfer-encoding, while the zlib header contains just the minimal
compression parameters needed to interpret the compressed data.  You
might consider whether you prefer to define gzip-8bit or deflate-8bit.

AMC

content-class: urn:content-classes:message
Subject: RE: gzip-8bit
Date: Wed, 26 Feb 2003 17:32:27 -0800
Importance: normal
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>
Sender: [email-masked]

Bruce Lilly wrote:

>> BTW, does anyone know of NNTP or 8BitMIME ESMTP transport
>> implementations that do not support 998 octets per line?

> Don't forget POP and IMAP!

Good point.  I may have missed something obvious, but my quick read of
RFC 1939 says that POP3 only supports RFC 822 messages.  Can someone
comment on POP3's capabilities to deal with 8BitMIME and binary content?
Presumably, at least 8BitMIME is supported, or else POP3 servers would
have been corrupting all sorts of European charsets for years.

Can conforming IMAP4 implementations only support 7bit bodies?  If so,
what the capabilities negotiation to indicate 8BitMIME support?

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Wed, 26 Feb 2003 20:20:15 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Dan Kohn <[email-masked]>
CC: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Dan Kohn wrote:

> Or, more specifically, Ned and I are now thinking of escaping the octets
> NUL, CR, LF, TAB, SP, and "=" (the SP and TAB prevent potential
> corruption on some buggy MTAs, and the "=" is used as the escaping
> character).

I'd be tempted to add '@' as some buggy UAs tend to grope, agonizingly
slowly, and hogging CPU resources, through binary content in the vain
hope of finding an email address.  But it's not worth encumbering a
transfer encoding with that; let user pressure for the bugware
suppliers to squash the bugs.

> The other alternative is to expect encoders to be a little smarter to
> always have a 998 line count rather than having it be probabilistic.
> The encoder could process each octet one at a time, escape the six that
> need to be, output the result while incrementing a counter by the number
> of octets outputted (either 1 or 2), add a CRLF when the counter reaches
> 997, and then reset the counter.

I noticed that in Ned's draft, and I like it.

> BTW, does anyone know of NNTP or 8BitMIME ESMTP transport
> implementations that do not support 998 octets per line?

Don't forget POP and IMAP!

From: Joseph Reagle <[email-masked]>
Organization: W3C
To: John C Klensin <[email-masked]>, Chris Lilley <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: draft-freed-mime-p4-00.txt
Date: Wed, 26 Feb 2003 19:45:29 -0500
Cc: "[email-masked]" <[email-masked]>, Dan Kohn <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Sender: [email-masked]

On Wednesday 26 February 2003 19:11, John C Klensin wrote:
> Joseph,
>
> For whatever it is worth, a registration in the "no prefix" (IETF) tree
> _is_ an endorsement.  I think it would be quite unfortunate to lose that.

If this is the case, perhaps the text should be changed to read:

   3.2.8 Publication Requirements
   Other than /-IETF-/ registrations in the standards tree, the registration
   of a data type does not imply endorsement, approval, or
   recommendation by the IANA or the IETF or even certification that the
   specification is adequate. 

> As a general observation, we need to be careful to not overgeneralize
> from the W3C experience and relationships.  There _is_ a special
> relationship between W3C and IETF, including the regular coordination
> efforts.   If we need to reflect that relationship in registration
> procedures as a special case (I hope that isn't necessary, but am still
> trying to think the comments through), then we should do that, probably
> in a separate document.  This one also has to cover organizations and
> relationships that, bluntly, we may trust less than we trust the W3C.

However, I'm not advocating the proposal above, it's only closer to what you 
seem to be suggesting. While I appreciate the special relationship between 
the IETF and W3C, it *is* possible to conceive that a media type definition 
from some other standards group (including the W3C) in the "no prefix" tree 
might have something in it that the IETF doesn't exactly agree with. While 
the IETF might trust the W3C enough to invite it to exist in the "no 
prefix" tree, I would not presume that means the IETF "endorses" every 
registration of the W3C, only that it probably won't screw up too bad (from 
the IETF point of view <grin/>) and if there is an issue we can work 
together to resolve it.

Regardless, this isn't a biggie, it just seems simpler to me to not look for 
"endorsements" in the tree in any case, but instead look at the 
status/level and quality of the spec, and if the IETF and W3C are 
comfortable sitting next to each other on a branch, that's that. <smile/>

Date: Wed, 26 Feb 2003 18:44:48 -0500
From: Lawrence Greenfield <[email-masked]>
To: Dan Kohn <[email-masked]>
Cc: [email-masked]
Subject: Re: gzip-8bit 
Sender: [email-masked]

   Date: Wed, 26 Feb 2003 15:04:19 -0800
   From: "Dan Kohn" <[email-masked]>
[...]
   The other alternative is to expect encoders to be a little smarter to
   always have a 998 line count rather than having it be probabilistic.
   The encoder could process each octet one at a time, escape the six that
   need to be, output the result while incrementing a counter by the number
   of octets outputted (either 1 or 2), add a CRLF when the counter reaches
   997, and then reset the counter.

This is what should be required. There's absolutely no reason to bet
on probabilities here when counting the output octets is so easy and
guarantees correct behavior even when faced with a smart adversary.

If there was a substantial amount of computation involved in counting
output octets, perhaps it would be worth considering. But compared to
deflate it's nothing. Don't waste your time on it.

Larry

To: "Dan Kohn" <[email-masked]>
cc: [email-masked]
Subject: Re: gzip-8bit 
Date: Wed, 26 Feb 2003 16:41:15 -0700
From: Lyndon Nerenberg <[email-masked]>
Sender: [email-masked]

>Specifically, isn't the key
>statistic I want the bucket count for the most frequently occurring
>octet divided by the size of the file divided by 900?

Perhaps. You can derive that easily from the data I gave. What I
don't know right now is if there is any locality to repeated
occurrences of a particular octet.

>Would it be possible for you to tell me, across all runs of
>900 octets across of all of your files, what is the largest count of
>those 6 octets that you see.

Do you mean for each 900 octet chunk of the file (i.e. octets 1-900,
901-1800, ...), or do you want this for every possible 900 octet region in
the file? Either way, this will help answer my locality question. I
need to modify my histogram scanner to handle these windowed views
of the data. I'll try to have some new data for you in the next day
or two (workload permitting).

--lyndon

content-class: urn:content-classes:message
Subject: RE: gzip-8bit 
Date: Wed, 26 Feb 2003 15:04:19 -0800
Importance: normal
From: "Dan Kohn" <[email-masked]>
To: "Lyndon Nerenberg" <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

This is superb, but my statistics are rusty enough that I'd appreciate
your advice on how to interpret it.  Specifically, isn't the key
statistic I want the bucket count for the most frequently occurring
octet divided by the size of the file divided by 900?  As long as that
number is relatively low, shouldn't I be safe from exceeding the 998
limit?

Or, more specifically, Ned and I are now thinking of escaping the octets
NUL, CR, LF, TAB, SP, and "=" (the SP and TAB prevent potential
corruption on some buggy MTAs, and the "=" is used as the escaping
character).  Would it be possible for you to tell me, across all runs of
900 octets across of all of your files, what is the largest count of
those 6 octets that you see.  The average count (I believe) should be
6/256 * 900, or 21.  But, what's the max?  If it's very close to 49,
than we should probably choose a line count less than 900.

The other alternative is to expect encoders to be a little smarter to
always have a 998 line count rather than having it be probabilistic.
The encoder could process each octet one at a time, escape the six that
need to be, output the result while incrementing a counter by the number
of octets outputted (either 1 or 2), add a CRLF when the counter reaches
997, and then reset the counter.

BTW, does anyone know of NNTP or 8BitMIME ESMTP transport
implementations that do not support 998 octets per line?

Also, I've taken the liberty of cc'ing ietf-822, which is the best place
to discuss a new CTE.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600> 

-----Original Message-----
From: Lyndon Nerenberg [mailto:[email-masked]] 
Sent: Sunday, January 26, 2003 13:39
To: Dan Kohn
Subject: Re: gzip-8bit 

>900 is chosen as the unescaped octet stream length because RFC 2822
>prohibits lines to exceed 998 octets, plus the ending CRLF.  Since the
>number of octets that will be escaped is not known, 900 seemed to
>provide a large amount of margin (i.e., room for 49 escaped octets).
>Since each octet output by gzip should be approximately equally likely,
>and there are 256 possibilities, there should be on average 4 / 256 *
>900 = 14 octets per line that need to be escaped.

The octet frequency distribution in gzipped files isn't as flat as you
might imagine. I ran a histogram analysis against a selection of gzipped
files I had at hand to determine just how flat the distribution of
octets is. The results are below, formatted as follows:

Column 1:  size of the file, in octets.
Column 2:  the bucket count for the least frequently occurring octet.
Column 3:  the bucket count for the most frequently occurring octet.
Column 4:  the mean of the bucket counts.
Column 5:  the standard deviation of the bucket counts.
Column 6:  the standard deviation divided by the mean.

607	0.000000	9.000000	2.371094	1.697593
.71
655	0.000000	10.000000	2.558594	1.717467
.67
708	0.000000	9.000000	2.765625	1.695511
.61
899	0.000000	10.000000	3.511719	1.932418
.55
899	0.000000	10.000000	3.511719	1.932418
.55
899	0.000000	10.000000	3.511719	1.932418
.55
1179	0.000000	12.000000	4.605469	2.244048
.48
1211	0.000000	11.000000	4.730469	2.141847
.45
1511	1.000000	16.000000	5.902344	2.398371
.40
1660	1.000000	15.000000	6.484375	2.666295
.41
2815	4.000000	22.000000	10.996094	3.487139
.31
3357	4.000000	23.000000	13.113281	3.710055
.28
8573	19.000000	53.000000	33.488281	5.825578
.17
9380	23.000000	64.000000	36.640625	6.731148
.18
9435	17.000000	65.000000	36.855469	6.604820
.17
9435	17.000000	65.000000	36.855469	6.604820
.17
16366	42.000000	87.000000	63.929688	8.777798
.13
20868	58.000000	121.000000	81.515625	9.744377
.11
21003	54.000000	145.000000	82.042969	10.826134
.13
21003	54.000000	145.000000	82.042969	10.826134
.13
21804	49.000000	169.000000	85.171875	12.860495
.15
21804	49.000000	169.000000	85.171875	12.860495
.15
23601	70.000000	125.000000	92.191406	10.737933
.11
31060	93.000000	160.000000	121.328125	13.064210
.10
34068	96.000000	173.000000	133.078125	13.709662
.10
50087	127.000000	246.000000	195.652344	18.090923
.09
58335	177.000000	298.000000	227.871094	18.606613
.08
58885	175.000000	285.000000	230.019531	17.314971
.07
70085	218.000000	334.000000	273.769531	22.023372
.08
80865	246.000000	381.000000	315.878906	22.972066
.07
113342	381.000000	509.000000	442.742188	26.927781
.06
115182	385.000000	633.000000	449.929688	31.520872
.07
152701	503.000000	684.000000	596.488281	30.160506
.05
152701	503.000000	684.000000	596.488281	30.160506
.05
188502	663.000000	891.000000	736.335938	36.784375
.04
198948	686.000000	993.000000	777.140625	36.907027
.04
213189	680.000000	1037.000000	832.769531	58.903702
.07
247590	831.000000	1261.000000	967.148438	47.645188
.04
288760	913.000000	1256.000000	1127.968750	55.693418
.04
311240	884.000000	1667.000000	1215.781250	112.340716
.09
311240	884.000000	1667.000000	1215.781250	112.340716
.09
377272	1307.000000	1602.000000	1473.718750	55.318317
.03
388020	1358.000000	2170.000000	1515.703125	73.552740
.04
485364	1713.000000	2170.000000	1895.953125	75.495536
.03
517284	1736.000000	2366.000000	2020.640625	94.305706
.04
538505	1893.000000	2489.000000	2103.535156	82.821997
.03
560353	1989.000000	2648.000000	2188.878906	85.107268
.03
560353	1989.000000	2648.000000	2188.878906	85.107268
.03
750234	2357.000000	3343.000000	2930.601562	194.399168
.06
872048	2785.000000	4867.000000	3406.437500	401.799473
.11
879673	2986.000000	3998.000000	3436.222656	135.385421
.03
963682	3087.000000	5098.000000	3764.382812	354.615937
.09
997920	2281.000000	7349.000000	3898.125000	461.347009
.11
1046770	3596.000000	5469.000000	4088.945312	284.685800
.06
1112976	3989.000000	7686.000000	4347.562500	254.226423
.05
1327104	3862.000000	6749.000000	5184.000000	468.322080
.09
1687669	5924.000000	7186.000000	6592.457031	204.379086
.03
1873351	6614.000000	8776.000000	7317.777344	261.643008
.03
2072772	7342.000000	10161.000000	8096.765625	305.602913
.03
2158683	7677.000000	10622.000000	8432.355469	301.601096
.03
2564291	9426.000000	11323.000000	10016.761719	281.364730
.02
3348820	9985.000000	16478.000000	13081.328125	1147.182354
.08
5772632	20494.000000	29802.000000	22549.343750	902.669842
.04
27407781	103531.000000	117263.000000	107061.644531
2093.466687	 .01
34950972	124450.000000	209336.000000	136527.234375
6884.022945	 .05
37495739	130774.000000	178224.000000	146467.730469
5398.963450	 .03
42047988	144620.000000	201436.000000	164249.953125
9260.798167	 .05
195211548	685275.000000	941261.000000	762545.109375
37221.965966	 .04
224484882	787278.000000	1017605.000000	876894.070312
34405.892708	 .03
596657226	2131931.000000	2887000.000000	2330692.289062
89687.773147	 .03

To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Wed, 26 Feb 2003 14:58:57 -0800
Lines: 66
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:
> Russ Allbery wrote:

>> To take an example, if someone posts to a local Stanford newsgroup and
>> also cc's [email-masked], that message will show up at a mail
>> to news gateway, since we gate all mail to postmaster into a newsgroup.
>> And that gateway can't trust the Newsgroups header in the message; it
>> will do completely the wrong thing.

> I don't see the problem; if the message hits an injector first and the
> gateway also tries to inject to the specified stanford.foo or whatever
> newsgroup as well as the gated postmaster group, surely the message-id
> will already be there and there won't be a duplicate or other problem.

You're missing a couple of things.

First, the message has already been posted to the group the original
poster was trying to post it to before it ever goes through the mail to
news gateway.  So if the gateway then can't post becuase of a duplicate
message ID, the message never shows up in the postmaster group; in effect,
the cc to postmaster doesn't work and the message is, from the perspective
of people reading through that gateway, lost.  That's obviously bad.

Second, even if you could work around that, the newsgroups for gatewayed
role address mail are not accessible to normal Stanford users for obvious
reasons, which means that they also don't allow crossposting with regular
Stanford groups.  So even if you could patch the message up to crosspost,
that doesn't work.

There have to be two messages posted, to two separate groups, and the mail
to news gateway has to ignore the Newsgroups header in the mailed copy
that it receives.  There's no other way to make this work.

> And if by quirk the mailed copy hits the gateway before the posted copy
> is injected, the same applies. So what's the problem? 

We actually want to see the message in both groups.  :)

> Is the postmaster gateway mangling message-ids or something?

Of course.  It has to.  In fact, most non-trivial mail to news gateways
have to rewrite message IDs.  It's a requirement for any gateway of a
mailing list that might be gatewayed by multiple people independently.

As for the other issue of IMAP servers re-serving news messages via IMAP,
I think we've pretty much talked around this issue.  I continue to believe
that there are points in the IMAP server code where it knows it's dealing
with news articles and *could* do a conversion, for pretty much any IMAP
server out there, although there may be some rare cases where it's tricky
to find that spot.  I also believe that this is rather annoying to have to
do.

I agree that if an existing IMAP gateway for news messages is presented
with messages with UTF-8 headers, it's not going to work correctly, and
that in some circumstances keeping such messages away from an IMAP gateway
that's not yet been modified may be tricky.

The rest I think is just disagreements of opinion over what level of
backward compatibility is mandatory versus desireable, and since I'm not
really arguing for solution (A) anyway, just trying to summarize it well
enough that people can evaluate it, I think I'll leave it at that.  I
don't think at this point we're adding new information that would be of
use to people who haven't yet made up their mind.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Wed, 26 Feb 2003 23:00:16 +0100
From: Chris Lilley <[email-masked]>
Reply-To: Chris Lilley <[email-masked]>
Organization: W3C
To: [email-masked]
CC: Joseph Reagle <[email-masked]>, [email-masked], [email-masked], Dan Kohn <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: draft-freed-mime-p4-00.txt
Sender: [email-masked]

On Wednesday, February 26, 2003, 8:42:20 PM, ned+xml-mime wrote:

> Joseph Reagle wrote:
>> Please note that that document provides guidance for registration in the
>> present (interim) context of RFC2048 -- not necessarily the policy when
>> draft-freed-mime-p4-00.txt would be operational. For instance, it's not
>> clear to me that we would still have to generate even "stub" ietf-drafts
>> once it is operational. I nope that sending emails to the various lists
>> with a reference to the W3C document will be sufficient.

nxmmc> Actually, it should be quite clear that this will no longer be
nxmmc> necessary.

Re-reading this, I agree that the statement is not entirely clear and
could use some editing to convey what Ned confirms is the intended
meaning.

"The normal IETF processes should be followed for all registrations in
the standards tree, with the posting of an internet-draft being a
necessary first step. "

That could clearly be misinterpreted that an I-D is necesasary in all
cases, even though it says should not SHOUL

"Proposed registrations in the standards tree by other standards
bodies should be communicated to the IESG (at [email-masked]). "

This is clear and leads me to believe that rewording the first
statement as

"The normal IETF processes should be followed for all registrations in
the IETF tree, with the posting of an internet-draft being a
necessary first step. "

would convey the intended meaning. It looks like a simple editorial
oversight in the generalization from 'IETF' to 'standards'.

-- 
 Chris                            mailto:[email-masked]

From: Joseph Reagle <[email-masked]>
Organization: W3C
To: Chris Lilley <[email-masked]>, [email-masked], [email-masked]
Subject: Re: draft-freed-mime-p4-00.txt
Date: Wed, 26 Feb 2003 16:56:54 -0500
Cc: [email-masked], [email-masked], Dan Kohn <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

On Wednesday 26 February 2003 16:41, Chris Lilley wrote:
> nxmmc> The IETF has this concept of recommending something in an RFC.
> nxmmc> I think that's close enough to an endorsement that I don't want
> nxmmc> to say the IETF never endorses anything.
>
> This is well put. Equally, W3C has a concept of recommending something
> and indeed our full standards track documents are called
> Recommendations to emphasise this.
>
> Perhaps Joseph is saying that an implied disclaimer of 'anything not
> in the IETF tree may be inadequate' does not add anything and is
> harmful, just as the W3C saying 'anything in the IETF tree is not a
> W3C Rec and might not be any good' would be harmful.

I don't disagree with this, but it was not quite my point. To me it seemed 
that the statement was not needed because nothing in the IANA registry need 
necessarily be seen as an endorsement. It's simpler for me to see the 
registry as a location for a non-colliding token with reference to a 
definition: a registration request or specification. If you want to know 
the endorsement associated with the definition, go read that. There you can 
see if it's a vendor registration request, an InfoRFC, Proposed Standard, 
Standard, Candidate REC, REC, obsoleted, deprecated, etc.

Date: Wed, 26 Feb 2003 22:41:58 +0100
From: Chris Lilley <[email-masked]>
Reply-To: Chris Lilley <[email-masked]>
Organization: W3C
To: [email-masked], [email-masked]
CC: Joseph Reagle <[email-masked]>, [email-masked], [email-masked], Dan Kohn <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: draft-freed-mime-p4-00.txt
Sender: [email-masked]

On Wednesday, February 26, 2003, 8:42:20 PM, ned+xml-mime wrote:

> Joseph Reagle wrote:
>> On Wednesday 26 February 2003 12:54, Dan Kohn wrote:
>> > This looks very good.  It still might be useful to provide a little more
>> > of the information from
>> > <http://www.w3.org/2002/06/registering-mediatype> to give hints to other
>> > standards bodies about the ordering of their draft, to publishing an
>> > I-D, [snip]

>> For instance, it's not clear to me that we would still have to
>> generate even "stub" ietf-drafts once it is operational. I nope
>> that sending emails to the various lists with a reference to the
>> W3C document will be sufficient.

nxmmc> Actually, it should be quite clear that this will no longer be
nxmmc> necessary.

Great.

>>    3.2.8 Publication Requirements
>>    Other than IETF registrations in the standards tree, the registration
>>    of a data type does not imply endorsement, approval, or
>>    recommendation by the IANA or the IETF or even certification that the
>>    specification is adequate.

>> jmr: Could this be simplified by saying that, "presense in the standards
>> tree does not necessary imply endorsement by IANA or IETF." One should not
>> look at the tree for endorsement at all, but, at the specification's
>> level/maturity (draft, proposed draft, standard, Recommendation, etc.)
>> within their organization.

nxmmc> The IETF has this concept of recommending something in an RFC.
nxmmc> I think that's close enough to an endorsement that I don't want
nxmmc> to say the IETF never endorses anything.

This is well put. Equally, W3C has a concept of recommending something
and indeed our full standards track documents are called
Recommendations to emphasise this.

Perhaps Joseph is saying that an implied disclaimer of 'anything not
in the IETF tree may be inadequate' does not add anything and is
harmful, just as the W3C saying 'anything in the IETF tree is not a
W3C Rec and might not be any good' would be harmful.

-- 
 Chris                            mailto:[email-masked]

Date: Wed, 26 Feb 2003 16:14:25 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], IMAP Interest List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], IMAP Interest List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: draft-ietf-usefor-article-09
Sender: [email-masked]

Charles Lindsey wrote:

> Here are the diffs from the latest snapshot on the landfield site. Please
> respond quickly if you spot any obvious howlers.

> + [Tentative paragraph to deal with IMAP]
> + 
> +    This requirement includes the transmissiom paths between posting
> +    agents, injecting agents, relaying agents, serving agents and reading
> +    agents, but it does NOT include paths traversed by Netnews articles
> +    that have been converted to Email (8.8.1.1).  In particular, this
> +    means that IMAP4 servers which provide access to Netnews SHOULD
> +    implement the extension described in section 4.4.3.

IMAP is not Email only as implied.  IMAP handles all 822/2822/MIME
messages, including news.  IMAP *is* on "the transmission path [...]
and reading agents".

> !    are potentially available.  Although it will usually be unnecessary
> !    to use language tagging within headers, the tagging facilities
> !    provided in [UNICODE 3.2] (code points U+E0000 through U+E007F) MAY
> !    be used for that purpose.

That's not consistent with RFC 2277's requirement that language
be preserved in the protocol; it can't be buried in non-standard
tags buried inside the charset -- it has to use RFC 3066 standard
language tags (in the subset of ASCII specified in 3066, not
translated to some whacko range and then further obfuscated
by a UTF, unless there is a clear and solid justification for
departing from 3066).  Moreover it cannot be made dependent on
charset support; preservation of the language information is
required even if the charset is not supported.

> + 4.4.3.  The NEWS-8BIT-HEADERS IMAP Extension
>   
> + [This section is highly tentative, and serves as a placeholder to
> + indicate that an IMAP extension will be needed in order to ensure
> + consistency with the present form of this draft. It shows the minimum
> + extension that seems to be necessary, and would require significant
> + further work for any final version.]
>   
> +    The current IMAP4 protocol [RFC 2060] forbids 8-bit character in

"current" should be elided; that aspect of the protocol has not changed.
"character" should be "characters".

> +    headers (so as to conform with the previous Netnews standard [RFC
> +    1036] amd with the current Email standards).
> + [That reference to RFC 2060 should be changed to refer to [RFC 2060bis]
> + if that has been accepted by the time this standard is published.]
> + 
> +    Implementations of IMAP4 conforming to this extension MUST
> + 
> +    1. In the case of Netnews messages only, accept 8-bit octets in
> +       headers (part specifiers HEADER or MIME, or the header portion of
> +       a MESSAGE/RFC822 part) and pass them on to the client unchanged in
> +       any FETCH response;

There is no way to distinguish "Netnews" messages from any other
messages.  This text contradicts RFC 2046 section 5.2.1 which requires
that header fields in message/rfc822 be 7-bit. This WG has no
authority to change the definition of MIME message/rfc822.

What about multipart composite media types? RFC 2046 section 5.1
requires MIME header fields in multipart types be in 7-bit
US-ASCII (possibly including 2047 encoded-words representing
8-bit text) only.  Why the disparate treatment?

Where is 8-bit header content negotiated by the sender *before* presenting
the data to an IMAP server (which might well be via placing an article
in a news spool area)?  Where is the requirement that the *sender*
fall back to RFC 1036/822/2822/MIME-compliant content in the event that
the IMAP sever does not support the extended syntax (i.e. in the default
case)?  Exactly how is this negotiation to take place, especially since
the IMAP server may be an autonomous process and may in fact be running
on a different machine accessing a common news spool via a file sharing
protocol such as NFS or SMB?

What about IMAP2?

> ***************
> *** 1616,1624 ****
>      The forms "UT" and "GMT" (indicating universal time) are to be
> !    regarded as obsolete synonyms for "+0000". They MUST be be accepted,
> !    and passed on unchanged, by all agents, but they MUST NOT be
> !    generated as part of new articles by posting and injecting agents.
> !    The date-time MUST be semantically valid as required by [RFC 2822].
> !    Although folding white space is permitted throughout the date-time
> !    syntax, it is RECOMMENDED that a single space be used in each place
> !    that FWS appears (whether it is required or optional).
>   
> --- 1670,1678 ----
>      The forms "UT" and "GMT" (indicating universal time) are to be
> !    regarded as obsolete synonyms for "+0000". They MUST be accepted, and
> !    passed on unchanged, by all agents, but they MUST NOT be generated as
> !    part of new articles by posting and injecting agents.  The date-time
> !    MUST be semantically valid as required by [RFC 2822].  Although
> !    folding white space is permitted throughout the date-time syntax, it
> !    is RECOMMENDED that a single space be used in each place that FWS
> !    appears (whether it is required or optional).

The change seems to be line breaks only.  What is the meaning of "RECOMMENDED"
and how does that differ from the standard terms MAY and SHOULD?  Why the
special case for UT and GMT being unchanged -- that implies that anything else,
including standard numeric offsets, may be changed?  What about CFWS? Is a
space "RECOMMENDED" (whatever that means) where 2822 specifies CFWS (optional
or mandatory) rather than FWS?

Date: Wed, 26 Feb 2003 12:14:51 -0800 (PST)
From: [email-masked]
Subject: Re: draft-freed-mime-p4-00.txt
To: Dan Kohn <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> http://www.ietf.org/internet-drafts/draft-freed-mime-p4-00.txt

> This looks very good.  It still might be useful to provide a little more
> of the information from
> <http://www.w3.org/2002/06/registering-mediatype> to give hints to other
> standards bodies about the ordering of their draft, to publishing an
> I-D, to last calls, to their publication, to informational RFC
> publication.  I believe this caused some confusion with the W3C until it
> was worked out, and informatively referencing the W3C process might even
> be useful.

I actually this it is sort of the other way around -- this procedure needs to
be updated to reflect the new draft. I don't see anything in the W3C procedure
that really belongs in registration procedure and which isn't already there.

> Also, RFC 3023 updated RFC 2048.  Could you please reference RFC 3023
> and explicitly mention the prohibition from RFC 3023 that, "media
> subtypes that do not represent XML MIME entities MUST NOT be allowed to
> register with a '+xml' suffix".

Good idea. Added.

> You might reference this whole paragraph from section 7 of RFC 3023:

>    [T]hose registering [XML] media types SHOULD use the '+xml'
>    convention unless they have a particularly compelling reason not
> to....
>    The registrar for the IETF tree will encourage new XML-
>    based media type registrations in the IETF tree to follow this
>    guideline.  Registrars for other trees SHOULD follow this convention
>    in order to ensure maximum interoperability of their XML-based
>    documents.  Similarly, media subtypes that do not represent XML MIME
>    entities MUST NOT be allowed to register with a '+xml' suffix.

Rather than reiterall the rules for XML media type registration I've
added a section that refers to RFC 3023.

				Ned

From: Joseph Reagle <[email-masked]>
Organization: W3C
To: [email-masked]
Subject: Re: draft-freed-mime-p4-00.txt
Date: Wed, 26 Feb 2003 15:02:41 -0500
Cc: [email-masked], [email-masked], Dan Kohn <[email-masked]>, [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

On Wednesday 26 February 2003 14:42, [email-masked] wrote:
> > Please note that that document provides guidance for registration in
> > the present (interim) context of RFC2048 -- not necessarily the policy
> > when draft-freed-mime-p4-00.txt would be operational. For instance,
> > it's not clear to me that we would still have to generate even "stub"
> > ietf-drafts once it is operational. I nope that sending emails to the
> > various lists with a reference to the W3C document will be sufficient.
>
> Actually, it should be quite clear that this will no longer be
> necessary.

It's clear that an RFC track is no longer REQUIRED for the IESG to direct 
IANA to create an entry for the standards tree. But it's not clear how one 
asks, absent the RFC, the IESG to take the action. Once the new policy is 
in place I will want to provide some very simple, walk-me-through-it, steps 
for the W3C of how one goes about getting the IESG to take the action.

From: John Cowan <[email-masked]>
Subject: Re: draft-freed-mime-p4-00.txt
To: [email-masked] (Joseph Reagle)
Date: Wed, 26 Feb 2003 14:46:31 -0500 (EST)
Cc: [email-masked], [email-masked], [email-masked] (Dan Kohn), [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

Joseph Reagle scripsit:

> I just noted and read this and had a few small comments:

An even smaller one.  In 3.1.5:

#    From time to time and as required by the community, the IANA may,
#   with the advice and consent of the IESG, create new top-level
#   registration trees.

It's "by and with the advice and consent", i.e. by the advice and with the
consent.  This phrase is very old; in the U.S. Constitution the President
makes treaties "b. and w. the a. and c. of the Senate", and in the
U.K. and the Dominions, laws are enacted "by the Queen's most Excellent
Majesty, b. and w. the A.  and C. of the Lords Spiritual and Temporal,
and Commons, in this present Parliament assembled", and have been since
at least 1689 (the English Bill of Rights) and probably much longer.

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.reutershealth.com
"If I have seen farther than others, it is because I was standing on
the shoulders of giants."
        --Isaac Newton

Date: Wed, 26 Feb 2003 11:42:20 -0800 (PST)
From: [email-masked]
Subject: Re: draft-freed-mime-p4-00.txt
To: Joseph Reagle <[email-masked]>
Cc: [email-masked], [email-masked], Dan Kohn <[email-masked]>, [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> On Wednesday 26 February 2003 12:54, Dan Kohn wrote:
> > This looks very good.  It still might be useful to provide a little more
> > of the information from
> > <http://www.w3.org/2002/06/registering-mediatype> to give hints to other
> > standards bodies about the ordering of their draft, to publishing an
> > I-D, to last calls, to their publication, to informational RFC
> > publication.  I believe this caused some confusion with the W3C until it
> > was worked out, and informatively referencing the W3C process might even
> > be useful.

> Please note that that document provides guidance for registration in the
> present (interim) context of RFC2048 -- not necessarily the policy when
> draft-freed-mime-p4-00.txt would be operational. For instance, it's not
> clear to me that we would still have to generate even "stub" ietf-drafts
> once it is operational. I nope that sending emails to the various lists
> with a reference to the W3C document will be sufficient.

Actually, it should be quite clear that this will no longer be
necessary.

> > http://www.ietf.org/internet-drafts/draft-freed-mime-p4-00.txt

> I just noted and read this and had a few small comments:

>    3.2.8 Publication Requirements
>    Other than IETF registrations in the standards tree, the registration
>    of a data type does not imply endorsement, approval, or
>    recommendation by the IANA or the IETF or even certification that the
>    specification is adequate.

> jmr: Could this be simplified by saying that, "presense in the standards
> tree does not necessary imply endorsement by IANA or IETF." One should not
> look at the tree for endorsement at all, but, at the specification's
> level/maturity (draft, proposed draft, standard, Recommendation, etc.)
> within their organization.

The IETF has this concept of recommending something in an RFC. I think that's
close enough to an endorsement that I don't want to say the IETF never
endorses anything.

>    3.2.8 ... The stanards
>    3.3.3 ... whateveer

>    4.1.3 Publication Requirements
>    All access types MUST be described by an RFC.

> To be clear, IETF is reserving transfer encodings and access types to its
> own process and are not forseen to be developed by other standards
> organizations? (I don't forsee it being otherwise, just wanted to be sure.)

Yes, that's indeed the case.

				Ned

From: Joseph Reagle <[email-masked]>
Organization: W3C
To: <[email-masked]>, <[email-masked]>, "Dan Kohn" <[email-masked]>
Subject: Re: draft-freed-mime-p4-00.txt
Date: Wed, 26 Feb 2003 14:19:39 -0500
Cc: <[email-masked]>, <[email-masked]>, [email-masked]
Sender: [email-masked]

On Wednesday 26 February 2003 12:54, Dan Kohn wrote:
> This looks very good.  It still might be useful to provide a little more
> of the information from
> <http://www.w3.org/2002/06/registering-mediatype> to give hints to other
> standards bodies about the ordering of their draft, to publishing an
> I-D, to last calls, to their publication, to informational RFC
> publication.  I believe this caused some confusion with the W3C until it
> was worked out, and informatively referencing the W3C process might even
> be useful.

Please note that that document provides guidance for registration in the 
present (interim) context of RFC2048 -- not necessarily the policy when 
draft-freed-mime-p4-00.txt would be operational. For instance, it's not 
clear to me that we would still have to generate even "stub" ietf-drafts 
once it is operational. I nope that sending emails to the various lists 
with a reference to the W3C document will be sufficient.

> http://www.ietf.org/internet-drafts/draft-freed-mime-p4-00.txt

I just noted and read this and had a few small comments:

   3.2.8 Publication Requirements
   Other than IETF registrations in the standards tree, the registration
   of a data type does not imply endorsement, approval, or
   recommendation by the IANA or the IETF or even certification that the
   specification is adequate. 

jmr: Could this be simplified by saying that, "presense in the standards 
tree does not necessary imply endorsement by IANA or IETF." One should not 
look at the tree for endorsement at all, but, at the specification's 
level/maturity (draft, proposed draft, standard, Recommendation, etc.) 
within their organization.

   3.2.8 ... The stanards 
   3.3.3 ... whateveer

   4.1.3 Publication Requirements
   All access types MUST be described by an RFC.  

To be clear, IETF is reserving transfer encodings and access types to its 
own process and are not forseen to be developed by other standards 
organizations? (I don't forsee it being otherwise, just wanted to be sure.)

Date: Wed, 26 Feb 2003 13:30:52 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Russ Allbery wrote:

> To take an example, if someone posts to a local Stanford newsgroup and
> also cc's [email-masked], that message will show up at a mail to
> news gateway, since we gate all mail to postmaster into a newsgroup.  And
> that gateway can't trust the Newsgroups header in the message; it will do
> completely the wrong thing.

I don't see the problem; if the message hits an injector first and
the gateway also tries to inject to the specified stanford.foo or
whatever newsgroup as well as the gated postmaster group, surely
the message-id will already be there and there won't be a duplicate
or other problem.  And if by quirk the mailed copy hits the gateway
before the posted copy is injected, the same applies. So what's the
problem?  Is the postmaster gateway mangling message-ids or something?

>>Incidentally, the issue of gateways that "have addresses corresponding
>>to newsgroups" is another area where the canonical name being in an
>>ASCII-compatible form is at least highly desirable, because address
>>local-parts are constrained to a subset of ASCII.
> 
> 
> Not really.  I mean, it would be useful to be able to use IDN or something
> here, but the mail address corresponding to a newsgroup can be anything
> you want.  There's no inherent linkage.

Well I did say "desirable", and I did point out that proponents of
punycode should look at what's being done for address local-parts.
I would add that based on past history, not only is it likely that
both IDN and i22d local-parts will be standardized well before any
Usefor draft specifying newsgroup name i18n becomes an RFC, but IDN
and i22d local-parts will likely be widely *implemented* in UAs before
that happens.  So some relevant support is likely to exist by the time
i22d newsgroup names become standardized, and unless the local-part
work is ignored or proves to be unsuitable for some valid reason, it
may well simply be a drop-in.

> So at no point does the IMAP server call open() on a disk file?  The
> articles just appear magically in memory?

What are you proposing; that the OS open() call do some transformation
when called by an IMAP server but not when called by any other process?
Or that the OS somehow magically determine whether a file is "news" or
"mail" and flag that somehow (e.g. odd-numbered file-descriptor for
"news", even otherwise)?  How would the OS know (bear in mind that any
disk file might be opened, including one saved to an ordinary file
by AT&T mail or BSD mailx or trn or Outlook Express (which handles
both "news" and "mail" messages and can save them to files (which
have a ".eml" suffix regardless or whether somebody thinks the message
is "news" or "mail")))?  Even if that were feasible, it would still be
a backwards incompatibility.

> Are you aware of IMAP servers that use the news spool storage format for
> IMAP mailboxes?

Mark has addressed this issue for the UW implementation; there is
on-line documentation available for other implementations if
for some reason his affirmative response is not sufficient. Or
one could read the O'Reilly "Managing IMAP" book, though the material
is rather dated (ISBN 0-596-00012-X, or go to
http://www.oreilly.com/catalog/mimap/).

> I don't believe that the contention that proposal (A) requires any
> impossible modifications is even remotely viable or defendable, and none
> of your posts have convinced me otherwise.  I find this contention very
> frustrating, since one of the things that has poisoned this debate from
> the beginning has been a complete unwillingness of people on both sides to
> actually read, understand, and analyze the proposals of the people on the
> other side without preformed conclusions.  As a result, we've had a ton of
> articles from both perspectives that *start* by saying "well, obviously
> your solution won't work."

I don't have a problem with defects in a proposal being pointed out,
whether it's one of my proposals or somebody else's.  And in the
final analysis, the burden should be on the proponents of a scheme to
address the relevant issues in sufficient detail that others can see
that it will work compatibly.  It ought not to be a case of "Here's
my half-baked scheme; tell me why it won't work", or "(waving hands)
some magic occurs here and we all live happily ever after".  If that
means that a proponent has to learn about IMAP or LMTP or stringprep
or punycode or whatever, that's the price of being a proponent --
welcome to reality.  Consider that the writer of "well, obviously
your solution won't work" may well be correct, and it may indeed be
obvious to somebody familiar with a relevant aspect of news -- that
means that the proponent of the scheme hasn't done his homework.

> People are going to have to change things to get non-ASCII newsgroups to
> work.  There's no way around this, and some point of the combined news and
> mail system is going to have to bear some pain.  The purpose of my summary
> and analysis was to try to present, as well as I could, how the pain would
> be distributed for the different proposed solutions.  After that exercise
> has been completed, we then move on to debating about who has to deal with
> the pain, which is the hard part.

The degree of pain associated with any backward incompatibiliy ('Y')
or fundamental incompatibility for any scheme seems to be such as to
preclude those schemes from serious consideration. At minimum, any
such scheme would require staged implementation, which means that
initially there would have to be a MUST NOT generate rule, and that
means no benefit for at least a couple of years.  Even before that,
sufficient detail on an extension negotiation and fallback method
at various places where there are incompatibilities needs to be worked
out in sufficient detail to ensure that the scheme is feasible, and
that's likely to take a considerable amount of time.  If i18n of
newsgroup name protocol elements is as urgent an issue as some think,
surely an approach that doesn't necessitate such delays is called
for, and that pretty much rules out A and B due to the 'Y's.  If
a scheme has only 'N's, 'C's and 'D's, there doesn't seem to be any
reason why it cannot begin to be implemented immediately, with
corresponding benefit being available immediately to early adopters.

Content-Class: urn:content-classes:message
Subject: draft-freed-mime-p4-00.txt
Date: Wed, 26 Feb 2003 09:54:17 -0800
Importance: normal
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

http://www.ietf.org/internet-drafts/draft-freed-mime-p4-00.txt

This looks very good.  It still might be useful to provide a little more
of the information from
<http://www.w3.org/2002/06/registering-mediatype> to give hints to other
standards bodies about the ordering of their draft, to publishing an
I-D, to last calls, to their publication, to informational RFC
publication.  I believe this caused some confusion with the W3C until it
was worked out, and informatively referencing the W3C process might even
be useful.

Also, RFC 3023 updated RFC 2048.  Could you please reference RFC 3023
and explicitly mention the prohibition from RFC 3023 that, "media
subtypes that do not represent XML MIME entities MUST NOT be allowed to
register with a '+xml' suffix".  

You might reference this whole paragraph from section 7 of RFC 3023:

   [T]hose registering [XML] media types SHOULD use the '+xml'
   convention unless they have a particularly compelling reason not
to....
   The registrar for the IETF tree will encourage new XML-
   based media type registrations in the IETF tree to follow this
   guideline.  Registrars for other trees SHOULD follow this convention
   in order to ensure maximum interoperability of their XML-based
   documents.  Similarly, media subtypes that do not represent XML MIME
   entities MUST NOT be allowed to register with a '+xml' suffix.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Wed, 26 Feb 2003 12:21:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: "D. J. Bernstein" <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

D. J. Bernstein wrote:

>    * We can use GoofyCode in various places.

Is the "Goofy" in "GoofyCode" supposed to be longhand for "q"
(as in "Goofymail") or for "djb" (as in "Goofydns")?  If
not please do try to use the correct term for whatever it is
you're unsucessfully attempting to refer. Otherwise your
rantings are likely to be dismissed as nothing more than
ranting.

Date: Wed, 26 Feb 2003 12:05:55 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Dan,

if you can't send a message without misrepresenting someone else,
then shut the hell up. 

Keith

On 25 Feb 2003 22:17:31 -0000
"D. J. Bernstein" <[email-masked]> wrote:

> 
> With existing software, users often encounter failures when they try to
> use non-ASCII characters. Example: a user types an alpha, and it shows
> up as an empty box, rather than an alpha, on another user's screen.
> 
> To make non-ASCII characters work, we're going to have to change some
> software. There are many strategies for doing this, and they have 
> different costs. For example:
> 
>    * We can support UTF-8 everywhere. This means changing quite a bit
>      of code in pine, sendmail, etc., and doing a massive upgrade.
> 
>    * We can use GoofyCode in various places. This means changing even
>      more code in pine, sendmail, etc.; changing code in many programs
>      that already work with UTF-8; and doing a far more massive upgrade.
> 
> A different strategy is to run in terror from the prospect of a massive
> upgrade---to pretend that non-ASCII support is merely an option. This is
> the strategy that the GoofyCode proponents are talking about when they
> claim that GoofyCode doesn't demand any upgrades. Of course, this
> strategy also fails to make non-ASCII characters work.
> 
> Notice that Crispin is complaining about, e.g., the ``implications in
> GUI programs and screen editors'' of the fact that ``one screen space''
> is sometimes encoded as more than ``one byte.'' This is true for UTF-8,
> GoofyCode, and all other Unicode encodings. Crispin simply doesn't want
> to support non-ASCII characters!
> 
> ---D. J. Bernstein, Associate Professor, Department of Mathematics,
> Statistics, and Computer Science, University of Illinois at Chicago

Date: Wed, 26 Feb 2003 12:02:50 -0500
From: Keith Moore <[email-masked]>
To: usenet-format <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: i18n headers straw poll
Sender: [email-masked]

the only thing that your example demonstrates is that there are SMTP
servers that don't insist that HELO be a valid domain name.

On Tue, 25 Feb 2003 11:13:44 -0500
Bruce Lilly <[email-masked]> wrote:

> 
> [ietf-822 added to copies due to the IDN and msg-id issues]
> [the charset of the body content of this message may be
>   incorrectly labeled]
> 
> J.B. Moreno wrote:
> > On 2/24/03 4:48 PM, Bruce Lilly at <[email-masked]> wrote:
> 
> >>>>  IMO this is a killer for Punnycode anywhere except the
> >>>> Newsgroups header -- you post a message that looks like junk to everyone
> >>>> (which is the current case for punycode) and it'll get flamed to death (well,
> >>>> by those that actually see it, punycode is likely to be caught by a lot of
> >>>> spam filters).
> > 
> >> 
> >> 
> >>Are you claiming that IDNs cannot or will not work? Or for that matter
> >>local-parts if/when they are standardized?
> > 
> > 
> > No, domain names have no pre-existing history that will be a problem
> 
> If you mean non-conforming use as in the case of Usenet newsgroup
> names, they do have a pre-existing history; the following header
> excerpt constitutes an existence proof of that (and does the same
> for local-part as well):
> 
> Received: from [61.72.6.29] (helo=�����ǻ��)
>          by mx02.mrf.mail.rcn.net with esmtp (Exim 3.35 #4)
>          id 18MVgZ-0000lw-00
>          for [email-masked]; Thu, 12 Dec 2002 10:55:39 -0500
> Received: from mail pickup service by �����ǻ�� with Microsoft SMTPSVC;
>           Fri, 14 Dec 2001 00:58:54 +0900
> From: "=?euc-kr?B?yKu6uLTjtOfA2g==?=" <[email-masked]>
> To: <[email-masked]>
> Subject: =?euc-kr?B?W8irurhdILzux8649CDHz7OqILi4teW8xbytILW3ILi5wMwgufa8vA==?=
>          =?euc-kr?B?v+R+fn5+fn4=?=
> Date: Fri, 14 Dec 2001 00:58:53 +0900
> MIME-Version: 1.0
> Content-Type: multipart/alternative;
>          boundary="----=_NextPart_000_45A0_01C1843A.8043A150"
> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
> Message-ID: <�����ǻ��CcSkpLiSV00001169@�����ǻ��>
> 
> Note the msg-id. Also the Received fields.  I have no idea
> what charset or language is supposed to be in use, since there's
> no indication (I could guess that it might be euc-kr as in
> the properly-tagged From and Subject fields, but that would
> be no more than a guess, and might be wrong[*]).  Nevertheless,
> there is clearly some non-ASCII stuff in both the domain
> (to the right of the '@' in the msg-id, which is identical
> to what was (illegally) supplied in the ESMTP EHLO command
> as documented in the top Received field, and to what was
> claimed as a domain name in the lower Received field) and
> the local-part (to the left of the '@' in the msg-id). Yes,
> such use is non-conforming (and there are plently of instances
> of non-conformance in this excerpt), but nevertheless such
> use is hereby documented.
> 
> Q.E.D.
> 
> Anyway, as far as non-ASCII newsgroup names are concerned,
> Charles has repeatedly stated that there are *no* such
> groups, so either there is "no pre-existing history that
> will be a problem" for newsgroup names or Charles is wrong.
> Either way punycode seems to be a viable approach to
> newsgroup name i18n -- the existence of one or more
> non-conforming newsgroup names or (spam) mail messages
> isn't an obstacle.  And I haven't seen it proposed in
> any other places but IDN, local-part, and newsgroup names
> (though if it works for IDN, it'll probably end up in
> Path and Xref, and I wouldn't be surprised if there were
> consideration for Distribution as well).
> 
> Notes:
> * As RFC 1557 (the reference for EUC-KR) notes, ESC $ ) C
>    must appear before any S0 characters, and as there's no
>    such sequence, the charset used for the alleged domain
>    name is certainly not valid EUC-KR. Moreover, EUC-KR
>    uses two octets per Korean character, and the alleged
>    local-part has an odd number of octets.  Whatever it
>    is, in the words of Shakespeare "it was as Greek to me"
>    (actually it's not Greek, since it clearly isn't
>    ISO-8859-7 either; it is equally not Korean and not Greek).
> 

Date: Wed, 26 Feb 2003 11:48:13 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: [email-masked], IETF-822 Mailing List <[email-masked]>
Subject: Re: Message-IDs (Was Re: Kohn draft (was RE: USEFOR: Current situation and next steps))
Sender: [email-masked]

[ietf-822. where the msg-id issue is being discussed, is copied]

Andrew Gierth wrote:
>>>>>>"Seth" == Seth Breidbart <[email-masked]> writes:
> 
> 
>  >> It it were entirely up to me, and I had a clean start, I think I
>  >> would remove quoted-strings and quoted-pairs entirely from
>  >> msg-ids. But I doubt that is practicable at this stage.
> 
>  Seth> Why?  Is there actually software that uses them?
> 
> I've certainly seen quoted strings in mail message-ids; in fact the
> original host for this list used them (in Resent-Message-ID).

The relevant question is does any existing software currently
have a need to generate them.  If they were ever used, relegating
the construct to obs- syntax still provides for reading old
messages, and if a construct was never used, it could be dropped
entirely.  Has quoted-pair within a no-fold-quote ever actually
been used in a msg-id?

Date: 25 Feb 2003 22:17:31 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

With existing software, users often encounter failures when they try to
use non-ASCII characters. Example: a user types an alpha, and it shows
up as an empty box, rather than an alpha, on another user's screen.

To make non-ASCII characters work, we're going to have to change some
software. There are many strategies for doing this, and they have 
different costs. For example:

   * We can support UTF-8 everywhere. This means changing quite a bit
     of code in pine, sendmail, etc., and doing a massive upgrade.

   * We can use GoofyCode in various places. This means changing even
     more code in pine, sendmail, etc.; changing code in many programs
     that already work with UTF-8; and doing a far more massive upgrade.

A different strategy is to run in terror from the prospect of a massive
upgrade---to pretend that non-ASCII support is merely an option. This is
the strategy that the GoofyCode proponents are talking about when they
claim that GoofyCode doesn't demand any upgrades. Of course, this
strategy also fails to make non-ASCII characters work.

Notice that Crispin is complaining about, e.g., the ``implications in
GUI programs and screen editors'' of the fact that ``one screen space''
is sometimes encoded as more than ``one byte.'' This is true for UTF-8,
GoofyCode, and all other Unicode encodings. Crispin simply doesn't want
to support non-ASCII characters!

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 25 Feb 2003 13:33:10 -0800 (Pacific Standard Time)
From: Mark Crispin <[email-masked]>
To: Russ Allbery <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Tue, 25 Feb 2003, Russ Allbery wrote:
> Are you aware of IMAP servers that use the news spool storage format for
> IMAP mailboxes?  I'm not, and I would have thought that would be an
> untenable storage format since it doesn't contain enough metadata in an
> indexed form to provide basic IMAP functionality without a huge speed hit.

I am.  UW imapd is quite capable of exporting a local news spool as the
#news namespace.  The news spool is, indeed, quite an untenable storage
format for IMAP due to the lack of useful metadata.  [But try to tell that
to the maildir groupies...]

> If the IMAP server doesn't use the same storage format for mailboxes and
> for news articles, then it is internally aware that the news messages are
> something different than the rest of its mailboxes.

Actually, it does and it isn't; but that's admittedly an implementation
issue.  However, I find myself quite unwilling to accept the premise that
news messages are something different than any other kind of messages.

> I don't believe that the contention that proposal (A) requires any
> impossible modifications is even remotely viable or defendable, and none
> of your posts have convinced me otherwise.

Let's try a different approach, which will help explain the problem
better.

It is relatively straightforward to undertake an extension to a protocol
which requires no changes to any other protocols or their implementations.

It is nearly impossible to undertake an extension that requires changes to
other protocols or their implementations.

I don't know if it has actually happened, but working groups have been
threatened with litigation if they do something that would cause an
existing standards-compliant implementation to become non-compliant (or
it does something non-compliant and they can't point a finger at someone
else's non-compliance as being the cause).  This is, regrettably, the
world that we live in today.

The path of least resistance is always to do something that requires no
changes to anything else.  It may be ugly, and it may be highly desirable
to change.  Nonetheless, if it can be ignored, it won't generate
opposition.

The choice is simple.  Go with UTF-8 in headers, and face substantial
opposition; or go with something that stays compliant with RFC 2822 (such
as the punycode method suggested by the Kohn draft) and not be opposed.

Considering all the other issues that Usefor needs to decide, this choice
should be a no-brainer.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 25 Feb 2003 13:05:52 -0800
Lines: 140
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> While it might be rare (I haven't analyzed *every* gateway), it does
> happen and it is an area where method A has issues that don't apply to
> the other methods.

I agree with this.

In the case where you have a generalized mail to news gateway that takes
a Newsgroups header in the message header or body and uses it verbatim,
method (A) would cause that mail to news gateway to not work correctly
with any article posted or crossposted to a non-ASCII newsgroup.

I think this is a pretty uncommon situation, but I suppose it could
potentially arise.

> And trn seems to be irrelevant in this case, since it isn't going to see
> the article until after the gateway has caused it to be injected,

This isn't necessarily true.

When you're trying to do a lot of mail to news gatewaying, and I've set up
quite a few of these for a lot of odd and interesting reasons, including
many cases where the user doesn't actually know that a given address is
going to a gateway, you run into messages that have been posted and mailed
by agents like trn that include a Newsgroups header.

To take an example, if someone posts to a local Stanford newsgroup and
also cc's [email-masked], that message will show up at a mail to
news gateway, since we gate all mail to postmaster into a newsgroup.  And
that gateway can't trust the Newsgroups header in the message; it will do
completely the wrong thing.

Some of these are special situations that don't arise all that often, but
mail to news gateways are used for a rather wide variety of applications.

> Incidentally, the issue of gateways that "have addresses corresponding
> to newsgroups" is another area where the canonical name being in an
> ASCII-compatible form is at least highly desirable, because address
> local-parts are constrained to a subset of ASCII.

Not really.  I mean, it would be useful to be able to use IDN or something
here, but the mail address corresponding to a newsgroup can be anything
you want.  There's no inherent linkage.

> The issue of gateway name compatibility was missing from your table;

That's because I don't consider it to be a relevant issue for our current
discussion.  It's too trivial to work around for any specific case.

> as column 15, it should be:

>     15
> -+---
> A| I (or Y if you prefer)
> B| N
> C| N
> D| N

Making the mail address be the punycode version of the newsgroup name
doesn't actually help the user one whit, since their mail client isn't (at
least right now) going to know how to form that address anyway.  It's just
a bunch of letters they have to memorize, like any other randomly chosen
address.  So this column is Y for all proposals; in every case, if you
want someone's mail client to automatically form a mail to news gateway
address from a desired newsgroup name, you need support for taking the
UTF-8 newsgroup name and transforming it into a mail address, which means
that the mail clients would have to be modified to have punycode support.

Again, though, I think this is a pretty small side issue in the overall
view of things, not worth worrying about to any large degree.  It's one of
those things that will sort itself out one way or another no matter which
strategy is picked and that gives no clear advantages to any strategy.

> What I use is not a "personal definition" it *is* "the standard
> definition" as used in scores of RFCs.  Either a proposed modification
> to a protocol would not result in illegal input to existing
> infrastructure, in which case the modification is backwards compatible,
> or there could be illegal input presented to that existing
> infrastructure, in which case the proposed modification is not backwards
> compatible.

Okay, I'm sorry.  I do see your point.  The issue of IMAP gateways in
particular are somewhat more problematic than any of the other
compatibility issues posed by solution (A) since under (A) there would now
be news messages flying around that were not compatible with the mail
message format, and one can construct some plausible scenarios where an
existing IMAP gateway would encounter those messages accidentally.

>> All messages in IMAP folders, including the news ones, have mail
>> syntax.  I think I've said this about four times now.

> You keep saying it but it doesn't work that way. Think of IMAP access to
> news as an abstraction where the articles in the news spool are
> presented to the user as messages in folders.

Yes, and the process performing that abstraction can change formats.  It's
certainly not even remotely pretty to do that work at that layer, but it
is entirely possible.

> No, at least in some implementations the articles aren't "obtained",
> they are exactly *the* same articles that reside in the news spool.

So at no point does the IMAP server call open() on a disk file?  The
articles just appear magically in memory?

Are you aware of IMAP servers that use the news spool storage format for
IMAP mailboxes?  I'm not, and I would have thought that would be an
untenable storage format since it doesn't contain enough metadata in an
indexed form to provide basic IMAP functionality without a huge speed hit.
If the IMAP server doesn't use the same storage format for mailboxes and
for news articles, then it is internally aware that the news messages are
something different than the rest of its mailboxes.

> As you have agreed, that is only *some* UAs, and others *would* require
> modification for that purpose with that method (indeed it may well be
> impossible for some).  And that ignores all of the other areas where A
> requires substantial modifications (some of which may also be
> impossible)

I don't believe that the contention that proposal (A) requires any
impossible modifications is even remotely viable or defendable, and none
of your posts have convinced me otherwise.  I find this contention very
frustrating, since one of the things that has poisoned this debate from
the beginning has been a complete unwillingness of people on both sides to
actually read, understand, and analyze the proposals of the people on the
other side without preformed conclusions.  As a result, we've had a ton of
articles from both perspectives that *start* by saying "well, obviously
your solution won't work."

People are going to have to change things to get non-ASCII newsgroups to
work.  There's no way around this, and some point of the combined news and
mail system is going to have to bear some pain.  The purpose of my summary
and analysis was to try to present, as well as I could, how the pain would
be distributed for the different proposed solutions.  After that exercise
has been completed, we then move on to debating about who has to deal with
the pain, which is the hard part.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 25 Feb 2003 11:12:06 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> On Mon, 24 Feb 2003 04:07:00 -0800 (PST), you wrote:

> >
> >> Juergen Helbing wrote:
> >
> >> > And because it seems necessary to re-specify yEnc entirely before it
> >> > is used as an official MIME encoding this might be the right time to
> >> > learn from the past and to do it better now.
> >
> >> Here's some procrastination (a gzip-8bit CTE) from my procrastination (a
> >> new Usefor draft) from my real work.  As always, comments are
> >> appreciated.
> >
> >How amusing. I just submitted an Internet Draft describing almost exactly the
> >same thing. I've attached a copy of what I wrote below.
> >So who wants to do  the merge?

> Amusing ?   I am really not amused.

I'm sorry to hear that. It was not my intention to cause offense.

> FYI:  Even a short notification from one of you both that such new
> encodings are under development would have saved me weeks of work for
> yEnc+MIME. :-(

Um, well, for me, this really was a spur of the moment thing. I had just
finished editing and submitting a revision to the media type registration
procedures and had a bit of time I wanted to spend on another document. I
really didn't want to start in on some big revision and I had read the
discussion of yEnc and thought, "why not?". So I took the beginnings of a draft
I started to write maybe 7-8 years ago on gzip encoding, replaced
deflate-binary with deflate-8bit using yEnc, converted the result from nroff to
XML, and sent it. I then checked mail and found that Dan had done something
similar but complimentary.

There was no plan here, and certainly no time to notify anyone. If you
had asked me if I was going to submit this draft a week ago, I would have
said "of course not".

> I wish you both the very best luck - and I'll lurk a little bit if you
> start actively realizing something.

Sounds good.

> As soon as one of you succeeded I will close that shitty  yenc.org
> site, bury that toy - and start promoting you.

> The good news for me:  I can do something else for the rest of the
> week - because the time was planned for yEnc+MIME. There is no need
> for it any longer.

I'm glad to hear there was some positive result.

				Ned

Date: Tue, 25 Feb 2003 12:43:20 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: usenet-format <[email-masked]>, [email-masked]
Organization: Bruce Lilly
To: usenet-format <[email-masked]>, [email-masked]
CC: Dan Kohn <[email-masked]>
Subject: Re: Tightening RFC 2822 msg-IDs
Sender: [email-masked]

Bruce Lilly wrote:

> msg-id          =       [CFWS] "<" id-left "@" id-right ">" [CFWS] ; 
> maximum length 250 octets excluding CFWS
> 
> id-left         =       dot-atom-text / no-fold-quote / obs-id-left
> 
> id-right        =       dot-atom-text / no-fold-literal / obs-id-right
> 
> no-fold-quote   =       DQUOTE *(qtext / printable-quoted-pair) DQUOTE
> 
> no-fold-literal =       "[" *(dtext / printable-quoted-pair) "]"
> 
> printable-quoted-pair = ("\" pqtext) / obs-qp
> 
> pqtext          =       %d33-61 /         ; Printable characters 
> excluding ">"
>                         %d63-126 /
>                         obs-text
> 
> dtext           =       DIGIT /
>                         "A" / "B" / "C" / "D" / "E" / "F" / ; 
> hexadecimal digits, case-insensitive
>                         "." /                               ; IPv4 
> separator
>                         ":"                                 ; IPv6 
> separator

There's still an issue there via qtext in the no-fold-quote,
since qtext allows NO_WS_CTLs and '>'. That can be taken
care of by yet another specific production that has printable
characters except backslash, DQUOTE, and '>', but it would
be much simpler to do away with the no-fold-quote entirely
if that's feasible.  For that matter, printable-quoted-pair
could probably be removed from no-fold-literal.

Date: Tue, 25 Feb 2003 12:36:04 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Seth Breidbart <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: message IDs (was Re: mail vs. news ???)
Sender: [email-masked]

Seth Breidbart wrote:
> As a general rule: I see no benefit to anybody in allowing long
> Message-Ids, weird characters via escape mechanisms in them, or
> anything similar.  To the extent that they are generated and used only
> by software, it doesn't matter what's there, and simpler is better.
> To the extent that they are used by people, simpler is better even
> more so.

The extreme case would be restricting the left side to
dot-atom-text only (for generation), getting rid of
no-fold-quote.  Does anybody have a problem with that?
It would certainly simplify the ABNF, and the semantics
would be a bit simpler to implement.

content-class: urn:content-classes:message
Subject: draft-gellens-format-bis-00.txt
Importance: normal
Date: Tue, 25 Feb 2003 09:21:20 -0800
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>
Sender: [email-masked]

This update looks great to me.  A few comments:

"with lines of no more than 997 characters (by convention usually no
more than 80)"

should be "with lines of no more than 998 characters (by convention
usually no more than 78)" to match Section 2.1.1 of RFC 2822.

"Numerous software products erroneously label this media type as
Text/Plain, resulting in much user discomfort."

Media type has a specific meaning in MIME that I don't think you want to
imply here.  I would suggest replacing "media type" with "kind of
textual content".

In section 5.1, I would replace both 80 and 79 with 78, as per RFC 2822.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

To: [email-masked]
Xref: clerew local.mime:2439
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Tightening RFC 2822 msg-IDs
Date: Tue, 25 Feb 2003 12:36:37 GMT
Lines: 20
Sender: [email-masked]

In <[email-masked]> Lawrence Greenfield <[email-masked]> writes:

>I believe having a Usenet format document that updates RFC 2822 on the
>standards track would be a very good thing, especially if all of the
>changes are like messageid: of the form of moving more syntax into the
>obsolete section and further restricting what is generated.

Agreed. Essentially there are a few never-used but trouble-causing
constructs that need to be pushed back into the 'obsolete syntax'.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 25 Feb 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: Usefor Mailing List <[email-masked]>
CC: IETF-822 Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>
Mail-Followup-To: IETF-822 Mailing List <[email-masked]>,IMAP mailing list <[email-masked]>,Usefor Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Charles Lindsey <[email-masked]> schrieb/wrote:
> There are a whole lot of things required by RFC 2822 that just do not work
> in news. Some examples:

> 1. You can't omit that SP after the ':' in a header.

Usually not done in mail, either.

> 2. The RFC 2822 syntax of the References-header breaks many threading user
> agents.

I don't see any additional problems to #3 and #3a.

> 3. Comments in unexpected places will break some existing software.

Add:
  3a. Additional whitespace, missing whitespace, unexpected folding.

All of these are known to break some mail software, too.

> 4. There are some features of the Message-ID [...] Of course, these
> quirky cases never arise in real life, but theoretically they could.

So there's nothing wrong if theses cases were outlawed for mail.

Claus
-- 
http://www.faerber.muc.de/

Date: 25 Feb 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: mail/news unification
Sender: [email-masked]

Charles Lindsey <[email-masked]> schrieb/wrote:
> In <[email-masked]> Dave Crocker <[email-masked]> writes:
> Yes, the superficial similarities are apparent, but there are real serious
> differences behind the scenes. You see this when you try to gateway
> between a newsgroup and a mailing list. The simple cases work fine, but
> just think what happens if an article is submitted to two mailing lists,
> or is posted to several newsgroups with more than one of them gatewayed to
> a mailing list. Life suddenly starts to get extremely complicated for the
> poor implementor(s).

Unfortunatly, noone seems to be interested in a solution. The
development of news and mail standards is done by different people.

Well, there are several problems:

On the mail side:

* Identifying what lists a message has been sent to (if you get a copy
  through another list or directly). Identifying which addresses are
  list addresses and which are ``personal'' or ``group'' addresses.
  Possible solution: List-To (and List-Followup-To) containing List-IDs.

* Retrieving metadata of a list you are not subscribed to (i.e. how to
  post, subscribe, where to find an archive, etc.)
  Possible solutions: URN lookup mechanism (does not work offline),
  including metadata in headers of all messages not sent through the
  list (possible spoofing problems).

* Identifying lists that actually are gatewayed newsgroups.
  Possible solution: URI (news:alt.foo.bar or nntp://news.example.com/
  example.foo.bar) as the List-ID.

On the news side:

* Identifying newsgroups that actually are gatewayed mailing lists.
  Possible solution: including the info in all messages posted (only
  moderated groups with a submission address that eventually forwards to
  the mailing list post address).

And, of course, the format differences.

Claus
-- 
http://www.faerber.muc.de/

Date: Tue, 25 Feb 2003 11:13:44 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: usenet-format <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
CC: usenet-format <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: i18n headers straw poll
Sender: [email-masked]

[ietf-822 added to copies due to the IDN and msg-id issues]
[the charset of the body content of this message may be
  incorrectly labeled]

J.B. Moreno wrote:
> On 2/24/03 4:48 PM, Bruce Lilly at <[email-masked]> wrote:

>>>>  IMO this is a killer for Punnycode anywhere except the
>>>> Newsgroups header -- you post a message that looks like junk to everyone
>>>> (which is the current case for punycode) and it'll get flamed to death (well,
>>>> by those that actually see it, punycode is likely to be caught by a lot of
>>>> spam filters).
> 
>> 
>> 
>>Are you claiming that IDNs cannot or will not work? Or for that matter
>>local-parts if/when they are standardized?
> 
> 
> No, domain names have no pre-existing history that will be a problem

If you mean non-conforming use as in the case of Usenet newsgroup
names, they do have a pre-existing history; the following header
excerpt constitutes an existence proof of that (and does the same
for local-part as well):

From: "=?euc-kr?B?yKu6uLTjtOfA2g==?=" <[email-masked]>
To: <[email-masked]>
Subject: =?euc-kr?B?W8irurhdILzux8649CDHz7OqILi4teW8xbytILW3ILi5wMwgufa8vA==?=
         =?euc-kr?B?v+R+fn5+fn4=?=
Date: Fri, 14 Dec 2001 00:58:53 +0900

Note the msg-id. Also the Received fields.  I have no idea
what charset or language is supposed to be in use, since there's
no indication (I could guess that it might be euc-kr as in
the properly-tagged From and Subject fields, but that would
be no more than a guess, and might be wrong[*]).  Nevertheless,
there is clearly some non-ASCII stuff in both the domain
(to the right of the '@' in the msg-id, which is identical
to what was (illegally) supplied in the ESMTP EHLO command
as documented in the top Received field, and to what was
claimed as a domain name in the lower Received field) and
the local-part (to the left of the '@' in the msg-id). Yes,
such use is non-conforming (and there are plently of instances
of non-conformance in this excerpt), but nevertheless such
use is hereby documented.

Q.E.D.

Anyway, as far as non-ASCII newsgroup names are concerned,
Charles has repeatedly stated that there are *no* such
groups, so either there is "no pre-existing history that
will be a problem" for newsgroup names or Charles is wrong.
Either way punycode seems to be a viable approach to
newsgroup name i18n -- the existence of one or more
non-conforming newsgroup names or (spam) mail messages
isn't an obstacle.  And I haven't seen it proposed in
any other places but IDN, local-part, and newsgroup names
(though if it works for IDN, it'll probably end up in
Path and Xref, and I wouldn't be surprised if there were
consideration for Distribution as well).

Notes:
* As RFC 1557 (the reference for EUC-KR) notes, ESC $ ) C
   must appear before any S0 characters, and as there's no
   such sequence, the charset used for the alleged domain
   name is certainly not valid EUC-KR. Moreover, EUC-KR
   uses two octets per Korean character, and the alleged
   local-part has an odd number of octets.  Whatever it
   is, in the words of Shakespeare "it was as Greek to me"
   (actually it's not Greek, since it clearly isn't
   ISO-8859-7 either; it is equally not Korean and not Greek).

Date: Tue, 25 Feb 2003 10:11:39 -0500 (EST)
From: Seth Breidbart <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: message IDs (was Re: mail vs. news ???)
Sender: [email-masked]

As a general rule: I see no benefit to anybody in allowing long
Message-Ids, weird characters via escape mechanisms in them, or
anything similar.  To the extent that they are generated and used only
by software, it doesn't matter what's there, and simpler is better.
To the extent that they are used by people, simpler is better even
more so.

Therefore, I support requirements that the Message-Id be restricted
for the ease and efficiency of software (implementation).

If anybody has a reason that a Message-Id should be able to contain a
Russian novel (in Russian), I'd be very interested in seeing it.

Headers (and body) meant to be read, understood, and entered by people
are, of course, a different matter.

Seth

Date: Tue, 25 Feb 2003 09:51:27 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Russ Allbery wrote:
> Bruce Lilly <[email-masked]> writes:

>>The scenario: User U wishes to crosspost to two newsgroups, one a
>>conventional newsgroup, and one an unmoderated extended newsgroup.  For
>>whatever reason (e.g. the conventional group is moderated),
> 
> 
> Moderation is not gatewaying in my document.

That was a mere example (N.B. the "e.g."). It could also be
that the user doesn't have access to an NNTP server that
permits posting to the newsgroups in question, or that the
user has access to email at the time, but not full TCP/IP
connectivity, or any of a number of reasons, i.e. "for
whatever reason".

>>he chooses to use the gateway to the conventional newsgroup.  His UA
>>inserts a Newsgroup field with the two names in punycode (since it's
>>being mailed) and sends it to the gateway. The gateway is somehow
>>supposed to magically convert the punycode names to UTF-8, but no
>>existing gateway does so.
> 
> 
> This isn't how most (nearly all) mail to news gateways work.  Mail to news
> gateways have addresses corresponding to newsgroups (whatever you want
> those addresses to be) and messages sent to those addresses are posted to
> those newsgroups.  Some gateways go a step farther and analyze the headers
> and support crossposting.
> 
> Gateways don't expect the e-mail message to have a Newsgroups header.  In
> fact, honoring the Newsgroups header in a mail message when processing the
> message at a mail to news gateway can cause a bunch of other problems.
> 
> So you're analyzing a situation that in practice is exceedingly rare and
> isn't the normal or recommended way of managing mail to news gateways,
> because it causes mishandling of messages sent, e.g., by a user agent like
> trn that doesn't use that meaning of Newsgroups in e-mail messages.

While it might be rare (I haven't analyzed *every* gateway), it does
happen and it is an area where method A has issues that don't apply
to the other methods.  And trn seems to be irrelevant in this case,
since it isn't going to see the article until after the gateway has
caused it to be injected, and at that point the Newsgroups field is
an accurate description of the newsgroups to which the article has
been posted.

Incidentally, the issue of gateways that "have addresses corresponding to
newsgroups" is another area where the canonical name being in an ASCII-
compatible form is at least highly desirable, because address local-parts
are constrained to a subset of ASCII.  Anyone who wishes to pursue
punycode for newsgroup names ought to at least take a look at the work in
progress on i18n for local-parts (as announced by Paul Hoffman to many
mailing lists, including Usefor, on Feb. 6).  The issue of gateway name
compatibility was missing from your table; as column 15, it should be:

    15
-+---
A| I (or Y if you prefer)
B| N
C| N
D| N

>>>>13 requires (for backwards compatibility) that the article format and
>>>>the format used in IMAP be tha same, which is not the case for A.
> 
> 
>>>No, it doesn't require that.  The IMAP server could be modified
>>
>>                                                          ========
>>In what way is that backwards compatible with the existing (i.e.
>>unmodified) installed base of IMAP servers?
> 
> 
> I don't find your personal definition of backwards compatible to be
> reasonable or useful.  I'm using the standard definition.

What I use is not a "personal definition" it *is* "the standard
definition" as used in scores of RFCs.  Either a proposed modification
to a protocol would not result in illegal input to existing infrastructure,
in which case the modification is backwards compatible, or there
could be illegal input presented to that existing infrastructure,
in which case the proposed modification is not backwards compatible.

> Under proposal (A), if that's what you're asking, existing IMAP servers
> will not work with new non-ASCII newsgroups without modification.

We're agreed on that -- "existing" and "without modification" refer
to the same thing.

>>That "process" is often reading directly from the spool (which under A
>>is in utf-8) or via NNTP (and an NNTP server has no way of determining
>>if its client is an IMAP server).  In any event, nothing now in that
>>process does such a conversion, so that is a backwards compatibility
>>issue which rates at least a 'Y'.

>>Likewise for a message moved from folder to folder (on the IMAP server;
>>please read a description of the IMAP protocol if you don't understand
>>that) by a client's user.
> 
> 
> All messages in IMAP folders, including the news ones, have mail syntax.
> I think I've said this about four times now.

You keep saying it but it doesn't work that way. Think of IMAP
access to news as an abstraction where the articles in the news
spool are presented to the user as messages in folders. *If*
and _only if_ the article format is the same as in the "mail"
format, that works. Otherwise, as in method A, it doesn't.

> There is a point at which the IMAP server obtains the messages from news.

No, at least in some implementations the articles aren't "obtained",
they are exactly *the* same articles that reside in the news spool.

>>What "gateway"?  IMAP servers get news articles from the news spool
>>directly, via NNTP, or via LMTP.
> 
> 
> That's a gateway, by the standard definition of a gateway.  A gateway is a
> system that talks two different protocols and moves data from one protocol
> to another.

Where the first method listed above is used, the articles aren't
moved, and there's no protocol involved, just files.  In some
cases there *might* be a file sharing protocol such as NFS or SMB
involved, but those are not news-specific, and you'll have a
hard time convincing file system software authors to embed
Newsgroup-header-field-transforming code into such software.

>>If anybody still wants to pursue A, they should present a comprehensive
>>plan as to how it can be achieved with existing (unmodified) IMAP
>>servers -- a plan that is agreeable to the IMAP experts.
> 
> 
> I don't think I'd make that strong of a requirement, but certainly one of
> the things that I hoped to accomplish by posting this summary and this
> discussion to the IMAP list is to make them aware of this discussion and
> the possible impacts on IMAP implementations.

I still say that A is incompatible with IMAP servers that directly
access the articles in the news spool.  And whatever plan ends up
in the draft is going to be reviewed as part of the last call, as
Dan has pointed out, so it's best to propose something viable in
the first place rather than to have it shot down and sent back to
be reworked.

>>How so -- B, C, and D all have more 'N's than A; A has more
>>incompatibilities ('Y' and 'I') than C or D and at least as many as B --
>>how can that be "least possible modifications"?
> 
> 
> See the note at the end of the original summary about how some news
> clients have been confirmed to just work with UTF-8 newsgroup names
> without any modification, including displaying them correctly.  This
> property is unique to (A) and not shared by any other proposal that I've
> seen.

As you have agreed, that is only *some* UAs, and others *would* require
modification for that purpose with that method (indeed it may well
be impossible for some).  And that ignores all of the other areas where
A requires substantial modifications (some of which may also be
impossible) where C and D require no modification or minimal
modification which does not entail an incompatibility (merely user
convenience).

From: [email-masked] (Juergen Helbing)
To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: gzip-8bit
Date: Tue, 25 Feb 2003 10:50:21 GMT
Sender: [email-masked]

On Mon, 24 Feb 2003 04:07:00 -0800 (PST), you wrote:

>
>> Juergen Helbing wrote:
>
>> > And because it seems necessary to re-specify yEnc entirely before it
>> > is used as an official MIME encoding this might be the right time to
>> > learn from the past and to do it better now.
>
>> Here's some procrastination (a gzip-8bit CTE) from my procrastination (a
>> new Usefor draft) from my real work.  As always, comments are
>> appreciated.
>
>How amusing. I just submitted an Internet Draft describing almost exactly the
>same thing. I've attached a copy of what I wrote below.
>So who wants to do  the merge?

Amusing ?   I am really not amused.

FYI:  Even a short notification from one of you both that such new
encodings are under development would have saved me weeks of work for
yEnc+MIME. :-(  

I wish you both the very best luck - and I'll lurk a little bit if you
start actively realizing something. 

As soon as one of you succeeded I will close that shitty  yenc.org
site, bury that toy - and start promoting you.

The good news for me:  I can do something else for the rest of the
week - because the time was planned for yEnc+MIME. There is no need
for it any longer.  

So long
-- 
Juergen

Date: Tue, 25 Feb 2003 00:56:37 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: IETF-822 Mailing List <[email-masked]>, Usefor Mailing List <[email-masked]>
Subject: Re: message IDs (was Re: mail vs. news ???)
Sender: [email-masked]

On Tue, 25 Feb 2003, Bruce Lilly wrote:
> > This contention utterly baffles me.  Please explain.
> 
> Mark can certainly speak for himself, but my interpretation of
> what he wrote is that NNTP could choose to use the bracketing
> as a way around the lack of general quoting...

Note omitted context:  the "contention" to which I was referring was
"doesn't belong in the header format specification", not "possible to
define this [bracketing] as a quoting rule".  I'm a bit skeptical of the
latter, but it's the former -- the assertion that a constraint crucial to
news interoperability has no place in the standard -- that I find bizarre.

> I wouldn't want to see the issue put in a separate document.
> It is part and parcel of the definition of a msg-id...
> And certainly the definition of msg-id -- including any length
> limit -- should apply uniformly with one possible exception...

I concur completely.  (And for those who don't read Usefor, it is *not*
common for Bruce and I to agree on something.)

> ...I would point out that none of RFCs 822, 850,
> or 1036 mentioned any sort of message-id upper length limit...
> ...So it appears that we are now considering
> introducing a formal limit lower than the inherent limit of
> 997 octets because those explicit cautions were disregarded.
> OK, let's not fix the blame, let's fix the problem.

There is no particular difficulty fixing the blame:  it's my fault.
I either introduced it or failed to fix it -- I no longer remember the
exact history of the particular bits of code -- when I overhauled the
dbz indexing package, extensively used in news software, circa 1990.

Unfortunately, I can think of no way to fix the problem.  Descendants of
my dbz release are still *very* widely used in news infrastructure, and
almost all still have this restriction.  Decreeing removal of the limit
will not accomplish it any time soon, and it's a net-wide interoperability
issue, not something that can be fixed (or automatically negotiated) one
host or one region at a time.  Even if it is thought desirable to make a
major effort to lift it in the long run -- and I'm unsure it's worth it --
the most we can do is "MUST accept, MUST NOT generate yet".

> > By analogy, it would seem a fine idea to move mail's 7-bit restriction
> > into a BCP.  Do you agree?  If not, why not?  Surely such an antiquated
> > and mail-specific limitation doesn't belong in "the header format 
> > specification".
> 
> Bad analogy...

Imperfect analogy, but my point was that what's sauce for the goose is
sauce for the gander:  news's message-ID interoperability issues are at
least as widespread in news -- tens of thousands of hosts, many different
software packages -- as mail's 7-bit interoperability issues are in mail,
and classing one as a vital part of the standard while dismissing the
other as merely "best practice" seems grossly inconsistent. 

                                                          Henry Spencer
                                                       [email-masked]

To: IETF-822 Mailing List <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>
Subject: Re: message IDs (was Re: mail vs. news ???)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 24 Feb 2003 21:25:17 -0800
Lines: 34
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> Mark can certainly speak for himself, but my interpretation of what he
> wrote is that NNTP could choose to use the bracketing as a way around
> the lack of general quoting for command parameters to work around the
> specific case of a backslash- quoted space inside a no-fold-quote in a
> msg-id.  Of course, then there would be \> to deal with...  It would
> have been nice if a quoting mechanism had been part of the NNTP
> protocol, but it wasn't and isn't so here we are, trying to work around
> that omission.

Adding quoting mechanisms is valuable when you have a real need to be able
to use the full range of available characters.

I see absolutely no useful purpose served by allowing whitespace in
message IDs, apart from being compatible with a portion of the mail
protocol that no one uses.  I would therefore go so far as to say that the
choice, viewed in isolation, is just good engineering in the NNTP
standard.  Adding an escape protocol when you don't need one and can
simply harmlessly limit the range of input instead means everyone has less
work they have to do, the protocol is less complex, and all software
implementing it is less complex.

> I'm not brushing off the issue, nor am I claiming that it is of no
> importance, so I feel no need to defend those particular straw men.  But
> I would point out that none of RFCs 822, 850, or 1036 mentioned any sort
> of message-id upper length limit;

Some limit falls naturally out of the NNTP protocol, yeah, rather than the
article format.  Although not the 250 one; that comes out of an old
software issue.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 25 Feb 2003 00:03:52 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: IETF-822 Mailing List <[email-masked]>, Usefor Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Henry Spencer <[email-masked]>
CC: IETF-822 Mailing List <[email-masked]>, Usefor Mailing List <[email-masked]>
Subject: Re: message IDs (was Re: mail vs. news ???)
Sender: [email-masked]

Henry Spencer wrote:
> On Mon, 24 Feb 2003, Mark Crispin wrote:
> 
>>I observe that wherever message-ids are used, they are inside a <> bracket
>>pair.  It is possible to define this as a quoting rule which would allow
>>embedded spaces.

>>Either way, this doesn't belong in the header format specification.  It
>>belongs in the NNTP specification, and a "best common practices" type
>>document.
> 
> 
> This contention utterly baffles me.  Please explain.

Mark can certainly speak for himself, but my interpretation of
what he wrote is that NNTP could choose to use the bracketing
as a way around the lack of general quoting for command
parameters to work around the specific case of a backslash-
quoted space inside a no-fold-quote in a msg-id.  Of course,
then there would be \> to deal with...  It would have been
nice if a quoting mechanism had been part of the NNTP protocol,
but it wasn't and isn't so here we are, trying to work around
that omission.

>>>The length limitation of 250
>>
>>I don't particularly worry about this, although as a practical header
>>format limitation it belongs an a "best common practices" type document
>>rather than the header format document itself.
> 
> 
> Sorry, this is flatly proposterous.  Again, it's a major interoperability
> matter, not just a "good practice" issue:  news systems will very probably
> be unable to exchange news successfully if it is violated. 

I wouldn't want to see the issue put in a separate document.
It is part and parcel of the definition of a msg-id, and we
have all suffered from the bit-of-specification-here,
another-bit-there that led to the consolidation that became
2822.  If somebody wants to put forward the proposition that
things were better with the message format specification
spread through a dozen or more documents, some contradicting
others...

And certainly the definition of msg-id -- including any length
limit -- should apply uniformly with one possible exception.
I.e. It should apply not only to Message-ID, but also to
Resent-Message-ID, References, Supersedes, and In-Reply-To
for obvious reasons. [and to MDN Original-Message-ID, but that's
not a header field]  The one possible exception is in Received
fields, where one option for the id is a msg-id, but that's
not reused in any of the other fields mentioned.  If there's
a pressing need to exempt Received, its syntax could use a
production which does not have a length restriction.

> The contention that "news and mail are the same" starts to sound odd when
> major interoperability issues for news are casually brushed off as being
> of no importance and not meriting mention in a standard, but detailed
> foibles of mail like the 7-bit restriction are expected to be considered
> sacrosanct.  One gets the distinct impression that "news and mail are the
> same" is being used to mean "so news's problems are of no importance".

I'm not brushing off the issue, nor am I claiming that it is
of no importance, so I feel no need to defend those particular
straw men.  But I would point out that none of RFCs 822, 850,
or 1036 mentioned any sort of message-id upper length limit;
indeed both 850 and 1036 caution against assumptions of a bound
on the length.  So it appears that we are now considering
introducing a formal limit lower than the inherent limit of
997 octets because those explicit cautions were disregarded.
OK, let's not fix the blame, let's fix the problem.

> By analogy, it would seem a fine idea to move mail's 7-bit restriction
> into a BCP.  Do you agree?  If not, why not?  Surely such an antiquated
> and mail-specific limitation doesn't belong in "the header format 
> specification".

Bad analogy. In the first place, the restriction didn't come
about because an explicit caution was disregarded; it is part
of the history and nature of the beast.  In the second
place, most of what appears in header fields consists of
protocol elements, and RFC 1958 is quite clear that those
should be in ASCII (actually it says case-independent ASCII).
In the third place, if your contention is that syntax issues
affecting interoperability belong in the format specification
(and I agree), then surely the restriction is appropriate
where it is.  In the fourth place, the restriction is not
mail-specific per se; it resulted from the origins of
messaging in telnet and ftp, and the protocol elements
themselves (e.g. domain names), which are widely used in
contexts other than mail, are limited to a subset of ASCII.

To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 24 Feb 2003 20:59:26 -0800
Lines: 217
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:
> Russ Allbery wrote:

>> I don't see how.

>> You mail the mail to news gateway address.  It adds a Newsgroups header
>> that contains the corresponding newsgroup.  Whether that header is in
>> UTF-8, punycode, or something else really makes little difference from
>> the perspective of the gateway.

> The scenario: User U wishes to crosspost to two newsgroups, one a
> conventional newsgroup, and one an unmoderated extended newsgroup.  For
> whatever reason (e.g. the conventional group is moderated),

Moderation is not gatewaying in my document.  I separated them for good
reasons, this being one of them.  I'm talking about mail to news
gatewaying that isn't moderation in the section on which you're
commenting.

> he chooses to use the gateway to the conventional newsgroup.  His UA
> inserts a Newsgroup field with the two names in punycode (since it's
> being mailed) and sends it to the gateway. The gateway is somehow
> supposed to magically convert the punycode names to UTF-8, but no
> existing gateway does so.

This isn't how most (nearly all) mail to news gateways work.  Mail to news
gateways have addresses corresponding to newsgroups (whatever you want
those addresses to be) and messages sent to those addresses are posted to
those newsgroups.  Some gateways go a step farther and analyze the headers
and support crossposting.

Gateways don't expect the e-mail message to have a Newsgroups header.  In
fact, honoring the Newsgroups header in a mail message when processing the
message at a mail to news gateway can cause a bunch of other problems.

So you're analyzing a situation that in practice is exceedingly rare and
isn't the normal or recommended way of managing mail to news gateways,
because it causes mishandling of messages sent, e.g., by a user agent like
trn that doesn't use that meaning of Newsgroups in e-mail messages.

>>> 13 requires (for backwards compatibility) that the article format and
>>> the format used in IMAP be tha same, which is not the case for A.

>> No, it doesn't require that.  The IMAP server could be modified
>                                                           ========
> In what way is that backwards compatible with the existing (i.e.
> unmodified) installed base of IMAP servers?

I don't find your personal definition of backwards compatible to be
reasonable or useful.  I'm using the standard definition.

Under proposal (A), if that's what you're asking, existing IMAP servers
will not work with new non-ASCII newsgroups without modification.

> That "process" is often reading directly from the spool (which under A
> is in utf-8) or via NNTP (and an NNTP server has no way of determining
> if its client is an IMAP server).  In any event, nothing now in that
> process does such a conversion, so that is a backwards compatibility
> issue which rates at least a 'Y'.

Funny, that's why I gave it a Y.

> In the case of LMTP transfer to the IMAP server, the server has no way
> to determine whether the message is "news" or "mail" and has no way to
> treat them differently.

The process doing LMTP does.  You clearly would not be able to feed new
non-ASCII newsgroups to an IMAP server using LMTP until you modified that
process to correctly encode the newsgroup names.

> Likewise for a message moved from folder to folder (on the IMAP server;
> please read a description of the IMAP protocol if you don't understand
> that) by a client's user.

All messages in IMAP folders, including the news ones, have mail syntax.
I think I've said this about four times now.

> You'll have to show how it's possible when there's no way for an IMAP
> server to differentiate (and therefore treat differently) "news" and
> "email".

No, I don't.

There is a point at which the IMAP server obtains the messages from news.
At that point, you do a conversion.  At every point that messages transit
from mail to news, in proposal (A), you have to do a conversion.  That's
the whole point of proposal (A).  Proposal (A) involves establishing a
separate news article format from the mail article format for messages
posted to non-ASCII groups (and those messages only) and then putting
gateways at every point of transition between news and mail.

You can certainly not like that proposal.  It makes gatewaying much more
of a pain for non-ASCII groups.  But it's quite definitely technically
feasible.

>> In the way that I phrased them in that summary, they can be.  There's a
>> converting gateway at (13) so that any articles present in the IMAP
>> system are mail-compatible.

> What "gateway"?  IMAP servers get news articles from the news spool
> directly, via NNTP, or via LMTP.

That's a gateway, by the standard definition of a gateway.  A gateway is a
system that talks two different protocols and moves data from one protocol
to another.

Currently, those gateways can be exceptionally easy (as long as they don't
have to deal with existing news messages with 8-bit header content).
Proposal (A) would make them significantly more complex for non-ASCII
groups.

>> Note, however, that it has no impact on existing ASCII groups.

> Do any of the schemes?

No.

> Is that relevant in differentiating them?

No.  It's relevant in addressing your claim that this proposal is not
backward compatible.  It would not be backward compatible if it affected
existing non-ASCII groups and the arrangements in place for them.

> I wouldn't call myself an IMAP implementor, but I'm familiar with the
> rudiments of the protocol and the messaging model, and I administer my
> own IMAP server.  As far as I can tell, there's no reason to convert a
> perfectly legal ASCII punycoded name; it would simply be left as the
> ASCII-compatible name.

The only reason that I could think of is that it may allow the IMAP client
to correctly display the name of the folder without having to understand
punycode; in other words, it might be a useful intermediate measure to
help out clients until they fully understand punycode.

>> Which is why I said "in order to properly display the names."  Until
>> the names are displayed properly, we haven't actually achieved our
>> goals; we've just succeeded in not breaking anything.  The whole point
>> of this is obviously to show the user real non-ASCII group names.

> Yes, but that boils down to the distinction between categories
> (not methods) C and D.  They were defined as:

> D means change is very desirable but not absolutely necessary, and C
> means change would be convenient but unmodified software is still fairly
> usable.

> I interpret "usable" to mean that the user can still post, follow-up,
> and read articles.

I didn't.  And it's my document.  :)

I interpreted "usable" as "the end user sees correctly displayed non-ASCII
newsgroup names," since otherwise we don't have usable non-ASCII group
names.  (I *didn't*, however, require that the news server administrator
see non-ASCII newsgroup names correctly displayed.  This was intentional.
The point of the protocol is the end users, not the server administrators;
some inconvenience for server administrators can be more easily tolerated
if the protocol is working for the end users.)

Another word besides usable may have been better.  "Non-ASCII functional"
perhaps.

>>>>  | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
>>>> -+------------------------------------------------------
>>>> A| D   C   N   N   D   Y   N   N   N   Y   N   D   Y   D
>>>> B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   C   D
>>>> C| D   C   C   C   D   N   N   N   N   N   N   N   C   D

>> I believe that your modifications are wrong, as noted above, although
>> some of them are debatable.  (13) and (14) for (A) are definitely
>> wrong.

> At best they are both 'Y',

(13) is a Y.  I have absolutely no dispute there.  (14) is not; that's the
whole point of making (13) a Y.  If (13) does the conversion, (14) doesn't
have to.  (Although (14) would then see punycode newsgroup names, so it's
still a D.)

>> (D) is also wrong.  (1) and (5) are a D.  If news readers are not
>> modified, the correct newsgroup names are not displayed, which means
>> that proper internationalization has not occurred.

> See my comments above re "usable" and the distinction.

Right.  That's just a wording difference.

> If anybody still wants to pursue A, they should present a comprehensive
> plan as to how it can be achieved with existing (unmodified) IMAP
> servers -- a plan that is agreeable to the IMAP experts.

I don't think I'd make that strong of a requirement, but certainly one of
the things that I hoped to accomplish by posting this summary and this
discussion to the IMAP list is to make them aware of this discussion and
the possible impacts on IMAP implementations.

>> I am not arguing that's the best approach; personally, I favor (C).
>> However, it is a possible approach and has some significant advantages
>> from the perspective of making the most possible work with the least
>> possible modifications to existing news software.

> How so -- B, C, and D all have more 'N's than A; A has more
> incompatibilities ('Y' and 'I') than C or D and at least as many as B --
> how can that be "least possible modifications"?

See the note at the end of the original summary about how some news
clients have been confirmed to just work with UTF-8 newsgroup names
without any modification, including displaying them correctly.  This
property is unique to (A) and not shared by any other proposal that I've
seen.

Mark Crispin doesn't feel that this is likely to be all that useful in
practice due to other GUI issues raised by Unicode support in general, and
he does have a valid point.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Mon, 24 Feb 2003 22:56:17 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Message-IDs (Was Re: Kohn draft (was RE: USEFOR: Current situation and next steps))
Sender: [email-masked]

Charles Lindsey wrote:

> But the real nasty only came to light well after RFC 2822 was published.
> Consider the following two cases:
> 
>     <"foo\bar"@baz.com>
>     <"foobar"@baz.com>
>     <[email-masked]>
> 
> Now read RFC 2822 VERY carefully. All those three are semantically the
> same. If ever you have to ask the question "is this msg-id the same one as
> that masg-id" you MUST answer Yes in the case of any pair out of those
> three.

Assuming the r in the last is a typo, yes, but now we've strayed
from syntax to semantics, which is a different kettle of fish.

> And how many backslashed characters did you find at all (redundant or
> not)? And how many quoted strings for that matter.

I didn't count them.  I suspect there were few if any.  Nor did
I check for backslash-quoted > (which apparently would cause
trouble for some news software).

To: [email-masked]
Xref: clerew local.mime:2435 local.usefor:16230
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Message-IDs (Was Re: Kohn draft (was RE: USEFOR: Current situation and next steps))
To: [email-masked], [email-masked]
Date: Mon, 24 Feb 2003 19:06:25 GMT
Lines: 76
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>> 2822 did specifically make the message-id syntax more restrictive than
>> 822 (822 allowed whitespace in quoted strings, 2822 only allows it in
>> quoted-pair (i.e. preceeded by \)

>If you look carefully, you'll see that not only must the whitespace
>be backslash-escaped, that in turn must be within a quoted-string.
>I.e.
>   <foo [email-masked]>    illegal
>   <foo\ [email-masked]>   illegal
>   <"foo bar"@baz.com>  illegal
>   <"foo\ bar"@baz.com> legal

Sure, but that is guaranteed to be misrecognized by any conforming NNTP
implementation.

There is a long history to this matter. In the DRUMS days, we spotted a
number of problems, which we persuaded them to fix. That is how the
special id-left and id-right syntax got into RFC 2822. But as time went
on, we kept finding more and more quirky cases. The "\ " problem was
spotted just about as DRUMS was going to press, but it was too late to
stop it.

But the real nasty only came to light well after RFC 2822 was published.
Consider the following two cases:

    <"foo\bar"@baz.com>
    <"foobar"@baz.com>
    <[email-masked]>

Now read RFC 2822 VERY carefully. All those three are semantically the
same. If ever you have to ask the question "is this msg-id the same one as
that masg-id" you MUST answer Yes in the case of any pair out of those
three. Of course, that is exactly the question that is asked on Usenet
millions of times every second and, as I said to Dan once, if that
equivalence were forced to be implemented on the existing network, it
would bring it to its knees within minutes.

However, I was assured that there did exist software in the mail world (in
threading readers, presumably, that did indeed follow that semantics to
the letter.

And of course, that situation is never going to arise in the Real World
(TM), so there is absolutely nothing to be gained by allowing it.

>I have a small collection (22937 messages). I checked the msg-ids in
>top-level header fields Message-ID, Received, In-Reply-To, References,
>Resent-Message-ID, Content-ID, and Supersedes. There were none that
>were simultaneously
>1. legal under RFC 2822 generate rules
>and
>2a. longer than 250 octets including angle brackets
>  or
>2b. had backslash-escaped space or control characters in a quoted string.

And how many backslashed characters did you find at all (redundant or
not)? And how many quoted strings for that matter.

Anyway, if you look in section 5.3 of the Usefor draft, you will find a
syntax that fixes it. Yes, it is ugly, but it is the minimal one that
solves the problem. I could live with a simpler one, even if it ruled out
a few more obscure, but otherwise harmless, cases.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 24 Feb 2003 21:36:00 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: IETF-822 Mailing List <[email-masked]>, Usefor Mailing List <[email-masked]>
Subject: message IDs (was Re: mail vs. news ???)
Sender: [email-masked]

On Mon, 24 Feb 2003, Mark Crispin wrote:
> I observe that wherever message-ids are used, they are inside a <> bracket
> pair.  It is possible to define this as a quoting rule which would allow
> embedded spaces.

RFCs 822, 2822, and 1036 (and all informal successors of 1036) all say
that the <> are *part* of the message ID.

> Either way, this doesn't belong in the header format specification.  It
> belongs in the NNTP specification, and a "best common practices" type
> document.

This contention utterly baffles me.  Please explain.  This is a major
interoperability issue for news:  it could cause articles generated on one
system to be unprocessable on another, or impossible to transmit over some
transmission paths.  Most certainly it belongs in the news format spec. 
If it does not belong in "the header format specification", then said
specification is not sufficient, by itself, as a news format standard.

> > The length limitation of 250
> 
> I don't particularly worry about this, although as a practical header
> format limitation it belongs an a "best common practices" type document
> rather than the header format document itself.

Sorry, this is flatly proposterous.  Again, it's a major interoperability
matter, not just a "good practice" issue:  news systems will very probably
be unable to exchange news successfully if it is violated. 

The contention that "news and mail are the same" starts to sound odd when
major interoperability issues for news are casually brushed off as being
of no importance and not meriting mention in a standard, but detailed
foibles of mail like the 7-bit restriction are expected to be considered
sacrosanct.  One gets the distinct impression that "news and mail are the
same" is being used to mean "so news's problems are of no importance".

> The fact that something is in a BCP doesn't mean that it can be ignored.
> BCPs can, and do, contain statements of the form "such-and-such will not
> work, so don't do it."

By analogy, it would seem a fine idea to move mail's 7-bit restriction
into a BCP.  Do you agree?  If not, why not?  Surely such an antiquated
and mail-specific limitation doesn't belong in "the header format 
specification".

What, exactly, is the distinction between standards-track documents and
BCPs, if vital interoperability issues are not suitable for the former?

                                                          Henry Spencer
                                                       [email-masked]

Date: Mon, 24 Feb 2003 17:45:01 -0800 (Pacific Standard Time)
From: Mark Crispin <[email-masked]>
To: Andrew Gierth <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Sat, 22 Feb 2003, Andrew Gierth wrote:
> Message-ids appear in NNTP as both command parameters and response
> parameters (neither of which can contain spaces, though this isn't
> spelled out for command parameters - it's just a consequence of the
> fact that spaces separate parameters and there is no quoting or
> escaping mechanism).

I observe that wherever message-ids are used, they are inside a <> bracket
pair.  It is possible to define this as a quoting rule which would allow
embedded spaces.

Or do NNTP implementations want to permit embedded "<" and ">" characters
inside a message-id?

Either way, this doesn't belong in the header format specification.  It
belongs in the NNTP specification, and a "best common practices" type
document.

> The length limitation of 250

I don't particularly worry about this, although as a practical header
format limitation it belongs an a "best common practices" type document
rather than the header format document itself.

The fact that something is in a BCP doesn't mean that it can be ignored.
BCPs can, and do, contain statements of the form "such-and-such will not
work, so don't do it."

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: Mon, 24 Feb 2003 20:16:49 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Russ Allbery wrote:
> Bruce Lilly <[email-masked]> writes:
> 
>>Russ Allbery wrote:
> 
> 
>>>One or another of these proposed options affect every single component
>>>of this system except for (11).  In the case of (11), none of these
>>>three proposals will affect any existing mail to news gateways.
>>>Existing mail to news gateways may not be able to handle new non-ASCII
>>>newsgroups
> 
> 
>>Certainly that affects those gateways!  A would require such a gateway
>>to affect a transformation which no existing gateway does.
> 
> 
> I don't see how.
> 
> You mail the mail to news gateway address.  It adds a Newsgroups header
> that contains the corresponding newsgroup.  Whether that header is in
> UTF-8, punycode, or something else really makes little difference from the
> perspective of the gateway.

The scenario: User U wishes to crosspost to two newsgroups, one
a conventional newsgroup, and one an unmoderated extended newsgroup.
For whatever reason (e.g. the conventional group is moderated), he
chooses to use the gateway to the conventional newsgroup.  His UA
inserts a Newsgroup field with the two names in punycode (since
it's being mailed) and sends it to the gateway. The gateway is
somehow supposed to magically convert the punycode names to UTF-8,
but no existing gateway does so.  I.e. message format A is not
backwards-compatible with existing gateways. What will happen
is that the gateway (seeing only friendly ASCII in the newsgroups
field) will blithely pass it on to an injection agent.  Now you
might say that it is then the injection agent's responsibility to
perform the magic conversion, but no existing injection agent does
that either.

>>13 requires (for backwards compatibility) that the article format and
>>the format used in IMAP be tha same, which is not the case for A.
> 
> 
> No, it doesn't require that.  The IMAP server could be modified
                                                          ========
In what way is that backwards compatible with the existing (i.e.
unmodified) installed base of IMAP servers?

>>As IMAP has been described in other mailing list postings, I won't go
>>into detail, but any scheme where the header field format differs
>>between "news" and "email" won't work with IMAP, and that rules out A.
> 
> 
> I think you should have read my summary more carefully.  The process that
> gets the news into the IMAP server would have to do a conversion.

That "process" is often reading directly from the spool (which under
A is in utf-8) or via NNTP (and an NNTP server has no way of determining
if its client is an IMAP server).  In any event, nothing now in that
process does such a conversion, so that is a backwards compatibility
issue which rates at least a 'Y'.  In the case of LMTP transfer to
the IMAP server, the server has no way to determine whether the message
is "news" or "mail" and has no way to treat them differently. Likewise
for a message moved from folder to folder (on the IMAP server; please
read a description of the IMAP protocol if you don't understand that)
by a client's user.

> I'm not saying that this is at all pretty for the IMAP server or at all
> ideal from an IMAP standpoint; I'm simply saying that it's technically
> possible and one can't rule out solution (A) by saying that it's not.

You'll have to show how it's possible when there's no way for an
IMAP server to differentiate (and therefore treat differently)
"news" and "email".

>>Those are all backwards incompatibilities.  Moreover the issues
>>affecting 13 incorrectly assume that "news" and "mail" can be
>>differentiated.
> 
> 
> In the way that I phrased them in that summary, they can be.  There's a
> converting gateway at (13) so that any articles present in the IMAP system
> are mail-compatible.

What "gateway"?  IMAP servers get news articles from the news spool
directly, via NNTP, or via LMTP.

> If the IMAP server talks to the NNTP server in real time, that conversion
> would have to be done in real time.

By whom? No existing IMAP server trats NNTP specially.  No existing
NNTP server performs conversions for some clients and not for
others.

> Note, however,
> that it has no impact on existing ASCII groups.

Do any of the schemes? Is that relevant in differentiating them?

>>And while *some* UAs might be usable unmodified *some* _will_ require
>>modification to support UTF-8 I/O;
> 
> 
> Right.
> 
> 
>>>IMAP servers (13) may want to recode the newsgroup names from punycode
>>>to UTF-7, but would not need to make any transformations to the
>>>articles themselves.
> 
> 
>>Why would anybody want to "recode [...] punycode to UTF-7"?
>>As noted above, there may be issues with IMAP<->NNTP communications.
> 
> 
> I honestly don't know; there was some discussion of IMAP using mUTF-7
> folder names, and I didn't know if that was an issue that would have to be
> taken into account.  You'd have to ask an IMAP implementor.

I wouldn't call myself an IMAP implementor, but I'm familiar with
the rudiments of the protocol and the messaging model, and I
administer my own IMAP server.  As far as I can tell, there's
no reason to convert a perfectly legal ASCII punycoded name; it
would simply be left as the ASCII-compatible name.  Especially
as in either case there is code to be written, and punycode will
have to be supported for IDN -- I doubt if anybody is fond of new
uses for UTF-7.

>>>(C) punycode everywhere
>>>=======================
> 
> 
>>>This proposal mandates modifications to the posting agents (1) and the
>>>news readers (5) in order to properly display the names.
> 
> 
>>Strictly speaking, that is not mandated; 1 and 5 can still be used by
>>users.
> 
> 
> Which is why I said "in order to properly display the names."
> 
> Until the names are displayed properly, we haven't actually achieved our
> goals; we've just succeeded in not breaking anything.  The whole point of
> this is obviously to show the user real non-ASCII group names.

Yes, but that boils down to the distinction between categories
(not methods) C and D.  They were defined as:

D means change is very desirable
but not absolutely necessary, and C means change would be convenient but
unmodified software is still fairly usable.

I interpret "usable" to mean that the user can still post,
follow-up, and read articles.  Yes, display of non-ASCII
names, where available, would be "convenient".  Maybe
we're trying to split too fine a hair between C and D;
at least neither is a show-stopper.

>>>No modifications are required to (2) or (4), the NNTP servers, although
>>>without modifications the server administrator would have to work with
>>>encoded group names.  It would provide a much better user interface if
>>>the administrative tools implemented punycode encoding and decoding for
>>>easier handling of non-ASCII newsgroup names.
> 
> 
>>Use of the canonical name may in fact be a benefit, e.g. in the case of
>>an administrator not familiar with Oriental, Cyrillic, Hebrew, Arabic,
>>Devanagari, etc. letterforms when dealing with some non-ASCII names.
>>Therefore the last statement above "It would provide a much better..."
>>is questionable.
> 
> 
> I was making the assumption that one could always work in either the
> encoded or decoded version of the name if that change were made.  I think
> that it's obvious that having the choice is clearly superior than having
> to use the encoded name and much preferrable in more mundane situations,
> such as European news admins dealing with groups whose names could be
> represented by ISO 8859-15.

Yes, choice is good; if the hypothetical administrative tools with
codecs can be overridden by their users (i.e. the administrators),
great.

>>>  | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
>>> -+------------------------------------------------------
>>> A| D   C   N   N   D   Y   N   N   N   Y   N   D   Y   D
>>> B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   C   D
>>> C| D   C   C   C   D   N   N   N   N   N   N   N   C   D
> 
> 
>>Based on the comments above, there are a few errors, I've had to
>>add a I (for incompatible) category, and I've added a fourth row:
> 
> 
>>  | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
>>-+------------------------------------------------------
>>A| Y   C   N   N   Y   Y   N   N   N   Y   Y   Y   I   I
>>B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   Y   D
>>C| C   C   C   C   C   N   N   N   N   N   N   N   C   C
>>D| C   C   C   C   C   N   N   N   N   N   N   N   C   C
> 
> 
> I believe that your modifications are wrong, as noted above, although some
> of them are debatable.  (13) and (14) for (A) are definitely wrong.

At best they are both 'Y', and you'll need to convince me that
that's feasible.

> (D) is also wrong.  (1) and (5) are a D.  If news readers are not
> modified, the correct newsgroup names are not displayed, which means that
> proper internationalization has not occurred.

See my comments above re "usable" and the distinction.  Anyway,
it's not worth quibbling; either 'C' or 'D' is far preferable
to the 'Y's of A and B.

> Similarly, this proposal
> changes either (13) or (14) to a D, since a client reading through IMAP
> would now have to look up somewhere what the *real* group name is, and
> that would have to be done either in the client or in the IMAP server.

Good point. I agree.

> I agree with your change to (13) for (B); I think my original summary was
> wrong.  (13) may actually be an N for (C).
> 
> (14) for (C) is, again, D.  The IMAP client would have to understand the
> punycode newsgroup name to display it correctly.

Whatever; as above, at least it certainly isn't a 'Y' or 'I'
as with method A.

>>A is completely incompatible with IMAP,
> 
> 
> Disagree, as mentioned above.

If anybody still wants to pursue A, they should present a
comprehensive plan as to how it can be achieved with existing
(unmodified) IMAP servers -- a plan that is agreeable to the
IMAP experts.  Frankly, though, in light of the lack of 'Y's
in methods C and D, it's probably not worth spending more
time on A or B.

>>Any Y or I is an incompatibility and rules out any chance of IESG
>>approval (barring a transition plan and protocol negotiation with
>>fallback where feasible).
> 
> 
> Don't be absurd.  Y says that the software requires modifications in order
> to work with non-ASCII groups.  Nothing about that inherently would rule
> out any chance of IESG approval; it doesn't break anything about use of
> that software with the existing ASCII groups.

It means that new articles present the existing infrastructure
with illegal content, and that is certainly a barrier unless,
as stated, there is a transition plan and protocol negotiation
to cover extended syntax, with fallback for backwards compatibility.
That would have to be worked out in fairly good detail for
either A or B (C and D have no such issues).

> There are no cases with any of these three proposals where existing news
> software completely breaks, as existing news software is already eight-bit
> clean (as noted as an assumption several times in my summary, an
> assumption that has already been discussed here at length).  To avoid
> problems with IMAP servers, simply don't expose the new groups via IMAP
> until the IMAP server has been modified to deal with them.

IMAP *is* "existing news software" (actually a protocol, but it's
implemented in software).  And utf-8 in article fields (method A)
breaks IMAP.  "[S]imply don't expose the new groups via IMAP"
doesn't work; if articles (even crosposted ones, though
the extended groups are not present) exist in the news spool,
they're available via IMAP (at least in some implementations
as administered in some places).

> I am not arguing that's the best approach; personally, I favor (C).
> However, it is a possible approach and has some significant advantages
> from the perspective of making the most possible work with the least
> possible modifications to existing news software.

How so -- B, C, and D all have more 'N's than A; A has more
incompatibilities ('Y' and 'I') than C or D and at least as
many as B -- how can that be "least possible modifications"?

Date: Mon, 24 Feb 2003 18:32:22 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: usenet-format <[email-masked]>, [email-masked]
Organization: Bruce Lilly
To: usenet-format <[email-masked]>, [email-masked]
Subject: Re: Tightening RFC 2822 msg-IDs
Sender: [email-masked]

I mention in passing that POP3 protocol's UIDL (RFC 1939, p. 12)
provides for a UIDL consisting of characters in the range
0x21-0x7e (printable US-ASCII graphic characters) with a
maximum length of 70.  I further note that some POP3 server
implementations use a message's msg-id as the UIDL, and
some of those implementations do so even for msg-ids longer
than 70 characters.

N.B. I am NOT proposing any change to 2822 because of that.

Date: Mon, 24 Feb 2003 22:35:24 +0000 (GMT)
From: Charles Lindsey <[email-masked]>
Reply-To: Charles Lindsey <[email-masked]>
Subject: Re: yEnc (was RE: mail vs. news ???)
To: [email-masked]
Cc: [email-masked]
Content-MD5: CTJvyd2Oz4dDekyFRHWmmA==
Sender: [email-masked]

	On Mon, 24 Feb 2003 09:39:04 GMT
	[email-masked] (Juergen Helbing) said...

> On Sun, 23 Feb 2003 16:36:07 GMT, you wrote:
> 
> >Yes, I have been round this before. It is clear that, even if the Yenc
> >people were to use
> >    Content-Transfer-Encoding: x-yenc
> >which they could start doing tomorrow, things would be much simpler.
> 
> For whom would be things "much simpler" ?

People writing news user agents, who would not need to
decode the Subject line, but rather would just look at the
Content-Transfer-Encoding line, which they have to look at anyway just
for this purpose. And everybody else who came across these articles and
who currently fails to understand the strange things in the Subject
line, but who would understand the CTE header and realise that there was
a new encoding available that they should get hold of.

And also the users, who could just write subject headers pertaining to
what their binary was about.

> 
> >They seem
> >convinced that the IETF is totally against all change, and that they would
> >be heavily sat upon, and that the whole process would be far too
> >cumbersome and time consuming, 
> 
> If I'm looking at the latest discussions about USEFOR here then I
> really dont want to imagine what could happen to a yEnc RFC.

The discussions are far worse on the Usefor list :-( .
> 
> And there is one thing you should never forget:
> 
> 
> So please apologize that I dont want to be one who is messing up the
> worldwide eMail system 'just to save 25% of bandwidth' - which is
> negligable for Mail - but important for Binary Newsgroups (where yEnc
> belongs to).

I doubt anyone will bother to implement Yenc in pure mail newsreaders.
Indeed, your standard should say that it SHOULD be implemented in
News and MAY be implemented in mail. And that gateways to mail should
downsize it to base 64.

But, as I already said, if you call it x-yenc, you don't even need a
standard or permission from the IETF or anything.
> 
> >and "hey it works fine already, we don't need no stinkin' IETF". 
> 
> Yes, yEnc is working pretty good. It is saving Gigs of bandwidth every
> day for USENET. 
> 
> FYI: Usenet does not need IETF. The neticens there have invented many
> nice things on Usenet even without the admins and us - the developers.
> 
> These people dont _care_ about us (including myself).
> 
I know the feeling :-( .
> 
> >All very sad, but that's how some people regard
> >the IETF. So we are stuck with indicating the presence of yEnc by a flag
> >in the Subject header. It is a mess.
> 
> Update: More and more posts appear on Usenet even without the
> mandatory keyword "yEnc" in the subject line. It is necessary to scan
> the body.

Which is why you should be using the header that was designed for the
purpose.
> 
> 
> 
> Usenet is not eMail.
> And Binary Usenet is not Usenet.

But they all have to coexist.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: usenet-format <[email-masked]>, [email-masked]
Subject: Re: Tightening RFC 2822 msg-IDs
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 24 Feb 2003 15:05:40 -0800
Lines: 18
Sender: [email-masked]

Henry Spencer <[email-masked]> writes:

> That said, it would certainly be good to at least *reduce* the format
> differences, if it can be done without serious installed-base impact on
> either side.  We news folks could certainly supply a wishlist of format
> restrictions.  Tightening up message-ID syntax is a big one, given how
> crucial message IDs are to news handling, processing, display, etc., and
> given that restrictions on them are already embedded in both our
> software and our protocols.

Charles Lindsey already did a lot of excellent work in that regard before
the final publication of RFC 2822, and RFC 2822 already incorporates a lot
of the additional restrictions that news needed.  There really aren't that
many left (it's a lovely change of pace to be able to easily summarize the
remaining header differences in a fairly short e-mail message).

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 24 Feb 2003 15:01:25 -0800
Lines: 168
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:
> Russ Allbery wrote:

>> One or another of these proposed options affect every single component
>> of this system except for (11).  In the case of (11), none of these
>> three proposals will affect any existing mail to news gateways.
>> Existing mail to news gateways may not be able to handle new non-ASCII
>> newsgroups

> Certainly that affects those gateways!  A would require such a gateway
> to affect a transformation which no existing gateway does.

I don't see how.

You mail the mail to news gateway address.  It adds a Newsgroups header
that contains the corresponding newsgroup.  Whether that header is in
UTF-8, punycode, or something else really makes little difference from the
perspective of the gateway.

> 13 requires (for backwards compatibility) that the article format and
> the format used in IMAP be tha same, which is not the case for A.

No, it doesn't require that.  The IMAP server could be modified to convert
formats when dealing with non-ASCII groups.  A modification is required
here for all IMAP servers that want to deal with non-ASCII group names,
though.

> As IMAP has been described in other mailing list postings, I won't go
> into detail, but any scheme where the header field format differs
> between "news" and "email" won't work with IMAP, and that rules out A.

I think you should have read my summary more carefully.  The process that
gets the news into the IMAP server would have to do a conversion.

I'm not saying that this is at all pretty for the IMAP server or at all
ideal from an IMAP standpoint; I'm simply saying that it's technically
possible and one can't rule out solution (A) by saying that it's not.

> Those are all backwards incompatibilities.  Moreover the issues
> affecting 13 incorrectly assume that "news" and "mail" can be
> differentiated.

In the way that I phrased them in that summary, they can be.  There's a
converting gateway at (13) so that any articles present in the IMAP system
are mail-compatible.

If the IMAP server talks to the NNTP server in real time, that conversion
would have to be done in real time.  This could potentially be extremely
obnoxious.  It's a significant negative for solution (A).  Note, however,
that it has no impact on existing ASCII groups.

> And while *some* UAs might be usable unmodified *some* _will_ require
> modification to support UTF-8 I/O;

Right.

>> IMAP servers (13) may want to recode the newsgroup names from punycode
>> to UTF-7, but would not need to make any transformations to the
>> articles themselves.

> Why would anybody want to "recode [...] punycode to UTF-7"?
> As noted above, there may be issues with IMAP<->NNTP communications.

I honestly don't know; there was some discussion of IMAP using mUTF-7
folder names, and I didn't know if that was an issue that would have to be
taken into account.  You'd have to ask an IMAP implementor.

>> (C) punycode everywhere
>> =======================

>> This proposal mandates modifications to the posting agents (1) and the
>> news readers (5) in order to properly display the names.

> Strictly speaking, that is not mandated; 1 and 5 can still be used by
> users.

Which is why I said "in order to properly display the names."

Until the names are displayed properly, we haven't actually achieved our
goals; we've just succeeded in not breaking anything.  The whole point of
this is obviously to show the user real non-ASCII group names.

>> No modifications are required to (2) or (4), the NNTP servers, although
>> without modifications the server administrator would have to work with
>> encoded group names.  It would provide a much better user interface if
>> the administrative tools implemented punycode encoding and decoding for
>> easier handling of non-ASCII newsgroup names.

> Use of the canonical name may in fact be a benefit, e.g. in the case of
> an administrator not familiar with Oriental, Cyrillic, Hebrew, Arabic,
> Devanagari, etc. letterforms when dealing with some non-ASCII names.
> Therefore the last statement above "It would provide a much better..."
> is questionable.

I was making the assumption that one could always work in either the
encoded or decoded version of the name if that change were made.  I think
that it's obvious that having the choice is clearly superior than having
to use the encoded name and much preferrable in more mundane situations,
such as European news admins dealing with groups whose names could be
represented by ISO 8859-15.

>>   | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
>>  -+------------------------------------------------------
>>  A| D   C   N   N   D   Y   N   N   N   Y   N   D   Y   D
>>  B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   C   D
>>  C| D   C   C   C   D   N   N   N   N   N   N   N   C   D

> Based on the comments above, there are a few errors, I've had to
> add a I (for incompatible) category, and I've added a fourth row:

>   | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
> -+------------------------------------------------------
> A| Y   C   N   N   Y   Y   N   N   N   Y   Y   Y   I   I
> B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   Y   D
> C| C   C   C   C   C   N   N   N   N   N   N   N   C   C
> D| C   C   C   C   C   N   N   N   N   N   N   N   C   C

I believe that your modifications are wrong, as noted above, although some
of them are debatable.  (13) and (14) for (A) are definitely wrong.

(D) is also wrong.  (1) and (5) are a D.  If news readers are not
modified, the correct newsgroup names are not displayed, which means that
proper internationalization has not occurred.  Similarly, this proposal
changes either (13) or (14) to a D, since a client reading through IMAP
would now have to look up somewhere what the *real* group name is, and
that would have to be done either in the client or in the IMAP server.

I agree with your change to (13) for (B); I think my original summary was
wrong.  (13) may actually be an N for (C).

(14) for (C) is, again, D.  The IMAP client would have to understand the
punycode newsgroup name to display it correctly.

>> The above summary I believe correctly indicates that proposal (B)
>> requires the most changes to be made to the news system itself.

> 7-9 are irrelevant (identical for all rows),

They are for these proposals.  I included the mail transport system
because other proposals would have required it to change.

> A is completely incompatible with IMAP,

Disagree, as mentioned above.

> Any Y or I is an incompatibility and rules out any chance of IESG
> approval (barring a transition plan and protocol negotiation with
> fallback where feasible).

Don't be absurd.  Y says that the software requires modifications in order
to work with non-ASCII groups.  Nothing about that inherently would rule
out any chance of IESG approval; it doesn't break anything about use of
that software with the existing ASCII groups.

There are no cases with any of these three proposals where existing news
software completely breaks, as existing news software is already eight-bit
clean (as noted as an assumption several times in my summary, an
assumption that has already been discussed here at length).  To avoid
problems with IMAP servers, simply don't expose the new groups via IMAP
until the IMAP server has been modified to deal with them.

I am not arguing that's the best approach; personally, I favor (C).
However, it is a possible approach and has some significant advantages
from the perspective of making the most possible work with the least
possible modifications to existing news software.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Mon, 24 Feb 2003 13:10:26 -0800
To: [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Format=Flowed/RFC 2646 Bis
Sender: [email-masked]

A draft which is intended to replace RFC 2646 has been sent in; 
because of the crush of last-minute submissions, there may be a delay 
befpre the announcement appears.  During this time it is available at 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-format-bis-00.txt>.

This draft specifies the DelSp parameter to solve the non-Western 
language problem.  Despite the kludge level, this appeared (during 
discussions last year) to be the least problematic solution.  I 
believe it's already been implemented in Apple's Mail application and 
Eudora.

I've also incorporated all the corrections/bugs that were sent to me 
since RFC 2646 was published.  See Appendix A for a list of changes 
from RFC 2646.

This message is BCCd to those who participated in the discussions 
which led to DelSp, and also to those who sent in corrections or bugs.

Abstract

     This specification establishes two parameters to be used with the
     Text/Plain media type, and, in the presence of these parameters, the
     use of trailing whitespace to indicate flowed lines.  This results
     in an encoding which appears as normal Text/Plain in older
     implementations, since it is in fact normal Text/Plain, yet provides
     for superior wrapping, flowing, and quoting.

     This standard supersedes the one specified in RFC 2646, "The
     Text/Plain Format Parameter", and adds the DelSp parameter to
     accommodate languages/coded character sets in which ASCII spaces are
     not used or appear rarely.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
I've been good, and I've been bad, but common sense I never had.
                                       --New Order, "Shellshock"

Date: Mon, 24 Feb 2003 16:13:17 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked], [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Russ Allbery wrote:
> This is a summary of what I believe to be the options for handling Unicode
> newsgroup names in an IETF standard.  So far, I see three separate viable
> options for handling encoded newsgroup names through the entire protocol.
> Those three options are:
> 
>   (A) UTF-8 in articles and NNTP, punycode in e-mail and IMAP
>   (B) punycode in e-mail, IMAP, and articles, UTF-8 in NNTP
>   (C) punycode everywhere

There is another option, viz. use any not-already-in-use name (i.e.
protocol element) and put the i18n in the description associated
with that name (said desciption could be charset- and language-tagged).

But first, let's consider some implications for the 3 options presented
which were not covered:

A and B each imply that an IMAP SEARCH and an NNTP wildmat will differ,
which is a pain for client authors.  C of course doesn't have that
problem, nor does the fourth method described briefly above.

> I am also making the assumption that a standard requiring e-mail or IMAP
> to handle unencoded UTF-8 in message headers and in newsgroup names is not
> a viable option, due to strong oppposition from the e-mail community.

Whatever "the email community" is supposed to mean, such a proposal
simply won't pass the IESG process because of the backwards
incompatibility.

>   (1) A newsreader posting via NNTP.
>   (2) The NNTP server accepting posts from a client.
>   (3) The NNTP transit server relaying posts to other servers.
>   (4) The NNTP server providing messages to a client.
>   (5) A newsreader reading via NNTP.
>   (6) The NNTP server relaying a message posted to a moderated group.
>   (7) The local mail system of the NNTP server.
>   (8) The mail system of the moderation relay site.
>   (9) The local mail system of the moderator.
>  (10) The software used by the moderator of a newsgroup.
>  (11) A mail to news gateway.
>  (12) A news to mail gateway.
>  (13) An IMAP server serving Usenet messages to a client.
>  (14) An IMAP client reading Usenet messages from an IMAP server.
> 
> One or another of these proposed options affect every single component of
> this system except for (11).  In the case of (11), none of these three
> proposals will affect any existing mail to news gateways.  Existing mail
> to news gateways may not be able to handle new non-ASCII newsgroups

Certainly that affects those gateways!  A would require such a
gateway to affect a transformation which no existing gateway
does. That is a backwards compatibility issue. B, C, and the
fourth method do not affect 11.

> In the case of (14), I am making the assumption that changing the IMAP
> protocol is not an option.  This means that messages served to (14) will
> not contain unencoded UTF-8 in the headers, and newsgroup names in IMAP
> will not have unencoded UTF-8 names.  All of the work (if any) required to
> make the articles compatible with an IMAP environment would have to be
> born by (13).  In all of these cases, it will therefore be desirable for
> the IMAP client to be modified to understand punycode and display the
> newsgroup names correctly.

13 requires (for backwards compatibility) that the article format
and the format used in IMAP be tha same, which is not the case for
A.  B may also be a problem for some IMAP implementations (any that
communicate via NNTP as opposed to taking articles from a spool).

> (A) UTF-8 in articles and NNTP, punycode in e-mail and IMAP
> ===========================================================
> 
> This is Andrew's original proposal.  The canonical name of the newsgroup
> would be in UTF-8 without further encoding.  The Usenet article format
> would be defined to carry UTF-8 newsgroup names without further encoding
> in those headers that contain newsgroup names (Newsgroups, Followup-To,
> Control, and Xref).

As IMAP has been described in other mailing list postings, I won't
go into detail, but any scheme where the header field format differs
between "news" and "email" won't work with IMAP, and that rules out
A.  This is a fundamental incompatibility.

> At every point where a Usenet article must be conveyed via e-mail,
> specifically (6), (12), and (13), any non-ASCII content in Newsgroups and
> Followup-To would be encoded in punycode (or some other suitable encoding
> method).

13 and 14 are the areas which are incompatible with IMAP.

> It
> requires only minimal changes to (2), (3), and (4), the existing news
> transit system, to remove restrictions preventing creation of non-ASCII
> newsgroups.  It is believed that essentially all existing news transit and
> server systems still in active use can handle 8-bit newsgroup names
> without difficulties.  It would be desirable for (2), the injection agent,
> to be able to undo the mail encoding automatically.
[...]
> Additionally, a news reader (5) may be able to read such groups without
> modification if it already has support for 8-bit characters and can be
> configured appropriately, and similarly a news posting agent (1) may also
> be able to be used without modification.  Updates to (1) and (5) to
> provide Unicode character entry, canonicalization, and display would of
> course be extremely desirable.  (1) and (5) require no modifications to
> deal with existing ASCII newsgroups except modifications for 8-bit
> cleanliness to handle crossposted messages.
[...]
> Moderation software (10) would have to change in order to handle any
> non-ASCII groups, since the mail encoding would have to be decoded, or the
> moderator would have to arrange to use an updated injecting agent (2).
[...]
> Any news to mail gateway (12) would have to be modified if it received any
> messages crossposted to non-ASCII newsgroups and wanted to preserve the
> Newsgroups header in the e-mail message.  Failure to encode the headers
> appropriately would result in unencoded 8-bit text in the headers of a
> mail message, where it may be mangled or rejected by the mail system.
[...]
> Any IMAP server processing Usenet messages (13) would have to perform the
> same transformations, encoding newsgroup names in Newsgroups, Followup-To,
> and Control (and Xref if the IMAP server wished to maintain it).

Those are all backwards incompatibilities.  Moreover the issues affecting
13 incorrectly assume that "news" and "mail" can be differentiated. And
while *some* UAs might be usable unmodified *some* _will_ require
modification to support UTF-8 I/O; there is no alternative as that
must be what the UA puts in the article and communicates with via NNTP
(and as that applies both to origination and followups, it affects both
1 and 5).

> (B) punycode in e-mail, IMAP, and articles, UTF-8 in NNTP
> =========================================================

> All NNTP commands would take UTF-8 arguments for newsgroup names, and the
> newsgroup names returned by LIST, GROUP, and similar commands would be in
> UTF-8.

That's a backwards-compatibility for any IMAP servers dealing via
NNTP.

> NNTP servers (2) and (4) must be modified in order to carry non-ASCII
> newsgroups to decode the newsgroup headers when receiving messages so as
> to know what newsgroup into which to file them.  The active file would
> also need to be kept in UTF-8.  As above, it is believed that the other
> NNTP commands besides POST/IHAVE/TAKETHIS would work without modification
> because existing NNTP software is already 8-bit clean.  If the NNTP
> software is not modified, the newsgroups will show up in their punycode
> encoded form, possibly confusing compliant news reading software.

More backwards incompatibilities.

> IMAP servers (13) may want to recode the newsgroup names from
> punycode to UTF-7, but would not need to make any transformations to the
> articles themselves.

Why would anybody want to "recode [...] punycode to UTF-7"?
As noted above, there may be issues with IMAP<->NNTP communications.

> (C) punycode everywhere
> =======================

> This proposal mandates modifications to the posting agents (1) and the
> news readers (5) in order to properly display the names.

Strictly speaking, that is not mandated; 1 and 5 can still be used
by users. Users can still post, read, follow-up, etc.; decoding
for display (where underlying Unicode and font support is available)
is a nicety, but not a necessity.

> No modifications are required to (2) or (4), the NNTP servers, although
> without modifications the server administrator would have to work with
> encoded group names.  It would provide a much better user interface if the
> administrative tools implemented punycode encoding and decoding for easier
> handling of non-ASCII newsgroup names.

Use of the canonical name may in fact be a benefit, e.g. in the case
of an administrator not familiar with Oriental, Cyrillic, Hebrew, Arabic,
Devanagari, etc. letterforms when dealing with some non-ASCII names.
Therefore the last statement above "It would provide a much better..."
is questionable.

> IMAP servers (13) may again wish to recode punycode to UTF-7 for newsgroup
> names, but otherwise require no modification.

Why would anybody want to "recode punycode to UTF-7"?

> Summary
> =======
> 
> The following chart summarizes the backward compatibility issues for each
> proposal and each component of the news system.  For each portion of the
> news system, N means no change required, Y means change is required to
> correctly handle non-ASCII newsgroups, D means change is very desirable
> but not absolutely necessary, and C means change would be convenient but
> unmodified software is still fairly usable.
> 
>   | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
>  -+------------------------------------------------------
>  A| D   C   N   N   D   Y   N   N   N   Y   N   D   Y   D
>  B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   C   D
>  C| D   C   C   C   D   N   N   N   N   N   N   N   C   D

Based on the comments above, there are a few errors, I've had to
add a I (for incompatible) category, and I've added a fourth row:

  | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
-+------------------------------------------------------
A| Y   C   N   N   Y   Y   N   N   N   Y   Y   Y   I   I
B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   Y   D
C| C   C   C   C   C   N   N   N   N   N   N   N   C   C
D| C   C   C   C   C   N   N   N   N   N   N   N   C   C

> The above summary I believe correctly indicates that proposal (B) requires
> the most changes to be made to the news system itself.

7-9 are irrelevant (identical for all rows), and are the only
mail-specific items; everything else is in one way or another
part of "the news system". A is completely incompatible with
IMAP, and has more Y's than B. So with the caveat that A is
not viable, then yes, of the remaining viable approaches, B
has backwards incompatibilities whereas C and D do not.

> It's possible to
> use existing software without modification with either proposal (A)

Not IMAP.

> [...] but with other news readers it may be
> impossible to access a non-ASCII group because no Unicode entry is
> supported.

That is why 1 and 5 in A and B are Y. [even if they are considered
D rather than Y, the situation w.r.t. overall backwards compatibility
is unchanged, due to the presence of I's or other Y's in rows
A and B]

> Under proposal (C), we're guaranteed that nothing will break
> and that it will always be possible to access even non-ASCII groups, but
> no existing client will display the names correctly.

Likewise for D (other than clients that display, or can display,
the description, which inclues a few).

> Overall, it is somewhat less necessary to change client software with (A)
> than with (C); exactly how much less necessary is something of an open
> question.

Not so with IMAP clients; A is incompatible.

> Proposal (A) is the only proposal that requires changes to any system
> outside of the news system (other than changes to an IMAP client to
> understand the punycode newsgroup names, which are the same for all
> proposals).

As mentioned above, the only mail-specific parts are irrelevant, and
everything else is part of the news system.

 > Both (B) and (C) work with the moderation, e-mail, and IMAP
> infrastructure without any additional changes.

B has backwards incompatibilities with IMAP, at least where
there is communication between an IMAP server and an NNTP
server.

Any Y or I is an incompatibility and rules out any chance of IESG
approval (barring a transition plan and protocol negotiation with
fallback where feasible).  That leaves C and D.  C requires more
background work before it can even be specified adequately
(specifically the stringprep), whereas D requires only specification
of how the description field is handled (and certainly the current
guess-the-charset, guess-the-language situation would function, though
not at all ideally -- even if C is used rather than D, the charset
and language issues in the newsgroup descriptions need proper i18n
consideration, and that includes both charset and language).

Date: Mon, 24 Feb 2003 15:59:53 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: usenet-format <[email-masked]>, [email-masked]
Subject: Re: Tightening RFC 2822 msg-IDs
Sender: [email-masked]

On Sun, 23 Feb 2003, Lawrence Greenfield wrote:
>    ...what would ietf-822 folks think about
>    tightening the legal-to-generate Message-ID definition when RFC 2822
>    moves to draft standard? ...
> 
> I believe having a Usenet format document that updates RFC 2822 on the
> standards track would be a very good thing, especially if all of the
> changes are like messageid: of the form of moving more syntax into the
> obsolete section and further restricting what is generated.

Note that it is unlikely that a news standard can be phrased entirely in
such terms, if only because it must address news processing as well as
format (the title of the current Usefor WG draft is a bit misleading, as
examination of the content will show). 

And some of the restrictions that will be necessary for compatibility with
the news installed base are probably unacceptable on the mail side.  (This
is one reason why a general mail-to-news gateway is non-trivial software.)

That said, it would certainly be good to at least *reduce* the format
differences, if it can be done without serious installed-base impact on
either side.  We news folks could certainly supply a wishlist of format
restrictions.  Tightening up message-ID syntax is a big one, given how
crucial message IDs are to news handling, processing, display, etc., and
given that restrictions on them are already embedded in both our software
and our protocols. 

                                                          Henry Spencer
                                                       [email-masked]

content-class: urn:content-classes:message
Subject: RE: yEnc (was RE: mail vs. news ???)
Importance: normal
Date: Mon, 24 Feb 2003 09:55:01 -0800
From: "Dan Kohn" <[email-masked]>
To: "Juergen Helbing" <[email-masked]>, <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Juergen Helbing wrote:

>> I have tried to persuade the relevant people to produce an
>> internet-draft proposing it as an official CTE. But they don't want
>> to know. 

Juergen, could you please have the relevant people (including yourself)
take a look at the following and tell us what you think:

http://www.imc.org/ietf-822/mail-archive/msg03251.html
http://www.imc.org/ietf-822/mail-archive/msg03252.html

Given that the Area Director himself (and the MIME co-author who wrote
the scary warning against new CTE's) wrote one of these drafts, I'm
optimistic that it can get approved the IESG.  So, this would be a
really, really good time to get your comments in, before it gets locked
in stone.  (I wouldn't expect a lot of new CTE's going forward.)

[Stuff about the IETF being too much trouble elided.]

Here's your chance, Juergen, to have it operate quickly and well.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Mon, 24 Feb 2003 16:50:34 +0100
From: [email-masked] (Ralph Babel)
To: [email-masked], [email-masked]
Subject: Re: mail vs. news ???
Sender: [email-masked]

Dave Crocker wrote:

> Message-ids are not nearly as deeply and usefully and
> broadly used as we had expected them to be. However,
> they ARE used, to good effect, for duplicate
> detection and for thread construction.

Judging by the recent flood of duplicates on the Usefor
mailing-list, I'd say that we still have a long way to go.

Date: Mon, 24 Feb 2003 04:07:00 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: Dan Kohn <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> Juergen Helbing wrote:

> > And because it seems necessary to re-specify yEnc entirely before it
> > is used as an official MIME encoding this might be the right time to
> > learn from the past and to do it better now.

> Here's some procrastination (a gzip-8bit CTE) from my procrastination (a
> new Usefor draft) from my real work.  As always, comments are
> appreciated.

How amusing. I just submitted an Internet Draft describing almost exactly the
same thing. I've attached a copy of what I wrote below.

So who wants to do  the merge?

				Ned

Network Working Group                                           N. Freed
Internet-Draft                                          Sun Microsystems
Expires: August 25, 2003                               February 24, 2003

              Deflate-8bit and Deflate-base64: Compression
                  Content-Transfer-Encodings for MIME
                     draft-freed-mime-newenc-00.txt

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at http://
   www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on August 25, 2003.

Copyright Notice

   Copyright (C) The Internet Society (2003).  All Rights Reserved.

Abstract

   This document defines two additional MIME content-transfer-encodings,
   deflate-8bit and deflate-base64.  Adding these CTEs to MIME that
   provide facilities for loss-less, adaptive, general-purpose
   compression.  The first of these, deflate-8bit, produces 8bit output,
   while the second, deflate-base64, produces the same sort of output as
   the base64 content-transfer-encoding defined in RFC 2045.

Freed                   Expires August 25, 2003                 [Page 1]

Internet-Draft              Compression CTEs               February 2003

1. Introduction

   The MIME specification RFC 2045 [2] defines several
   Content-Transfer-Encodings:

   1.  7bit, used to label textual 7bit data,

   2.  8bit, used to label textual 8bit data,

   3.  binary, used to label binary data,

   4.  quoted-printable, normally used to transform 8bit textual data to
       7bit form, and

   5.  base64, normally used to transform binary data to 7bit form.

   All of these encodings produce output that greater than or equal to
   the input data in length.  In particular, quoted-printable can incur
   up to 300% overhead and base64 incurs a fixed 33% overhead.  This
   amount of overhead can be significant in some applications.

   This document defines two new CTEs that incorporate the popular
   deflate compression algorithm described in RFC 1951 [1].  The first
   of these, deflate-8bit, also incorporates a lightweight encoding
   based on the popular yEnc [6] encoding scheme.  The resulting
   material is often smaller than the input even when the output range
   is restricted to the base64 alphabet.

Freed                   Expires August 25, 2003                 [Page 2]

Internet-Draft              Compression CTEs               February 2003

2. Conventions Used In This Document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [3].

Freed                   Expires August 25, 2003                 [Page 3]

Internet-Draft              Compression CTEs               February 2003

3. Deflate Compression

   The deflate compression format described in RFC 1951 [1], as used by
   the PKZIP and gzip compressors and as embodied in the freely and
   widely distributed zlib [Gailly95] library source code, has the
   following features:

   o  An apparently unencumbered encoding and compression algorithm,
      with an open and publicly-available specification.

   o  low-overhead escape mechanism for incompressible data,

   o  heavily used for many years in networks, on modem and other
      point-to-point links to transfer files for personal computers and
      workstations,

   o  easily achieves 2:1 compression on the Calgary corpus [5] using
      less than 64KBytes of memory on both sender and receive.

Freed                   Expires August 25, 2003                 [Page 4]

Internet-Draft              Compression CTEs               February 2003

4. The Deflate-8bit Content-Transfer-Encoding

   The deflate-8bit encoding process consists of applying the deflate
   algorithm defined in RFC 1951 [1] to a MIME object in canonical form.
   Since all MIME objects are potentially independent of each other the
   compressor's history MUST be cleared prior to performing the
   compression operation.

   The output of dellfate algorithm is binary data.  This binary data is
   then encoded as follows:

   1.  42 is added to each octet modulo 256.

   2.  If the resulting octet has the decimal value 61 (equals sign), 13
       (CR), 10 (LF), or 0 (NULL) it must be escaped.  This is done by
       prefixing the octet with an octet of value 64 and adding 64 to
       the resulting octet modulo 256.

   3.  A CRLF sequence MUST be inserted into the output after every 256
       octets of output.  A trailing CRLF MUST also appear at the end of
       the data.  Implementations MUST be tolerant of CRLFs being
       inserted between the escape prefix and the octet it modifies.

   4.  Additional octets MAY be escaped using the previously described
       procedure.  In particular, implementations SHOULD escape any
       octets with the values 32 (space) or 9 (tab) that would otherwise
       appear at the end of a line.

   8bit data is produced by this encoding process.  As such, this CTE
   can only be used in conjunction with transports capable of handling
   8bit data.

   Decoding consists simply of reversing the encoding process, that is,
   first reversing the encoding described above and then applying the
   inflate algorithm.

Freed                   Expires August 25, 2003                 [Page 5]

Internet-Draft              Compression CTEs               February 2003

5. The Deflate-Base64 Content-Transfer-Encoding

   The deflate-base64 encoding process consists of applying the deflate
   algorithm defined in RFC 1951 [1] to a MIME object in canonical form.
   The result of the deflate compression operation is then further
   encoded using the base64 scheme defined in RFC 2045 [2].  Since all
   MIME objects are potentially independent of each other the
   compressor's history MUST be cleared prior to performing the
   compression operation.

   The output of this process has the same range as the base64 CTE, and
   can be used with any transport.

   Decoding consists simply of reversing the encoding process, that is,
   first reversing the base64 encoding and then applying the inflate
   algorithm.

Freed                   Expires August 25, 2003                 [Page 6]

Internet-Draft              Compression CTEs               February 2003

6. Appropriate Use

   As deflate-8bit produces 8bit material as output, it MUST NOT be used
   with transports that do not support 8bit, such as tranditional SMTP.
   Happily, most SMTP transports currently support the 8bitMIME SMTP
   extension and hence can accomodate the use of deflate-8bit

   Both deflate-8bit and deflate-base64 SHOULD only be used when the
   originator has some indication that the recipient can decode them.
   Note that this document does not specify a means by which such
   support can be indicated.

Freed                   Expires August 25, 2003                 [Page 7]

Internet-Draft              Compression CTEs               February 2003

7. Security Considerations

   The deflate algorithm is complex and hence prone to implementation
   errors.  In particular, certain inflate implementations are known to
   not perform sufficient checking of their input stream and hence may
   be vulnerable to certain forms of attack.  Aside from this, the new
   content-transfer-encodings specified in this document are believe not
   to raise any security considerations not already present in MIME
   itself.

Freed                   Expires August 25, 2003                 [Page 8]

Internet-Draft              Compression CTEs               February 2003

Normative References

   [1]  Deutsch, P., "DEFLATE Compressed Data Format Specification
        version 1.3", RFC 1951, May 1996.

   [2]  Freed, N. and N. Borenstein, "Multipurpose Internet Mail
        Extensions (MIME) Part One: Format of Internet Message Bodies",
        RFC 2045, November 1996.

   [3]  Bradner, S., "Key words for use in RFCs to Indicate Requirement
        Levels", BCP 14, RFC 2119, March 1997.

Freed                   Expires August 25, 2003                 [Page 9]

Internet-Draft              Compression CTEs               February 2003

Informative References

   [4]  Freed, N., Klensin, J. and J. Postel, "Multipurpose Internet
        Mail Extensions (MIME) Part Four: Registration Procedures", BCP
        13, RFC 2048, November 1996.

   [5]  Bell, T. and I. Witten, "Text Compression", Prentice-Hall
        Englewood Cliffs NJ, 1990.

   [6]  Helbing, J., "yEncode - A quick and dirty encoding for
        binaries", http://www.yenc.org/yenc-draft.1.3.txt version 1.3,
        2002.

Author's Address

   Ned Freed
   Sun Microsystems
   1050 Lakes Drive
   West Covina, CA  91790
   USA

   Phone: +1 626 850 4350
   EMail: [email-masked]

Freed                   Expires August 25, 2003                [Page 10]

Internet-Draft              Compression CTEs               February 2003

Intellectual Property Statement

   The IETF takes no position regarding the validity or scope of any
   intellectual property or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; neither does it represent that it
   has made any effort to identify any such rights.  Information on the
   IETF's procedures with respect to rights in standards-track and
   standards-related documentation can be found in BCP-11.  Copies of
   claims of rights made available for publication and any assurances of
   licenses to be made available, or the result of an attempt made to
   obtain a general license or permission for the use of such
   proprietary rights by implementors or users of this specification can
   be obtained from the IETF Secretariat.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights which may cover technology that may be required to practice
   this standard.  Please address the information to the IETF Executive
   Director.

Full Copyright Statement

   Copyright (C) The Internet Society (2003).  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assignees.

   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION

Freed                   Expires August 25, 2003                [Page 11]

Internet-Draft              Compression CTEs               February 2003

   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement

   Funding for the RFC Editor function is currently provided by the
   Internet Society.

Freed                   Expires August 25, 2003                [Page 12]


content-class: urn:content-classes:message
Subject: gzip-8bit
Importance: normal
Date: Mon, 24 Feb 2003 02:31:50 -0800
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Juergen Helbing wrote:

> And because it seems necessary to re-specify yEnc entirely before it
> is used as an official MIME encoding this might be the right time to
> learn from the past and to do it better now.

Here's some procrastination (a gzip-8bit CTE) from my procrastination (a
new Usefor draft) from my real work.  As always, comments are
appreciated.

Gzip-8bit Content-Transfer-Encoding

This document specifies a new standards-track Content-Transfer-Encoding
(CTE), designed mainly for use within the Usenet environment.  It
combines gzip [RFC 1952] compression (which uses RFC 1951 deflate) with
conversion from binary to 8bit, to create a encoding that is both
compact and can successfully transit nearly 100% of the Usenet
infrastructure.

Why a new CTE?

RFC 2045 says:

   Unlike media types and subtypes, the creation of new Content-
   Transfer-Encoding values is STRONGLY discouraged, as it seems likely
   to hinder interoperability with little potential benefit.

Obviously, gzip-8bit needs to overcome a high hurdle to justify
standardization.  The basic argument for a new CTE is that a significant
portion of Usenet traffic consists of posting binary content, and that
users are currently manually encoding that content with one or more
non-standardized schemes.  In particular, the adoption of [yEnc]
<http://www.yenc.com> shows the interest by implementers and end-users
in a format that does not have the 33% overhead over binary of base64.

It is believed that a standards-based replacement for yEnc could learn
from their technical and implementation experience, and provide a better
end-user service without causing any significant interoperability
failures.  (I.e., receivers will need to support the new CTE, but then
they also had to or have to learn to support yEnc.)

Why gzip?

Although significant portions of posted content (including many image,
video, and audio files) are already in a compressed format and so will
not benefit directly from gzip compression, gzip still provides
benefits.  Specifically, it provides a run length encoding that
minimizes a bias that otherwise exists in binary files toward excessive
NULs.  By contrast, gzip output is far more evenly distributed among the
256 possible octets.

For text and many executable files, gzip can provide lossless
significant compression, often of 50% or more.  The algorithm used is
fully described in RFCs 1951 and 1952, and is amenable for use as a
Unix-style software filter.  It is believed to unencumbered by patents.
Software implementing gzip is widely available, including in open source
libraries.

Although it may seem inefficient to run gzip on content that may not be
especially compressible, the processing cost and latency of doing so is
insignificant on modern hardware.  It is believed that creating a
single, new CTE that always compresses with gzip (and then always
encodes in 8bit), is far better than creating multiple, optional
variations.

Why 8bit?

Nearly the entire Usenet infrastructure (user agents, injectors, and
servers) is 8bit-clean, but not binary-clean.  Quoting RFC 2049:

    (2)   Many systems may elect to represent and store text data
          using local newline conventions.  Local newline
          conventions may not match the RFC822 CRLF convention --
          systems are known that use plain CR, plain LF, CRLF, or
          counted records.  The result is that isolated CR and LF
          characters are not well tolerated in general; they may
          be lost or converted to delimiters on some systems, and
          hence must not be relied on.

    (3)   The transmission of NULs (US-ASCII value 0) is
          problematic in Internet mail.  (This is largely the
          result of NULs being used as a termination character by
          many of the standard runtime library routines in the C
          programming language.) The practice of using NULs as
          termination characters is so entrenched now that
          messages should not rely on them being preserved.

What content can it be used for?

As with all CTEs, the gzip-8bit CTE can be used with any media type.
Content in the text top-level must be canonicalized per RFC 2049 before
being encoded.

What about splitting files?

Gzip-8bit is NOT compatible with message/partial, which requires 7bit
Content-Transfer-Encoding.  A new MIME type such as
application/news-partial could be specified, but is it believed that the
disadvantages of doing so outweigh the benefits.  

The gzip-8bit CTE

What about gateways?

RFC 2045 explicitly prohibits multiple encodings, so it is not
acceptable for a gateway from news to mail to simple further encode
gzip-8bit in base64.  Instead, such a gateway could decode the gzip-8bit
CTE and then encode in base64.  Alternatively, if the destination MTA or
MUA supports 8BitMIME (RFC 1652) or BinaryMIME (RFC 3030), the gateway
can attempt to forward the message without modifying the CTE.  However,
due to the hop-by-hop nature of email, such a message could encounter an
MTA that only supports 7bit transport, which would have no choice but to
reject the message.

What are the steps for encoding?

Textual content is converted to canonical format, as described in RFC
2049.  gzip encoding is applied to the content as described in RFCs 1951
& 1952.  The CRC-32 value MUST be set.  The fname parameter MUST NOT be
set, as Content-Disposition [RFC 2183] used with [RFC 2231] provides a
standards-based way of providing this information that also supports
internationalization.  More fundamentally, it would be a layer
violation, because a filename is an attribute of a media type, not a
CTE.

The output of that compression is processed 900 octets at a time.  For
each 900 octets, NUL is replaced with "=A", CR with "=B", LF with "=C",
and "=" with "=D".  The octet stream is then appended with CRLF, and the
next 900 octets are processed the same way.

900 is chosen as the unescaped octet stream length because RFC 2822
prohibits lines to exceed 998 octets, plus the ending CRLF.  Since the
number of octets that will be escaped is not known, 900 seemed to
provide a large amount of margin (i.e., room for 49 escaped octets).
Since each octet output by gzip should be approximately equally likely,
and there are 256 possibilities, there should be on average 4 / 256 *
900 = 14 octets per line that need to be escaped.

MIME sending agents MUST NOT create line lengths of more than 998 octets
(plus the CRLF).  MIME receiving agents SHOULD accept gzip-8bit content
of any line length.

What are the steps for decoding?

CR and LF characters are removed from the input stream.  The 4 escaped
octets are replaced with their unescaped translations.  The resulting
stream is decompressed as described in RFCs 1951 & 1952, and the CRC-32
value MUST be checked.  If the content is in the text top-level media
type, line endings are converted to local form.

Where's the registration?

RFC 2048 requires the following information for a new CTE:

Naming Requirements

The name "gzip-8bit" conforms to the syntax of the CTE header field.

Algorithm Specification Requirements

The algorithm used is fully described in RFC 1952 and in this document.
It is believed to unencumbered by patents.  Software implementing gzip
is widely available, including in open source libraries.

Input Domain Requirements

Gzip-8bit is applicable to an arbitrary sequence of octets of any length

Output Range Requirements

Gzip-8bit is explicitly designed to always output 8bit data.

Data Integrity and Generality Requirements

Gzip-8bit is fully invertible on any platform.

New Functionality Requirements

Gzip-8bit requires approximately 33% less bytes to represent most binary
objects than base64.  For textual and much executable content, it can
often create a 50% or more reduction in size.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

From: [email-masked] (Juergen Helbing)
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked]
Subject: Re: yEnc (was RE: mail vs. news ???)
Date: Mon, 24 Feb 2003 09:39:04 GMT
Sender: [email-masked]

Charles,

your message to the IETF-822 list needs some comments - and updates -
from my side. So I'm sending this reply to you (and perhaps it appears
on the list).

On Sun, 23 Feb 2003 16:36:07 GMT, you wrote:

>Yes, I have been round this before. It is clear that, even if the Yenc
>people were to use
>    Content-Transfer-Encoding: x-yenc
>which they could start doing tomorrow, things would be much simpler.

For whom would be things "much simpler" ?

>I have tried to persuade the relevant people to produce an internet-draft
>proposing it as an official CTE. But they don't want to know. 

Please realise that "they" (me) always wanted yEnc embedded into MIME
and that "they" (we) collected far more information about the entity
of problems than anybody else.

btw.: The work is "in progress", not burried.
The latest proposal from Russ Albery (posted to news.software.nntp)
is a part of that work.
My latest problem is that the "Content-ID" which is used to identify
multiparts can be easily _faked_ on Usenet. News people have far more
serious problems with MIME than the Mail people can imagine. The
solution to that problem is far more complicated and complex than yEnc
itself :-(

>They seem
>convinced that the IETF is totally against all change, and that they would
>be heavily sat upon, and that the whole process would be far too
>cumbersome and time consuming, 

If I'm looking at the latest discussions about USEFOR here then I
really dont want to imagine what could happen to a yEnc RFC.

And there is one thing you should never forget:

yEnc is a proposal for transporting binaries _efficiently_ and
crc-protected of USENET. As soon as it is a MIME encoding this entire
stuff would spread to eMail. And I would be really curious how many
mail-relais are corrupting 8-bit-posts then. Furthermore existing
eMail clients seem to _modify_ the body - especially when it comes to
leading/trailing tabs/spaces (but also ascii-dec-26)....

Usenet is a perfect playground to test and evaluate such toys as yEnc.
But eMail is a _business_critical_ application around the world.
I believe that this is the reason why this list (and IETF) is
extremely careful when it comes to changes for eMail.

So please apologize that I dont want to be one who is messing up the
worldwide eMail system 'just to save 25% of bandwidth' - which is
negligable for Mail - but important for Binary Newsgroups (where yEnc
belongs to).

>and "hey it works fine already, we don't need no stinkin' IETF". 

Yes, yEnc is working pretty good. It is saving Gigs of bandwidth every
day for USENET. 

FYI: Usenet does not need IETF. The neticens there have invented many
nice things on Usenet even without the admins and us - the developers.

These people dont _care_ about us (including myself).

>All very sad, but that's how some people regard
>the IETF. So we are stuck with indicating the presence of yEnc by a flag
>in the Subject header. It is a mess.

Update: More and more posts appear on Usenet even without the
mandatory keyword "yEnc" in the subject line. It is necessary to scan
the body.

More and more binary posts appear on Usenet with invalid filenames. It
is necessary to scan third party systems and protocols to know what's
inside a binary. 
More and more binary posts appear on Usenet without a subject - or
with a misleading or "encrypted" subject (uncle tom arrives tomorrow
at 7pm on JFK International)....

Usenet is not eMail.
And Binary Usenet is not Usenet.

The _real_ good question would be if it makes any sense to TRY to have
them compatible. From my latest experiences I'm even convinced to say:
Fingers Off. 

Last not least:

Introducing MIME in the Binary Usenet is one thing nobody ever was
able to do. All that "nice stuff" which is possible with MIME is not
what neticens expect there - or what they want. Together with the fact
that all newsreaders (and all news->web gateways) must be modified
then it seems to be more than hard work. And because the neticens
expect smooth transition (the posts must be at least usable on 99% of
the newsreaders by the help of external decoders) the yEnc embedding
into MIME must be realised carefully. 
This requires also a proper check of _all_ the MIME capable
news-readers out there. And I'd really like to know who wants to do
that job..... (It takes years of hard - unpayed - work).

That's my current status - 15 months after the publication of yEnc, 24
months after the first discussions about it.

Today things are more difficult than yesterday.

CU
-- 
Juergen
--------------------
www.yenc.org

From: [email-masked] (Juergen Helbing)
To: "Dan Kohn" <[email-masked]>
Cc: [email-masked]
Subject: Re: yEnc (was RE: mail vs. news ???)
Date: Mon, 24 Feb 2003 09:48:31 GMT
Sender: [email-masked]

On Sun, 23 Feb 2003 20:31:00 -0800, you wrote:

>An 8bit-safe gzip C-T-E would add real value.

I've been waiting for such a CTE since I entered the Internet and
received the first BitMap and Winword Docs by eMail  :-)

And because it seems necessary to re-specify yEnc entirely before it
is used as an official MIME encoding this might be the right time to
learn from the past and to do it better now.

CU
-- 
Juergen

To: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, [email-masked]
Subject: Re: yEnc (was RE: mail vs. news ???)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sun, 23 Feb 2003 23:49:09 -0800
Lines: 24
Sender: [email-masked]

Brad Templeton <[email-masked]> writes:

> As I have indicated before, there is no purpose for message/partial in
> USENET news, and thus no reason to do a news/partial form.

It's not worth even getting into this discussion with anyone who hasn't
read the *huge* recent discussion of this topic on news.software.nntp,
which covered the various issues around multipart posts on Usenet in more
detail than has ever been talked about on usenet-format.  There are a lot
of issues concerning multipart posts that are very peculiar to Usenet,
such as active attempts to corrupt posts by injecting faked fragments,
news server issues with handling and reassembling large files correctly,
and the *much* higher likelihood that one or more segments will be lost in
transit.

As usual, backward compatibility issues end up being significant in a lot
of places.  Any theoretical solution is pointless unless the people
posting binaries use it.

And yes, article fragmentation and reassembly at the NNTP protocol level
was one of the things that was discussed (extensively).

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sun, 23 Feb 2003 21:05:54 -0800
From: Brad Templeton <[email-masked]>
To: Dan Kohn <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, [email-masked]
Subject: Re: yEnc (was RE: mail vs. news ???)
Mail-Followup-To: Brad Templeton <[email-masked]>, Dan Kohn <[email-masked]>, Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, [email-masked]
Organization: http://www.templetons.com/brad
Sender: [email-masked]

On Sun, Feb 23, 2003 at 08:31:00PM -0800, Dan Kohn wrote:
> Of course, this should be defined as a C-T-E.  Since Section 5.2.2 of
> RFC 2046 restricts message/partial to be 7bit only, it would also seem
> worthwhile to specify an application/news-partial with nearly identical
> semantics, except for allowing use of 8bit content.  You would need to
> specify whether non-8BitMIME gateways were allowed to re-encode the

As I have indicated before, there is no purpose for message/partial
in USENET news, and thus no reason to do a news/partial form.

There are two reasons you might want to break a large file up into
components:
    a) You are going to transmit the file over a link that has
    technical problems carrying a large file, and so must
    disassemble and reassemble it to fit within the technical limits.

    b) You wish to bypass a policy restriction on the size of files.

Well, we're hardly here to encourage reason (b).  In general, people
setting policy restrictions on file sizes want to know the full size
of any article, and want to decide to accept or reject it for policy
reasons based on that.  They don't want to be deliberately mislead
about the size -- or have the size be available but hard to figure
out, because one must process all the parts to get it, usually defeating
the resource-saving purpose of the policy.

So that leaves (a).  This is a link issue.    When it comes to regular
USENET, there are few if any technical reasons for a limit below
a gigabyte with today's equipment.  Every protocol, ever tool,
we have has no reason to be unable to handle an article of any size
up to the file size limit of the host operating system.  Older tools
may have had limits, of course, but there is no reason for new ones
to.   Of course some OSs limit files to 2GB.

However, individual links may have problems.  It is the wrong design
to expect the poster and the newsreader to know and solve the problems
of the large number of intervening links.  Packet disassembly and
reassembly is the job of the router even in IP, the most end to end
of protocols.  Wisely so, as the sender doesn't know the vagaries of
the right chunk size for any given link.

So it's time to start telling the protocols that carry USENET to
do any disassembly and reassembly needed if they can't handle large
files, and to make that invisible to the format and the higher level
protocols.

content-class: urn:content-classes:message
Subject: RE: yEnc (was RE: mail vs. news ???)
Importance: normal
Date: Sun, 23 Feb 2003 20:31:00 -0800
From: "Dan Kohn" <[email-masked]>
To: "Usefor Mailing List" <[email-masked]>, "IMAP mailing list" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Larry Osterman wrote:

> Out of curiosity, what are those issues?  Is yEnc already encumbered
> with the same legacy interoperability issues that prevented uuencode
> from being standardized?

Essentially, yes.  See these implementation notes
<http://www.yenc.org/yEnc-Notes3.txt>.

My question is that if you first gzip the output (which I appreciate
will only slightly reduce the size of some image and sound files), won't
you be able to avoid long runs of NULs (due to the run length encoding).
If so, why not skip the rotation step entirely, simply escape the NULs,
CRs, and LFs, and add a CRLF every 900 or so characters.

Of course, this should be defined as a C-T-E.  Since Section 5.2.2 of
RFC 2046 restricts message/partial to be 7bit only, it would also seem
worthwhile to specify an application/news-partial with nearly identical
semantics, except for allowing use of 8bit content.  You would need to
specify whether non-8BitMIME gateways were allowed to re-encode the
content into base64 (yes, thereby violating the MIME no double-encoding
rule), or whether they should just drop the message on the floor.

The main lesson from yEnc, I believe, is that implementers and users
*will* adopt new technologies, if they add value.  An 8bit-safe gzip
C-T-E would add real value.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

To: [email-masked]
Xref: clerew local.mime:2431
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: mail/news unification
Date: Sun, 23 Feb 2003 16:53:23 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>To me, the only user-level difference that is major is one of
>addressing, between explicitly sending to one or more individuals,
>versus explicitly sending to a "group". In this regard, I believe that
>that netnews is essentially the same as an email list.

>Yes, one subscribes to a list but not to a newsgroup. However think
>about the mode of posting to an open mailing list that you are not
>subscribed to, and using the mailing list archive as the basis for
>reading contributions.  It is darn close to the style of a newsgroup.

Yes, the superficial similarities are apparent, but there are real serious
differences behind the scenes. You see this when you try to gateway
between a newsgroup and a mailing list. The simple cases work fine, but
just think what happens if an article is submitted to two mailing lists,
or is posted to several newsgroups with more than one of them gatewayed to
a mailing list. Life suddenly starts to get extremely complicated for the
poor implementor(s).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: Usefor Mailing List <[email-masked]>
Cc: IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 23 Feb 2003 23:56:34 +0000
Lines: 38
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 [message-ids]

 Bruce> If we're going to set a limit, we have to pick a number.
 Bruce> Let me try again, from a different perspective:

 Bruce> 1. The NNTP limit appears to be 501 octets, minus whetever the
 Bruce> longest command is, minus one

no, 510 minus the longest relevent command minus one, giving 501 as
the absolute maximum limit (due to TAKETHIS). The draft NNTP revisions
seem to have a limit of 12 characters for command names, which would
make the limit 497. In fact there's also an issue if you go over about
452 characters, when the ARTICLE or STAT response strings start to go
over 512 total; plus (I'd have to check the latest draft) possible
commands like "HDR someheader <id>".

Basically if you go over 501 then the transport collapses in ruin
(well, almost); whereas the 450-500 range just causes some minor
difficulty.

 Bruce> 3. So what's so special about 250, rather than 255 or 256?

it happens to be a value historically used by INN (present in various
1.x versions, then removed in 2.x, then explicitly added back when it
was discovered that yes, there really does have to be a limit
otherwise things break) and believed to be acceptable to other
software, while still giving plenty of headroom (message-ids of length
100-120 show up occasionally, ids of length 120-150 much more rarely,
that's about the longest seen in practice other than due to breakage
of some sort such as gateway loops).

In the absence of a really good reason (i.e. not merely aesthetic) to
use some other value, that's the one to pick.

-- 
Andrew.

Date: Sun, 23 Feb 2003 18:10:15 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Mark Crispin <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>, Keith Moore <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Mark Crispin wrote:
> On Sun, 23 Feb 2003, Bruce Lilly wrote:

>>2. It would be more convenient, IMO, to have a uniform limit
>>    for SMTP Path (a.k.a. angle-addr) and for msg-id rather than
>>    two close but different limits.
> 
> 
> SMTP addresses and message-ids may use a more or less equivalent syntax,
> but I think that it's stretching it to say that they are the same.

I didn't say they are the same (I did say that at one time
the syntax was identical).

If we're going to set a limit, we have to pick a number.
Let me try again, from a different perspective:

1. The NNTP limit appears to be 501 octets, minus whetever the
    longest command is, minus one

2. Typical limits for msg-id components and msg-id-like
    constructs seem to be a power of two, or one less than
    a power of two, e.g.

    maximum host name length         255  RFC 1035 [3.1]
    maximum subdomain length         63   RFC 1034 [3.5], RFC 1035 [2.3.1], also RFC 1123 [2.1] for host
    maximum domain name length       255  RFC 1034 [3.5], RFC 1035 [2.3.4], RFC 2821 [4.5.3.1]
    maximum Path (angle-addr) length 256  RFC 821 [4.5.3], RFC 2821 [4.5.3.1]

    The first three are related to the domain part of a msg-id,
    and the last has the same general form <local-part@domain>.

3. So what's so special about 250, rather than 255 or 256?

To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 23 Feb 2003 23:04:18 +0000
Lines: 16
Sender: [email-masked]

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> This business about spaces not being optional after the colon
 Mark> is silly, as are similar such "differences".  Yes, almost all
 Mark> messages have LWSP after the colon.  But a programming mistake,
 Mark> particularly in some version of rn or equivalent dinoware, is
 Mark> even less of a justification for a separate message format than
 Mark> a length limit.

the requirement for spaces (not just LWSP, but an actual SP character)
after colons is from 1036 (and was in 850 before that), so it's been
part of the news standards for as long as they have existed.
Following that requirement is not a "programming mistake".

-- 
Andrew.

Date: Sun, 23 Feb 2003 14:54:58 -0800 (PST)
From: [email-masked]
Subject: Re: mail vs. news ???
To: Bruce Lilly <[email-masked]>
Cc: Keith Moore <[email-masked]>, Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>, [email-masked]
Sender: [email-masked]

> Keith Moore wrote:
> >>There also appears to be precedent for such transport limits
> >>finding their way into the message format specification. RFC 822
> >>had no line length limit; the 998 octet limit in 2822 sections
> >>2.1.1, 2.3, and 3.5 appears to be derived from the SMTP transport
> >>limitation.
> >
> >
> > yes, but that one is completely necessary.  if you can't transport
> > a rfc 2822 message in SMTP then it isn't likely to go very far.

> Practically speaking, true, though there *are* other transport
> mechanisms (e.g. UUCP) for which it matters not.  And I have
> received messages via SMTP with longer lines than 998 octets.

> > not that it's a bad idea to bound the length of message-ids, but
> > using this as excuse to say that message-ids are bound by the SMTP
> > limit on address length is quite a stretch.

> That was not my intent. I intended to point out that
> 1. there exists precedent for transport limits being introduced
>     into the message format, so in principle that should apply
>     to a msg-id length limit.
> 2. It would be more convenient, IMO, to have a uniform limit
>     for SMTP Path (a.k.a. angle-addr) and for msg-id rather than
>     two close but different limits. The comparison was by way of
>     pointing out the historical identity of the constructs (which
>     now differs w.r.t. CFWS), not as a justification.  If there's
>     a reason why 250 works for NNTP and 256 doesn't, that's another
>     matter.

While I do think that moving unlimited length msg-ids to the accept
syntax and limiting them in the generate syntax is a good idea, it does
raise the issue of what one does with overly long ids when generating a new
message. Discarding them is probably the cleanest solution.

				Ned

Date: Sun, 23 Feb 2003 14:40:45 -0800 (PST)
From: [email-masked]
Subject: Re: mail vs. news ???
To: Bruce Lilly <[email-masked]>
Cc: Dan Kohn <[email-masked]>, Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Sender: [email-masked]

> Dan Kohn wrote:

> > I'm working on a new version of Kohn draft that captures more of the
> > semantics from the Lindsey draft about news-specific restrictions.  I
> > believe laying them out against the context of RFC 2822 (rather than
> > expecting the implementer to compare the documents) will produce the
> > best interoperability.

> If our rechartered goal is still to produce a Standards Track RFC,
> there may be an issue regarding use of RFC 2822 (which I am informed
> is incorrectly indicated as Standards Track and that it does not in
> fact obsolete RFC 822 as stated in the rfc-index).

RFC 2822 does not and cannot obsolete RFC 822 at present, for the simple reason
that RFC 822 is a full standard whereas RFC 2822 is only a proposed standard.
The intent, however, is for RFC 2822, or more likely a revised version of RFC
2822, to obsolete RFC 822 once it reaches standard status.

This is a reasonably straightforward situation, but a bit more than an entry in
the RFC index is capable of expressing.

> If so, I believe
> the approach used in RFC 3282 (which says it's Standards Track), viz.
> supplying both RFC 822 EBNF and RFC 2822 ABNF may be a solution (the
> understanding being that they intend to specify the same restrictions).

Any I-D that is intended to become a proposed standard need only refer to RFC
2822; referring to RFC 822 in a normative way is a bad idea at this point.
(The situation for an I-D intended for draft is different -- it should be
obvious why -- but that's not an issue USEFOR has to deal with.)

> As for ambiguities in the MIME documents, while Charles' intent in
> clarifying those is laudable, it goes too far in the direction of
> repeating a lot of nitty-gritty detail.  How far we go in clarifying
> those issues is something where our area director may be able to
> provide some guidance, as he is in a position to know how soon there
> might be a rewrite of the MIME documents themselves which would
> remove the ambiguities.

Well, not to put too fine a point on it, but these endless squabbles are at
least part of the reason why a MIME update of anything besides part 4 is
unlikely to get done by the MIME editor in the near future ;-)

Regardless, if there are outright errors in MIME they should be corrected by
RFC Errata. Reiterating material from one standard in another is a bad idea
IMO; and you may rest assured I will push back on this practice should a
document containing such material ever make it to me for review.

> On a related note, IMO there should be a reference in the Status
> of This Memo section pointing to the errata page at
> http://www.rfc-editor.org/errat.html
> That of course isn't applicable to a draft, but when the draft
> moves to a published RFC, it is invaluable to implementors (the
> relative obscurity of that page is probably one of the reasons that
> some implementors have made errors).

The fact that the page hasn't existed for very long is a more likely
explanation.

> In reference to the use of
> MIME, it might be prudent to repeat the pointer to the errata page,
> as there are a number of errata for RFCs 2046, 2047, 2231, and for
> that matter 1036 (though I sincerely hope the "Re: " hack will be
> discarded).

> > In general, I'm thinking about suggesting a different tact from RFC
> > 2822, which said that Section 3 current syntax is MUST generate and
> > Section 4 obsolete syntax is MUST accept.

> To be a little more precise, the obs- syntax is MUST accept, MUST NOT
> generate. The non-obs- syntax still provides a great deal of leeway.
> Perhaps unnecessarily so; the current msg-id discussions are timely
> (I believe Pete Resnick stated on ietf-822 a few months ago that
> he is collecting comments etc. re 2822 in preparation for the next
> step in its progress) and appropriate.

Agreed. I would not be at all adverse to placing a limit on the size of message
ids.

				Ned

Date: Sun, 23 Feb 2003 17:51:13 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: usenet-format <[email-masked]>, [email-masked]
Organization: Bruce Lilly
To: Dan Kohn <[email-masked]>
CC: usenet-format <[email-masked]>, [email-masked]
Subject: Re: Tightening RFC 2822 msg-IDs
Sender: [email-masked]

Dan Kohn wrote:

> Could someone suggest the RFC 2822 syntax that would be compliant with
> RFC 1036 and established practice (i.e., shorter than 250 characters and
> doesn't include backslash-escaped space or control characters)?

I believe that was <= 250, and '>' was also excluded from the msg-id
interior.

msg-id          =       [CFWS] "<" id-left "@" id-right ">" [CFWS] ; maximum length 250 octets excluding CFWS

id-left         =       dot-atom-text / no-fold-quote / obs-id-left

id-right        =       dot-atom-text / no-fold-literal / obs-id-right

no-fold-quote   =       DQUOTE *(qtext / printable-quoted-pair) DQUOTE

no-fold-literal =       "[" *(dtext / printable-quoted-pair) "]"

printable-quoted-pair = ("\" pqtext) / obs-qp

pqtext          =       %d33-61 /         ; Printable characters excluding ">"
                         %d63-126 /
                         obs-text

dtext           =       DIGIT /
                         "A" / "B" / "C" / "D" / "E" / "F" / ; hexadecimal digits, case-insensitive
                         "." /                               ; IPv4 separator
                         ":"                                 ; IPv6 separator

Notes:
1. I have left optional CFWS around the msg-id, though that appears
    to be an issue with some news software. It is easily refined if
    necessary ("ownership" of CFWS is a remaining ambiguity in 2822).
2. dot-atom-text syntax is also used in dot-atom, and isn't itself
    readily amenable to specification of a length limit, even if one
    could ignore the impact on other constructs
3. there is no attempt to impose a separate left-side length limit
4. '>' is ecluded from the msg-id interior per 1036 sect. 2.1.5.
    To restore it pqtext becomes the single range %d33-126.
5. obs- syntax change is limited to the maximum length restriction.
6. dtext is limited to strictly what is necessary.

Date: Sun, 23 Feb 2003 14:42:23 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
cc: Keith Moore <[email-masked]>
Subject: Re: mail vs. news ???
Organization: Networks and Distributed Computing
Sender: [email-masked]

On Sun, 23 Feb 2003, Bruce Lilly wrote:
> 1. there exists precedent for transport limits being introduced
>     into the message format, so in principle that should apply
>     to a msg-id length limit.

I think that we have loud agreement here.  A message-id length limit is a
reasonable thing to go into a Messaging Implementation Recommendations
document.  It should not be used to justify news having its own message
format.

This business about spaces not being optional after the colon is silly, as
are similar such "differences".  Yes, almost all messages have LWSP after
the colon.  But a programming mistake, particularly in some version of rn
or equivalent dinoware, is even less of a justification for a separate
message format than a length limit.

Nonetheless, even silly things like this may go into an implementation
recommendations document anyway.  It's alright to document that a widely
used application has a baby programmer mistake which is triggered by
such-and-such, so don't do such-and-such.

> 2. It would be more convenient, IMO, to have a uniform limit
>     for SMTP Path (a.k.a. angle-addr) and for msg-id rather than
>     two close but different limits.

SMTP addresses and message-ids may use a more or less equivalent syntax,
but I think that it's stretching it to say that they are the same.  That
type of logical stretch is the sort of argument that tends to be punished
severely by the law of unintended consequences.

Your argument is stronger without this unnecessary addition.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: Sun, 23 Feb 2003 17:12:29 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>, [email-masked]
Subject: Re: mail vs. news ???
Sender: [email-masked]

Keith Moore wrote:
>>There also appears to be precedent for such transport limits
>>finding their way into the message format specification. RFC 822
>>had no line length limit; the 998 octet limit in 2822 sections
>>2.1.1, 2.3, and 3.5 appears to be derived from the SMTP transport
>>limitation.
> 
> 
> yes, but that one is completely necessary.  if you can't transport
> a rfc 2822 message in SMTP then it isn't likely to go very far.

Practically speaking, true, though there *are* other transport
mechanisms (e.g. UUCP) for which it matters not.  And I have
received messages via SMTP with longer lines than 998 octets.

> not that it's a bad idea to bound the length of message-ids, but
> using this as excuse to say that message-ids are bound by the SMTP
> limit on address length is quite a stretch.

That was not my intent. I intended to point out that
1. there exists precedent for transport limits being introduced
    into the message format, so in principle that should apply
    to a msg-id length limit.
2. It would be more convenient, IMO, to have a uniform limit
    for SMTP Path (a.k.a. angle-addr) and for msg-id rather than
    two close but different limits. The comparison was by way of
    pointing out the historical identity of the constructs (which
    now differs w.r.t. CFWS), not as a justification.  If there's
    a reason why 250 works for NNTP and 256 doesn't, that's another
    matter.

Date: Sun, 23 Feb 2003 16:54:39 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Dan Kohn <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Dan Kohn wrote:

> I'm working on a new version of Kohn draft that captures more of the
> semantics from the Lindsey draft about news-specific restrictions.  I
> believe laying them out against the context of RFC 2822 (rather than
> expecting the implementer to compare the documents) will produce the
> best interoperability.

If our rechartered goal is still to produce a Standards Track RFC,
there may be an issue regarding use of RFC 2822 (which I am informed
is incorrectly indicated as Standards Track and that it does not in
fact obsolete RFC 822 as stated in the rfc-index).  If so, I believe
the approach used in RFC 3282 (which says it's Standards Track), viz.
supplying both RFC 822 EBNF and RFC 2822 ABNF may be a solution (the
understanding being that they intend to specify the same restrictions).

As for ambiguities in the MIME documents, while Charles' intent in
clarifying those is laudable, it goes too far in the direction of
repeating a lot of nitty-gritty detail.  How far we go in clarifying
those issues is something where our area director may be able to
provide some guidance, as he is in a position to know how soon there
might be a rewrite of the MIME documents themselves which would
remove the ambiguities.

On a related note, IMO there should be a reference in the Status
of This Memo section pointing to the errata page at
http://www.rfc-editor.org/errat.html
That of course isn't applicable to a draft, but when the draft
moves to a published RFC, it is invaluable to implementors (the
relative obscurity of that page is probably one of the reasons that
some implementors have made errors).  In reference to the use of
MIME, it might be prudent to repeat the pointer to the errata page,
as there are a number of errata for RFCs 2046, 2047, 2231, and for
that matter 1036 (though I sincerely hope the "Re: " hack will be
discarded).

> In general, I'm thinking about suggesting a different tact from RFC
> 2822, which said that Section 3 current syntax is MUST generate and
> Section 4 obsolete syntax is MUST accept.

To be a little more precise, the obs- syntax is MUST accept, MUST NOT
generate. The non-obs- syntax still provides a great deal of leeway.
Perhaps unnecessarily so; the current msg-id discussions are timely
(I believe Pete Resnick stated on ietf-822 a few months ago that
he is collecting comments etc. re 2822 in preparation for the next
step in its progress) and appropriate.

 > Instead, I'm thinking of
> suggesting that agents SHOULD generate messages that honor current news
> restrictions (e.g., no Received headers, tighter Message IDs, SP after
> colon in all headers), while saying that agents compliant with the spec
> MUST accept RFC 2822 syntax.  That way, implmenters know what to do to
> maximize the likelihood that their messages will be processed, but
> there's also a push toward supporting a universal Internet Message
> Format syntax across all messaging protocols.
> 
> Of course, once the uniquely news syntax is specified clearly (i.e.,
> against the context of RFC 2822), the WG (assuming it's rechartered) can
> hash out all the MAYs, SHOULDs, and MUSTs.

For backwards compatibility, conforming to the more restrictive 1036
syntax, along with the additional restrictions which 2822 imposed,
should be a MUST for generation of new articles (i.e. don't send
unexpected cruft to existing implementations, and don't send obsolete
cruft either). Some of the 2822 obs- syntax was never legal in news
articles, so mandating acceptance of the full complex obs- syntax
might be construed as an unnecessary burden.  Note, however, as
mentioned before on Usefor, at least one of the obs- constructs
(unquoted dot in a phrase) is intended to be a furture relaxation of
the syntax, so the whole obs- syntax area will require careful
review.  For clarity to implementors, we should make it quite clear
that to the extent possible, convergence of the article format and
general message format is a goal (e.g. the SP-after-colon requirement
may be removed in a future revision, implementors should not assume
the presense of a space after colon).

Received fields deserve a bit of discussion. Carrying them through
mail->news->mail via gateways would probably be a bad idea, as
some mail transport software counts Received fields as part of
loop detection.  I believe they are elided by most, if not all,
mail->news gateways, and that seems reasonable.

That raises a related issue; we should define a set of "trace"
fields for news transport (that would of course include Path)
as 2822 has some handling requirements (sections 3.6, 3.6.6, 3.6.7)
regarding them.

Date: Sun, 23 Feb 2003 16:35:58 -0500
From: Lawrence Greenfield <[email-masked]>
To: [email-masked], "usenet-format" <[email-masked]>, Dan Kohn <[email-masked]>
Subject: Re: Tightening RFC 2822 msg-IDs
Sender: [email-masked]

   From: "Dan Kohn" <[email-masked]>
   Precedence: bulk

   It has been suggested that some legal but unused RFC 2822 Message-ID's
   will break a significant amount of deployed news infrastructure.  If, in
   fact, we're not able to find any existing software that generates such
   "dangerous" Message-IDs, what would ietf-822 folks think about
   tightening the legal-to-generate Message-ID definition when RFC 2822
   moves to draft standard?  This would encourage the trend toward using
   RFC 2822 as a universal Internet Messaging Format document, which can be
   used across mail and news.

I believe having a Usenet format document that updates RFC 2822 on the
standards track would be a very good thing, especially if all of the
changes are like messageid: of the form of moving more syntax into the
obsolete section and further restricting what is generated.

Larry

To: [email-masked]
Cc: [email-masked]
Subject: Re: mail vs. news ???
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sun, 23 Feb 2003 12:47:18 -0800
Lines: 18
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:

> Since it is unchanged from RFC 822, that's pretty hard to pin on 2822.
> If the 1036 restrictions need to remain in effect until software authors
> have had a chance to make revisions, fine.

RFC 822's References specificaction is useless junk.  RFC 2822 is the
first mail standard with a useful specification of References, taken
almost entirely from Usenet (and from mail clients copying existing
practice on Usenet).

I think the issue here is comments, yes?  If so, I think news software
should ideally learn to cope, although I have no idea why one would
intentionally put comments into the References header other than just
because one can.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sun, 23 Feb 2003 14:41:04 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: mail vs. news ???
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Ken Murchison <[email-masked]> writes:

>>- usenet puts a greater restriction on the headers (although still being
>>RFC 2822 compliant)

> There are a whole lot of things required by RFC 2822 that just do not work
> in news. Some examples:
> 
> 1. You can't omit that SP after the ':' in a header.

"Omitting" it is *not* _required_ by RFC 2822, nor is that any
different from RFC 1036s restriction re. 822, as noted by Ken.

> 2. The RFC 2822 syntax of the References-header breaks many threading user
> agents.

Since it is unchanged from RFC 822, that's pretty hard to pin
on 2822.  If the 1036 restrictions need to remain in effect
until software authors have had a chance to make revisions,
fine.

> 3. Comments in unexpected places will break some existing software.

The basis for 1036, viz. 822, allows comments essentially anywhere
in any structured field body.  If 1036 didn't specify an additional
restriction, no such places are "unexpected", and such "existing
software" is broken.

> 4. There are some features of the Message-ID which, if strictly followed,
> would totally break the news relaying mechanisms or which, if we were
> forced to accomodate them, would impose unacceptable performance
> penalties. Of course, these quirky cases never arise in real life, but
> theoretically they could.

Message IDs are being discussed in detail.

content-class: urn:content-classes:message
Subject: Tightening RFC 2822 msg-IDs
Importance: normal
Date: Sun, 23 Feb 2003 11:29:17 -0800
From: "Dan Kohn" <[email-masked]>
To: "usenet-format" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

It has been suggested that some legal but unused RFC 2822 Message-ID's
will break a significant amount of deployed news infrastructure.  If, in
fact, we're not able to find any existing software that generates such
"dangerous" Message-IDs, what would ietf-822 folks think about
tightening the legal-to-generate Message-ID definition when RFC 2822
moves to draft standard?  This would encourage the trend toward using
RFC 2822 as a universal Internet Messaging Format document, which can be
used across mail and news.

Could someone suggest the RFC 2822 syntax that would be compliant with
RFC 1036 and established practice (i.e., shorter than 250 characters and
doesn't include backslash-escaped space or control characters)?  Could
other people check their archives of messages to see how many extant
Message-IDs are legal under RFC 2822 Section 3 syntax but violate news
syntax?

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Sun, 23 Feb 2003 14:22:18 -0500
From: Keith Moore <[email-masked]>
To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: mail vs. news ???
Sender: [email-masked]

> There also appears to be precedent for such transport limits
> finding their way into the message format specification. RFC 822
> had no line length limit; the 998 octet limit in 2822 sections
> 2.1.1, 2.3, and 3.5 appears to be derived from the SMTP transport
> limitation.

yes, but that one is completely necessary.  if you can't transport
a rfc 2822 message in SMTP then it isn't likely to go very far.

not that it's a bad idea to bound the length of message-ids, but
using this as excuse to say that message-ids are bound by the SMTP
limit on address length is quite a stretch.

Keith

To: [email-masked]
Xref: clerew local.mime:2429
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: yEnc (was RE: mail vs. news ???)
Date: Sun, 23 Feb 2003 16:36:07 GMT
Lines: 37
Sender: [email-masked]

In <[email-masked]> "Eric A. Hall" <[email-masked]> writes:

>on 2/21/2003 4:25 PM Dan Kohn wrote:

>> Note that there's absolutely no reason why yEnc (perhaps with optional
>> gzip functionality added in) can't be specified as a new
>> Content-Transport-Encoding as an alternative to base64.  Further, if
>> message/partial doesn't do the job, than a new application/news-partial
>> could be specified.

>Unfortunately there are loads of issues that make this whole thing very
>complex. Its been tackled at least a few times, with the result typically
>being that everybody is too exhausted to continue discussing it.

Yes, I have been round this before. It is clear that, even if the Yenc
people were to use
    Content-Transfer-Encoding: x-yenc
which they could start doing tomorrow, things would be much simpler.

I have tried to persuade the relevant people to produce an internet-draft
proposing it as an official CTE. But they don't want to know. They seem
convinced that the IETF is totally against all change, and that they would
be heavily sat upon, and that the whole process would be far too
cumbersome and time consuming, and "hey it works fine already, we don't
need no stinkin' IETF". All very sad, but that's how some people regard
the IETF. So we are stuck with indicating the presence of yEnc by a flag
in the Subject header. It is a mess.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2430
Newsgroups: local.mime,local.usenet
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: mail vs. news ???
Date: Sun, 23 Feb 2003 16:47:53 GMT
Lines: 44
Sender: [email-masked]

In <[email-masked]> Ken Murchison <[email-masked]> writes:

>I find it interesting, if not disturbing, that some members of the
>usenet community seem to think that mail messages and usenet articles
>are not the same thing.  AFAICT, from reading the relevant standards,
>writing server code for SMTP/LMTP/IMAP/POP3/NNTP, and everyday use, mail
>messages and news articles both conform to RFC 2822 (RFC 1036 states as
>much).  The only differences that I'm aware of are the following:

>- usenet puts a greater restriction on the headers (although still being
>RFC 2822 compliant)
>- mail messages are typically tranmitted over a 1-to-1 protocol (SMTP)
>and news articles are typically transmitted over a 1-to-many protocol
>(NNTP)

There are a whole lot of things required by RFC 2822 that just do not work
in news. Some examples:

1. You can't omit that SP after the ':' in a header.

2. The RFC 2822 syntax of the References-header breaks many threading user
agents.

3. Comments in unexpected places will break some existing software.

4. There are some features of the Message-ID which, if strictly followed,
would totally break the news relaying mechanisms or which, if we were
forced to accomodate them, would impose unacceptable performance
penalties. Of course, these quirky cases never arise in real life, but
theoretically they could.

The Usefor draft tries to correct a lot of these discrepancies, but it is
full of "you MUST accept this, but you SHOULD NOT generate it yet".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: Usefor Mailing List <[email-masked]>
Cc: IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 23 Feb 2003 15:48:53 +0000
Lines: 34
Sender: [email-masked]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 >> it is not, in fact, merely an NNTP issue; message-ids are a key
 >> part of _every_ news transport method (since they are used to
 >> detect duplicates).  Every news server has to keep a history
 >> database (or the moral equivalent) indexed by message-id, whether
 >> it uses NNTP or not.  Message-ids are passed around a lot
 >> internally, stored in queue files, listed in ihave/sendme control
 >> messages for UUCP feeding, appear as parameters in control
 >> messages, and generally pervade the entire structure of news;
 >> whitespace or excessive length breaks things in really quite a lot
 >> of places.

 Bruce> Andrew, you mentioned that the control characters and '>'
 Bruce> which can appear (backslash-quoted inside a no-fold-quote) in
 Bruce> a legal 2822 msg-id are problematic.  Are those NNTP issues or
 Bruce> implementation issues?

'>' is an implementation issue, in that probably the vast majority of
existing implementations take advantage of the RFC1036 restrictions on
message-ids to parse them simply by looking for the closing >.
(There are also implementations that treat '<' in message-ids as
invalid; I do not know why this is.)

NNTP itself does not have any problems with '>' that I know of.

I do not know the extent to which control characters other than TAB,
CR, LF (and NUL of course) actually cause any issues at the transport
level, though I expect some clients would have an interesting time
with them. I don't recall it coming up as an operational issue, unlike
ids with spaces or excessively long ids.

-- 
Andrew.

Date: Sun, 23 Feb 2003 10:22:46 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: Mark Crispin <[email-masked]>, Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Andrew Gierth wrote:
>>>>>>"Mark" == Mark Crispin <[email-masked]> writes:

> The length limitation of 250 is from existing software rather than
> from previous standards (NNTP restricts the total length of a
> command-line to 512 including the CRLF). 250 is a better choice

A limit identical to that imposed by other common transports
might be more appropriate.  For example, RFC 822 used the
same syntax for msg-id as for angle-brscketed addresses:

      msg-id      =  "<" addr-spec ">"            ; Unique message id

and SMTP does have a 256 octet limit on such addresses in
SMTP commands (RFC 821 sect. 4.5.3, RFC 2821 sect. 4.5.3.1).
IMO, a *uniform* limit would be preferable to one arbitrary
limit in one place and a different arbitrary limit in another.
[the same sections of the two SMTP RFCs also impose separate
length limits on the local-part]

>  Mark> If NNTP has a limitation such as suggested above, then that
>  Mark> would add a transport imposed requirement to a document that
>  Mark> otherwise uses RFC 2822 and MIME as normative.  That is
>  Mark> something that can be respected.

There also appears to be precedent for such transport limits
finding their way into the message format specification. RFC 822
had no line length limit; the 998 octet limit in 2822 sections
2.1.1, 2.3, and 3.5 appears to be derived from the SMTP transport
limitation.

>  Mark> That is different from duplicating all of RFC 2822 and MIME
>  Mark> just to insert transport imposed requirements.  Doing so, which
>  Mark> the Usefor document has done, is ludicrous.

I believe that the motivation was introduction of raw utf-8, rather
than transport requirements.  Yes, it's ludicrous -- better to
address any desired extensions in a general manner, and likewise
for ambiguities in the MIME documents.  The appropriate mechanism
is the usual evolution of the relevant standards; that way any
potential incompatibilities will quickly become apparent and can
be communicated and resolved. That is certainly preferable to
a thinly-cloaked attempt to force incompatibilities on
interoperating protocols.

> it is not, in fact, merely an NNTP issue; message-ids are a key part
> of _every_ news transport method (since they are used to detect
> duplicates).  Every news server has to keep a history database (or the
> moral equivalent) indexed by message-id, whether it uses NNTP or not.
> Message-ids are passed around a lot internally, stored in queue files,
> listed in ihave/sendme control messages for UUCP feeding, appear as
> parameters in control messages, and generally pervade the entire
> structure of news; whitespace or excessive length breaks things in
> really quite a lot of places.

Andrew, you mentioned that the control characters and '>' which
can appear (backslash-quoted inside a no-fold-quote) in a legal
2822 msg-id are problematic.  Are those NNTP issues or implementation
issues? [for the moment, let's ignore the plethora of issues that
theoretically could arise similarly in a no-fold-literal (which
appears to be future-prediction on the part of 2822); let's
assume that the RHS is either a domain name consisting of only
letters, digits, and hyphen or is a valid IPv4 or IPv6 IP literal]

MBOX-Line: From [email-masked]  Sat Feb 22 14:26:22 2003
Date: 22 Feb 2003 23:10:00 +0200
From: [email-masked] (Kai Henningsen)
Old-To: [email-masked]
Cc: [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Status: RO
To: [email-masked]
Sender: [email-masked]

[email-masked]  wrote on 21.02.03 in <[email-masked]>:

> Part of the problem with having a group take so long to complete its work is
> that what's minimally acceptable to the IETF/IESG changes over time. It used
> to be that protocol could just use some encoding of Unicode, end of story.
> No more. Now that stringprep exists the IESG requires it be used when
> Unicode protocol elements are involved. Several recent specifications have
> been returned to their respective working groups because they didn't do
> this.
>
> I note in passing that while the current News Article Format Draft talks
> about normalization of group names, it does so without reference to
> stringprep. This would need to change: Stringprep covers stuff besides
> normalization, and more generally provides a checklist for Unicode usage
> each protocol needs to consider.

That's probably because stringprep wasn't even known to many people on  
USEFOR, and certainly that requirement wasn't.

Stringprep actually solves a number of problems that some people here  
didn't want to tackle, IIRC. That is, IMO, stringprep is a good thing.

However, I note that the IANA stringprep profile registry is currently  
*empty* (except for reference to a certain RFC NNNN [that is, reference to  
an unnamed draft] for IDN).

> I also note that Dan Kohn's alternative News Article Format draft uses
> punycode, which already includes a stringprep profile.

Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the  
current one, yes?). We obviously can't use a profile that says it must be  
used together with extra restrictions specific to IDN, but otherwise it  
seems that we could mostly just copy this one ... except shouldn't these  
profiles be reusable?

It's hard to figure this out by looking at current registrations (as  
recommended by the stringprep RFC, 2454) if there aren't any.

MfG Kai

MBOX-Line: From [email-masked]  Sat Feb 22 09:55:07 2003
Date: Sat, 22 Feb 2003 09:41:54 -0800 (PST)
From: Mark Crispin <[email-masked]>
Old-To: Andrew Gierth <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Organization: Networks and Distributed Computing
Status: O
To: [email-masked]
Sender: [email-masked]

On Sat, 22 Feb 2003, Andrew Gierth wrote:
> Mail systems do not appear to make any actual use of the message-id
> header other than for logging (and internally in some odd systems).

Huh?  Message-ID, In-Reply-To, and References are heavily used in mail for
threading.  IMAP even has a THREAD facility that stipulates how to use
these.

> None of SMTP, POP3 or IMAP ever use message-ids as protocol
> parameters.  News, on the other hand, uses message-ids as protocol
> parameters _all the time_, both for readers accessing messages and for
> server-to-server transfer.

Let's try to translate your comment into something useful.  Is it your
contention that:
	Some set of commands in NNTP use message-id as an argument.
	The syntax for that command precludes the use of space in
	that argument, and/or NNTP lacks the type of quoting mechanism
	found in IMAP and SMTP.

If so, please provide the details.  If confirmed, that is useful and can
be stated as a requirement.  It isn't clear to me that it's a requirement
for header syntax; rather, it's a transport imposed requirement.  The
effect is the same (you can't use message-ids with spaces in news), but
it's stated in the right place.

Note that there is quite a bit in RFC 2822 that is not permissible in mail
due to transport imposed requirements.  There was even more in RFC 822.
That doesn't mean that mail can not use RFC 2822.

> (and, FWIW, I checked about a hundred thousand message-ids taken from
> mail messages here, and only one of those had whitespace in and that
> was a Chinese spam which had four spaces in place of the domain-part,
> almost certainly due to misconfigured or broken spamming software.

That isn't an example of a syntactically valid RFC 2822 message-id.

>  Mark> And the IETF/IESG is supposed to respect this?
> yes

If NNTP has a limitation such as suggested above, then that would add a
transport imposed requirement to a document that otherwise uses RFC 2822
and MIME as normative.  That is something that can be respected.

That is different from duplicating all of RFC 2822 and MIME just to insert
transport imposed requirements.  Doing so, which the Usefor document has
done, is ludicrous.  What's worse, it has the highly undesirable effect of
cloaking the transport-imposed requirements.

A small document, such as Kohn's draft, which has a section of "NNTP
transport requirements" would show these requirements in clear contrast.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

MBOX-Line: From [email-masked]  Sat Feb 22 15:03:05 2003
Old-To: [email-masked] (Kai Henningsen)
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Simon Josefsson <[email-masked]>
Date: Sat, 22 Feb 2003 23:46:06 +0100
Status: O
To: [email-masked]
Sender: [email-masked]

[email-masked] (Kai Henningsen) writes:

>> I also note that Dan Kohn's alternative News Article Format draft uses
>> punycode, which already includes a stringprep profile.
>
> Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the  
> current one, yes?). We obviously can't use a profile that says it must be  
> used together with extra restrictions specific to IDN, but otherwise it  
> seems that we could mostly just copy this one ... except shouldn't these  
> profiles be reusable?

Yes.  Using nameprep for usefor is a bad idea, but the contents of
nameprep can be copied, renamed, polished, and then, IMHO, it would
turn into a good idea.

> It's hard to figure this out by looking at current registrations (as  
> recommended by the stringprep RFC, 2454) if there aren't any.

There are a few drafts with stringprep profile, here are links to
those I'm familiar with.  Note that most of them contain some minor
technical problems.

http://www.ietf.org/internet-drafts/draft-ietf-idn-nameprep-11.txt
http://www.ietf.org/internet-drafts/draft-ietf-xmpp-nodeprep-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-xmpp-resourceprep-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-sasl-anon-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-string-prep-03.txt

MBOX-Line: From [email-masked]  Sat Feb 22 11:55:18 2003
Date: Sat, 22 Feb 2003 11:31:03 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: Re: mail vs. news ???
Old-To: Russ Allbery <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Status: O
To: [email-masked]
Sender: [email-masked]

On Sat, 22 Feb 2003 11:05:19 -0800, Russ Allbery wrote:
> > 	Some set of commands in NNTP use message-id as an argument.
> > 	The syntax for that command precludes the use of space in
> > 	that argument, and/or NNTP lacks the type of quoting mechanism
> > 	found in IMAP and SMTP.
> Your statement above is correct.

If this is the case, then we add a small section to the Kohn document of the
form:

NNTP Transport Requirements

   The BLURDYBLOOP command in NNTP uses the message-id as an argument.
   The syntax for this command does not provide any way for spaces to
   appear in the message-id.  Therefore, message-ids in the Message-ID,
   In-Reply-To, and References headers of a message MUST NOT contain
   the space character.

Implementation Recommendations

   Some widely-used NNTP implementations are known to core dump, reformat
   reformat the hard drive, and watch Joe Millionaire on TV if given a Date:
   header containing a symbolic timezone in a coment, e.g.
	Date: Sat, 22 Feb 2003 10:26:00 -0800 (PST)
   It is recommended that such comments be removed prior to transmitting
   a message via NNTP.

etc. etc. as needed.

MBOX-Line: From [email-masked]  Sat Feb 22 11:42:52 2003
Date: Sat, 22 Feb 2003 11:31:03 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: Re: mail vs. news ???
Old-To: Russ Allbery <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Status: O
To: [email-masked]
Sender: [email-masked]

On Sat, 22 Feb 2003 11:05:19 -0800, Russ Allbery wrote:
> > 	Some set of commands in NNTP use message-id as an argument.
> > 	The syntax for that command precludes the use of space in
> > 	that argument, and/or NNTP lacks the type of quoting mechanism
> > 	found in IMAP and SMTP.
> Your statement above is correct.

If this is the case, then we add a small section to the Kohn document of the
form:

NNTP Transport Requirements

   The BLURDYBLOOP command in NNTP uses the message-id as an argument.
   The syntax for this command does not provide any way for spaces to
   appear in the message-id.  Therefore, message-ids in the Message-ID,
   In-Reply-To, and References headers of a message MUST NOT contain
   the space character.

Implementation Recommendations

   Some widely-used NNTP implementations are known to core dump, reformat
   reformat the hard drive, and watch Joe Millionaire on TV if given a Date:
   header containing a symbolic timezone in a coment, e.g.
	Date: Sat, 22 Feb 2003 10:26:00 -0800 (PST)
   It is recommended that such comments be removed prior to transmitting
   a message via NNTP.

etc. etc. as needed.

MBOX-Line: From [email-masked]  Sat Feb 22 13:37:57 2003
Date: Sat, 22 Feb 2003 16:37:23 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
Old-To: Usefor Mailing List <[email-masked]>
Cc: IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Status: O
To: [email-masked]
Sender: [email-masked]

Andrew Gierth wrote:
> 
> >>>>> "Dave" == Dave Crocker <[email-masked]> writes:
> 
>  AG> Mail systems do not appear to make any actual use of the
>  AG> message-id header other than for logging (and internally in some
>  AG> odd systems).  None of SMTP, POP3 or IMAP ever use message-ids as
>  AG> protocol parameters.
> 
>  Dave> Message-ids are not nearly as deeply and usefully and broadly
>  Dave> used as we had expected them to be.  However, they ARE used, to
>  Dave> good effect, for duplicate detection and for thread
>  Dave> construction.
> 
> at the layer of actual message transport, or just in the clients or
> client-side services such as IMAP threading? I really wasn't referring
> to the clients when I mentioned "mail systems".
> 
> I have heard of mail transports that assume a 1-1 correspondence
> between message-ids and body content but I believe this to be highly
> unusual; please correct me if I am mistaken on this point.

FYI, the LMTP daemon in Cyrus IMAP uses the message-id for duplicate
suppression (and to prevent mail loops) at the time of delivery.  This
isn't done as part of the protocol as in NNTP, but its done internally
for every message before its committed to disk.

MBOX-Line: From [email-masked]  Sat Feb 22 09:42:38 2003
Date: Sat, 22 Feb 2003 09:41:54 -0800 (PST)
From: Mark Crispin <[email-masked]>
Old-To: Andrew Gierth <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Organization: Networks and Distributed Computing
Status: O
To: [email-masked]
Sender: [email-masked]

On Sat, 22 Feb 2003, Andrew Gierth wrote:
> Mail systems do not appear to make any actual use of the message-id
> header other than for logging (and internally in some odd systems).

Huh?  Message-ID, In-Reply-To, and References are heavily used in mail for
threading.  IMAP even has a THREAD facility that stipulates how to use
these.

> None of SMTP, POP3 or IMAP ever use message-ids as protocol
> parameters.  News, on the other hand, uses message-ids as protocol
> parameters _all the time_, both for readers accessing messages and for
> server-to-server transfer.

Let's try to translate your comment into something useful.  Is it your
contention that:
	Some set of commands in NNTP use message-id as an argument.
	The syntax for that command precludes the use of space in
	that argument, and/or NNTP lacks the type of quoting mechanism
	found in IMAP and SMTP.

If so, please provide the details.  If confirmed, that is useful and can
be stated as a requirement.  It isn't clear to me that it's a requirement
for header syntax; rather, it's a transport imposed requirement.  The
effect is the same (you can't use message-ids with spaces in news), but
it's stated in the right place.

Note that there is quite a bit in RFC 2822 that is not permissible in mail
due to transport imposed requirements.  There was even more in RFC 822.
That doesn't mean that mail can not use RFC 2822.

> (and, FWIW, I checked about a hundred thousand message-ids taken from
> mail messages here, and only one of those had whitespace in and that
> was a Chinese spam which had four spaces in place of the domain-part,
> almost certainly due to misconfigured or broken spamming software.

That isn't an example of a syntactically valid RFC 2822 message-id.

>  Mark> And the IETF/IESG is supposed to respect this?
> yes

If NNTP has a limitation such as suggested above, then that would add a
transport imposed requirement to a document that otherwise uses RFC 2822
and MIME as normative.  That is something that can be respected.

That is different from duplicating all of RFC 2822 and MIME just to insert
transport imposed requirements.  Doing so, which the Usefor document has
done, is ludicrous.  What's worse, it has the highly undesirable effect of
cloaking the transport-imposed requirements.

A small document, such as Kohn's draft, which has a section of "NNTP
transport requirements" would show these requirements in clear contrast.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

MBOX-Line: From [email-masked]  Sat Feb 22 14:40:10 2003
Date: 22 Feb 2003 23:10:00 +0200
From: [email-masked] (Kai Henningsen)
Old-To: [email-masked]
cc: [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Status: RO
To: [email-masked]
Sender: [email-masked]

[email-masked]  wrote on 21.02.03 in <[email-masked]>:

> Part of the problem with having a group take so long to complete its work is
> that what's minimally acceptable to the IETF/IESG changes over time. It used
> to be that protocol could just use some encoding of Unicode, end of story.
> No more. Now that stringprep exists the IESG requires it be used when
> Unicode protocol elements are involved. Several recent specifications have
> been returned to their respective working groups because they didn't do
> this.
>
> I note in passing that while the current News Article Format Draft talks
> about normalization of group names, it does so without reference to
> stringprep. This would need to change: Stringprep covers stuff besides
> normalization, and more generally provides a checklist for Unicode usage
> each protocol needs to consider.

That's probably because stringprep wasn't even known to many people on  
USEFOR, and certainly that requirement wasn't.

Stringprep actually solves a number of problems that some people here  
didn't want to tackle, IIRC. That is, IMO, stringprep is a good thing.

However, I note that the IANA stringprep profile registry is currently  
*empty* (except for reference to a certain RFC NNNN [that is, reference to  
an unnamed draft] for IDN).

> I also note that Dan Kohn's alternative News Article Format draft uses
> punycode, which already includes a stringprep profile.

Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the  
current one, yes?). We obviously can't use a profile that says it must be  
used together with extra restrictions specific to IDN, but otherwise it  
seems that we could mostly just copy this one ... except shouldn't these  
profiles be reusable?

It's hard to figure this out by looking at current registrations (as  
recommended by the stringprep RFC, 2454) if there aren't any.

MfG Kai

MBOX-Line: From [email-masked]  Sat Feb 22 15:42:50 2003
Date: Sat, 22 Feb 2003 15:16:05 -0800 (PST)
From: [email-masked]
Subject: Re: Unicode newsgroup name options
Old-To: [email-masked]
Cc: [email-masked], [email-masked]
Status: O
To: [email-masked]
Sender: [email-masked]

> [email-masked]  wrote on 21.02.03 in <[email-masked]>:

> > Part of the problem with having a group take so long to complete its work is
> > that what's minimally acceptable to the IETF/IESG changes over time. It used
> > to be that protocol could just use some encoding of Unicode, end of story.
> > No more. Now that stringprep exists the IESG requires it be used when
> > Unicode protocol elements are involved. Several recent specifications have
> > been returned to their respective working groups because they didn't do
> > this.
> >
> > I note in passing that while the current News Article Format Draft talks
> > about normalization of group names, it does so without reference to
> > stringprep. This would need to change: Stringprep covers stuff besides
> > normalization, and more generally provides a checklist for Unicode usage
> > each protocol needs to consider.

> That's probably because stringprep wasn't even known to many people on
> USEFOR, and certainly that requirement wasn't.

I already pointed out that this is a new thing.

> Stringprep actually solves a number of problems that some people here
> didn't want to tackle, IIRC. That is, IMO, stringprep is a good thing.

> However, I note that the IANA stringprep profile registry is currently
> *empty* (except for reference to a certain RFC NNNN [that is, reference to
> an unnamed draft] for IDN).

That's to be expected given that the registry is brand new and no RFCs have
been published yet that define such a profile. 

> > I also note that Dan Kohn's alternative News Article Format draft uses
> > punycode, which already includes a stringprep profile.

> Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the
> current one, yes?). We obviously can't use a profile that says it must be
> used together with extra restrictions specific to IDN, but otherwise it
> seems that we could mostly just copy this one ... except shouldn't these
> profiles be reusable?

Some will be, some won't. If this one works for you great, if it needs tweaking 
use it as a starting point for your own.

There's also draft-ietf-ips-iscsi-string-prep-03.txt, which defines a
profile for iSCSI, if you want another one to look at.

> It's hard to figure this out by looking at current registrations (as
> recommended by the stringprep RFC, 2454) if there aren't any.

Unfortunately, somebody always has to go first.

				Ned

MBOX-Line: From [email-masked]  Sat Feb 22 13:05:34 2003
Old-To: Mark Crispin <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 20:47:06 +0000
Lines: 109
Status: O
To: [email-masked]
Sender: [email-masked]

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 > On Sat, 22 Feb 2003, Andrew Gierth wrote:
 >> Mail systems do not appear to make any actual use of the
 >> message-id header other than for logging (and internally in some
 >> odd systems).

 Mark> Huh?  Message-ID, In-Reply-To, and References are heavily used
 Mark> in mail for threading.  IMAP even has a THREAD facility that
 Mark> stipulates how to use these.

perhaps I was insufficiently precise: I was referring specifically to
the transport & distribution of messages rather than the clients.

 >> None of SMTP, POP3 or IMAP ever use message-ids as protocol
 >> parameters.  News, on the other hand, uses message-ids as protocol
 >> parameters _all the time_, both for readers accessing messages and for
 >> server-to-server transfer.

 Mark> Let's try to translate your comment into something useful.  Is it your
 Mark> contention that:
 Mark> 	Some set of commands in NNTP use message-id as an argument.
 Mark> 	The syntax for that command precludes the use of space in
 Mark> 	that argument, and/or NNTP lacks the type of quoting mechanism
 Mark> 	found in IMAP and SMTP.

 Mark> If so, please provide the details.

the NNTP spec can be found in RFC 977 (and the "Common extensions"
document, RFC 2980 if I recall correctly)

Message-ids appear in NNTP as both command parameters and response
parameters (neither of which can contain spaces, though this isn't
spelled out for command parameters - it's just a consequence of the
fact that spaces separate parameters and there is no quoting or
escaping mechanism). For ARTICLE/HEAD/BODY/STAT, the command parameter
can be either a message-id or an article number (or omitted), but the
response string always contains the message-id as a parameter. For the
CHECK/TAKETHIS extension, the message-id must be given as a parameter
and is returned in the result, and the appearance of the id in the
result is always used by the sending server (to match up commands and
responses, since CHECK/TAKETHIS are used for pipelined transfer). For
IHAVE, the message-id is a parameter but does not necessarily appear
in the result.

Breakage on CHECK/TAKETHIS is particularly bad because the resulting
syntax error response does not allow the sending server to properly
identify the failing message; this usually halts the flow of articles
on the affected link as the sending server continually retries the bad
article (until the sending system's admin notices and manually deletes
the bad entry from the outgoing queue). This is why it's important for
servers to actively reject bad message-ids upfront rather than try and
propagate them.

The length limitation of 250 is from existing software rather than
from previous standards (NNTP restricts the total length of a
command-line to 512 including the CRLF). 250 is a better choice
because (a) values that long do not legitimately occur in practice,
(b) there is no particular reason why excessively long values should
be allowed, especially given the large number of CHECK commands, (c)
the shorter limit allows for extensions to add parameters to commands
such as CHECK without also having to extend the command-line limit.

 >> (and, FWIW, I checked about a hundred thousand message-ids taken from
 >> mail messages here, and only one of those had whitespace in and that
 >> was a Chinese spam which had four spaces in place of the domain-part,
 >> almost certainly due to misconfigured or broken spamming software.

 Mark> That isn't an example of a syntactically valid RFC 2822 message-id.

it wasn't intended to be. I found no examples of syntactically valid
RFC 2822 message-ids in my mail archives which were not also valid for
news.

 Mark> And the IETF/IESG is supposed to respect this?

 >> yes

 Mark> If NNTP has a limitation such as suggested above, then that
 Mark> would add a transport imposed requirement to a document that
 Mark> otherwise uses RFC 2822 and MIME as normative.  That is
 Mark> something that can be respected.

 Mark> That is different from duplicating all of RFC 2822 and MIME
 Mark> just to insert transport imposed requirements.  Doing so, which
 Mark> the Usefor document has done, is ludicrous.

I entirely agree (and you'll find a long string of comments to that
effect from me in the Usefor archive).

 Mark> What's worse, it has the highly undesirable effect of cloaking
 Mark> the transport-imposed requirements.

 Mark> A small document, such as Kohn's draft, which has a section of
 Mark> "NNTP transport requirements" would show these requirements in
 Mark> clear contrast.

it is not, in fact, merely an NNTP issue; message-ids are a key part
of _every_ news transport method (since they are used to detect
duplicates).  Every news server has to keep a history database (or the
moral equivalent) indexed by message-id, whether it uses NNTP or not.
Message-ids are passed around a lot internally, stored in queue files,
listed in ihave/sendme control messages for UUCP feeding, appear as
parameters in control messages, and generally pervade the entire
structure of news; whitespace or excessive length breaks things in
really quite a lot of places.

-- 
Andrew.

MBOX-Line: From [email-masked]  Sat Feb 22 15:24:46 2003
Date: Sat, 22 Feb 2003 15:16:05 -0800 (PST)
From: [email-masked]
Subject: Re: Unicode newsgroup name options
Old-To: [email-masked]
Cc: [email-masked], [email-masked]
Status: O
To: [email-masked]
Sender: [email-masked]

> [email-masked]  wrote on 21.02.03 in <[email-masked]>:

> > Part of the problem with having a group take so long to complete its work is
> > that what's minimally acceptable to the IETF/IESG changes over time. It used
> > to be that protocol could just use some encoding of Unicode, end of story.
> > No more. Now that stringprep exists the IESG requires it be used when
> > Unicode protocol elements are involved. Several recent specifications have
> > been returned to their respective working groups because they didn't do
> > this.
> >
> > I note in passing that while the current News Article Format Draft talks
> > about normalization of group names, it does so without reference to
> > stringprep. This would need to change: Stringprep covers stuff besides
> > normalization, and more generally provides a checklist for Unicode usage
> > each protocol needs to consider.

> That's probably because stringprep wasn't even known to many people on
> USEFOR, and certainly that requirement wasn't.

I already pointed out that this is a new thing.

> Stringprep actually solves a number of problems that some people here
> didn't want to tackle, IIRC. That is, IMO, stringprep is a good thing.

> However, I note that the IANA stringprep profile registry is currently
> *empty* (except for reference to a certain RFC NNNN [that is, reference to
> an unnamed draft] for IDN).

That's to be expected given that the registry is brand new and no RFCs have
been published yet that define such a profile. 

> > I also note that Dan Kohn's alternative News Article Format draft uses
> > punycode, which already includes a stringprep profile.

> Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the
> current one, yes?). We obviously can't use a profile that says it must be
> used together with extra restrictions specific to IDN, but otherwise it
> seems that we could mostly just copy this one ... except shouldn't these
> profiles be reusable?

Some will be, some won't. If this one works for you great, if it needs tweaking 
use it as a starting point for your own.

There's also draft-ietf-ips-iscsi-string-prep-03.txt, which defines a
profile for iSCSI, if you want another one to look at.

> It's hard to figure this out by looking at current registrations (as
> recommended by the stringprep RFC, 2454) if there aren't any.

Unfortunately, somebody always has to go first.

				Ned

MBOX-Line: From [email-masked]  Sat Feb 22 11:13:27 2003
Date: Sat, 22 Feb 2003 10:34:36 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
Old-To: Andrew Gierth <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Status: O
To: [email-masked]
Sender: [email-masked]

Andrew,

Saturday, February 22, 2003, 2:05:34 AM, you wrote:
AG> Mail systems do not appear to make any actual use of the message-id
AG> header other than for logging (and internally in some odd systems).
AG> None of SMTP, POP3 or IMAP ever use message-ids as protocol
AG> parameters.

Message-ids are not nearly as deeply and usefully and broadly used as we
had expected them to be.  However, they ARE used, to good effect, for
duplicate detection and for thread construction.

When looking for utility, please include the layer ABOVE the usual IETF
protocols.  Think of it as the User-to-User protocol.  It's the one that
users actually care about.  The rest is just support infrastructure.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.866-358-5301

MBOX-Line: From [email-masked]  Sat Feb 22 09:45:00 2003
Date: Sat, 22 Feb 2003 11:37:08 -0500
From: Cyrus Daboo <[email-masked]>
Old-To: Andrew Gierth <[email-masked]>, Mark Crispin <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Status: O
To: [email-masked]
Sender: [email-masked]

Hi Andrew,

--On Saturday, February 22, 2003 10:05 AM +0000 Andrew Gierth 
<[email-masked]> wrote:

| Mail systems do not appear to make any actual use of the message-id
| header other than for logging (and internally in some odd systems).

Message-ID, In-Reply-To and References are an integral part of 'threading' 
email messages - its not just news. Email clients that do threading will 
typically use those. The IMAP THREAD extension also uses those.

-- 
Cyrus Daboo

MBOX-Line: From [email-masked]  Sat Feb 22 13:17:46 2003
Date: Sat, 22 Feb 2003 13:04:48 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
Old-To: Russ Allbery <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: mail/news unification
Status: RO
To: [email-masked]
Sender: [email-masked]

Russ,

Friday, February 21, 2003, 1:57:44 PM, you wrote:
RA> I personally am very strongly in favor of the unification of messaging
RA> formats,

Email "versus" Netnews debates have existed since the mid-70s.  (I count
the early spate of textual, messaging-oriented collaborations systems
that came out of the gas crisis of the early 70s as roughly equivalent
to netnews.)

I believe the basic problem with the debates has been a focus on
mechanism rather than human communication.  As soon as we focus on the
human functions, we can identify only a few, salient differences, I
believe.

To me, the only user-level difference that is major is one of
addressing, between explicitly sending to one or more individuals,
versus explicitly sending to a "group". In this regard, I believe that
that netnews is essentially the same as an email list.

Yes, one subscribes to a list but not to a newsgroup. However think
about the mode of posting to an open mailing list that you are not
subscribed to, and using the mailing list archive as the basis for
reading contributions.  It is darn close to the style of a newsgroup.

Imagine that Internet mail had a construct for labeling a mailing list.
(One might even hypothesize an additional addressing header, such as
"Follow-up" or... "Newsgroup", or more email-list styled as "List-To" or
the like.)

The infrastructure difference that is massive is, of course, the
relaying technology.  Netnews uses a flooding algorithm and mail is
multicast.   I have a suspicion there is some sort of opportunity for
convergence, there, but have not tried to think of how.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.866-358-5301

MBOX-Line: From [email-masked]  Sat Feb 22 12:48:05 2003
Old-To: Mark Crispin <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 20:47:06 +0000
Lines: 109
Status: O
To: [email-masked]
Sender: [email-masked]

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 > On Sat, 22 Feb 2003, Andrew Gierth wrote:
 >> Mail systems do not appear to make any actual use of the
 >> message-id header other than for logging (and internally in some
 >> odd systems).

 Mark> Huh?  Message-ID, In-Reply-To, and References are heavily used
 Mark> in mail for threading.  IMAP even has a THREAD facility that
 Mark> stipulates how to use these.

perhaps I was insufficiently precise: I was referring specifically to
the transport & distribution of messages rather than the clients.

 >> None of SMTP, POP3 or IMAP ever use message-ids as protocol
 >> parameters.  News, on the other hand, uses message-ids as protocol
 >> parameters _all the time_, both for readers accessing messages and for
 >> server-to-server transfer.

 Mark> Let's try to translate your comment into something useful.  Is it your
 Mark> contention that:
 Mark> 	Some set of commands in NNTP use message-id as an argument.
 Mark> 	The syntax for that command precludes the use of space in
 Mark> 	that argument, and/or NNTP lacks the type of quoting mechanism
 Mark> 	found in IMAP and SMTP.

 Mark> If so, please provide the details.

the NNTP spec can be found in RFC 977 (and the "Common extensions"
document, RFC 2980 if I recall correctly)

Message-ids appear in NNTP as both command parameters and response
parameters (neither of which can contain spaces, though this isn't
spelled out for command parameters - it's just a consequence of the
fact that spaces separate parameters and there is no quoting or
escaping mechanism). For ARTICLE/HEAD/BODY/STAT, the command parameter
can be either a message-id or an article number (or omitted), but the
response string always contains the message-id as a parameter. For the
CHECK/TAKETHIS extension, the message-id must be given as a parameter
and is returned in the result, and the appearance of the id in the
result is always used by the sending server (to match up commands and
responses, since CHECK/TAKETHIS are used for pipelined transfer). For
IHAVE, the message-id is a parameter but does not necessarily appear
in the result.

Breakage on CHECK/TAKETHIS is particularly bad because the resulting
syntax error response does not allow the sending server to properly
identify the failing message; this usually halts the flow of articles
on the affected link as the sending server continually retries the bad
article (until the sending system's admin notices and manually deletes
the bad entry from the outgoing queue). This is why it's important for
servers to actively reject bad message-ids upfront rather than try and
propagate them.

The length limitation of 250 is from existing software rather than
from previous standards (NNTP restricts the total length of a
command-line to 512 including the CRLF). 250 is a better choice
because (a) values that long do not legitimately occur in practice,
(b) there is no particular reason why excessively long values should
be allowed, especially given the large number of CHECK commands, (c)
the shorter limit allows for extensions to add parameters to commands
such as CHECK without also having to extend the command-line limit.

 >> (and, FWIW, I checked about a hundred thousand message-ids taken from
 >> mail messages here, and only one of those had whitespace in and that
 >> was a Chinese spam which had four spaces in place of the domain-part,
 >> almost certainly due to misconfigured or broken spamming software.

 Mark> That isn't an example of a syntactically valid RFC 2822 message-id.

it wasn't intended to be. I found no examples of syntactically valid
RFC 2822 message-ids in my mail archives which were not also valid for
news.

 Mark> And the IETF/IESG is supposed to respect this?

 >> yes

 Mark> If NNTP has a limitation such as suggested above, then that
 Mark> would add a transport imposed requirement to a document that
 Mark> otherwise uses RFC 2822 and MIME as normative.  That is
 Mark> something that can be respected.

 Mark> That is different from duplicating all of RFC 2822 and MIME
 Mark> just to insert transport imposed requirements.  Doing so, which
 Mark> the Usefor document has done, is ludicrous.

I entirely agree (and you'll find a long string of comments to that
effect from me in the Usefor archive).

 Mark> What's worse, it has the highly undesirable effect of cloaking
 Mark> the transport-imposed requirements.

 Mark> A small document, such as Kohn's draft, which has a section of
 Mark> "NNTP transport requirements" would show these requirements in
 Mark> clear contrast.

it is not, in fact, merely an NNTP issue; message-ids are a key part
of _every_ news transport method (since they are used to detect
duplicates).  Every news server has to keep a history database (or the
moral equivalent) indexed by message-id, whether it uses NNTP or not.
Message-ids are passed around a lot internally, stored in queue files,
listed in ihave/sendme control messages for UUCP feeding, appear as
parameters in control messages, and generally pervade the entire
structure of news; whitespace or excessive length breaks things in
really quite a lot of places.

-- 
Andrew.

MBOX-Line: From [email-masked]  Sat Feb 22 13:49:44 2003
Date: Sat, 22 Feb 2003 16:37:23 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
Old-To: Usefor Mailing List <[email-masked]>
CC: IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Status: O
To: [email-masked]
Sender: [email-masked]

Andrew Gierth wrote:
> 
> >>>>> "Dave" == Dave Crocker <[email-masked]> writes:
> 
>  AG> Mail systems do not appear to make any actual use of the
>  AG> message-id header other than for logging (and internally in some
>  AG> odd systems).  None of SMTP, POP3 or IMAP ever use message-ids as
>  AG> protocol parameters.
> 
>  Dave> Message-ids are not nearly as deeply and usefully and broadly
>  Dave> used as we had expected them to be.  However, they ARE used, to
>  Dave> good effect, for duplicate detection and for thread
>  Dave> construction.
> 
> at the layer of actual message transport, or just in the clients or
> client-side services such as IMAP threading? I really wasn't referring
> to the clients when I mentioned "mail systems".
> 
> I have heard of mail transports that assume a 1-1 correspondence
> between message-ids and body content but I believe this to be highly
> unusual; please correct me if I am mistaken on this point.

FYI, the LMTP daemon in Cyrus IMAP uses the message-id for duplicate
suppression (and to prevent mail loops) at the time of delivery.  This
isn't done as part of the protocol as in NNTP, but its done internally
for every message before its committed to disk.

MBOX-Line: From [email-masked]  Sat Feb 22 08:59:23 2003
Date: Sat, 22 Feb 2003 08:33:50 -0800 (PST)
From: [email-masked]
Subject: Re: mail vs. news ???
Old-To: Andrew Gierth <[email-masked]>
Cc: Mark Crispin <[email-masked]>, Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Status: RO
To: [email-masked]
Sender: [email-masked]

> >>>>> "Mark" == Mark Crispin <[email-masked]> writes:

>  Mark> Now we hear the claim that completely unnecessary restictions
>  Mark> in headers are necessary because of news software.

> who says those restrictions are unnecessary?

> Mail systems do not appear to make any actual use of the message-id
> header other than for logging (and internally in some odd systems).
> None of SMTP, POP3 or IMAP ever use message-ids as protocol
> parameters.  News, on the other hand, uses message-ids as protocol
> parameters _all the time_, both for readers accessing messages and for
> server-to-server transfer.

FWIW, draft-ietf-imapext-thread-12.txt is the obvious counterexample. But even
if what you say is true about the present email transport infrastructure, there
are plenty of email clients that support threading via message-id.

> (and, FWIW, I checked about a hundred thousand message-ids taken from
> mail messages here, and only one of those had whitespace in and that
> was a Chinese spam which had four spaces in place of the domain-part,
> almost certainly due to misconfigured or broken spamming software. So
> this clearly isn't a feature that anyone actually _needs_, or probably
> even wants.)

Which of course is illegal according to any of the relevant standards.

Mind you, I'm not saying that additional restrictions on message-ids in news
are inappropriate. At a minimum the restrictions of RFC 1036 have to be taken
into account. However, I do urge caution when attempting to cater to failure to
support the existing specifications in the installed base, however. This is a
game that has to be played very carefully.

				Ned

MBOX-Line: From [email-masked]  Sat Feb 22 11:20:39 2003
Old-To: Cyrus Daboo <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 19:07:01 +0000
Lines: 23
Status: O
To: [email-masked]
Sender: [email-masked]

>>>>> "Cyrus" == Cyrus Daboo <[email-masked]> writes:

 > | Mail systems do not appear to make any actual use of the message-id
 > | header other than for logging (and internally in some odd systems).

 Cyrus> Message-ID, In-Reply-To and References are an integral part of
 Cyrus> 'threading' email messages - its not just news. Email clients
 Cyrus> that do threading will typically use those. The IMAP THREAD
 Cyrus> extension also uses those.

that's primarily a client issue, or an issue of returning values from
the server (as, for example, XOVER does in news).

The more important use of message-ids in news is for clients to
request articles from the server, and for servers to offer articles to
each other - all this is done by means of message-ids. News servers
must therefore index stored articles by message-id, and message-ids
appear all the time in NNTP commands and responses. Therefore both the
no-whitespace and maximum length restrictions serve a necessary
purpose.

-- 
Andrew.

MBOX-Line: From [email-masked]  Sat Feb 22 11:07:33 2003
Old-To: Cyrus Daboo <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 19:07:01 +0000
Lines: 23
Status: O
To: [email-masked]
Sender: [email-masked]

>>>>> "Cyrus" == Cyrus Daboo <[email-masked]> writes:

 > | Mail systems do not appear to make any actual use of the message-id
 > | header other than for logging (and internally in some odd systems).

 Cyrus> Message-ID, In-Reply-To and References are an integral part of
 Cyrus> 'threading' email messages - its not just news. Email clients
 Cyrus> that do threading will typically use those. The IMAP THREAD
 Cyrus> extension also uses those.

that's primarily a client issue, or an issue of returning values from
the server (as, for example, XOVER does in news).

The more important use of message-ids in news is for clients to
request articles from the server, and for servers to offer articles to
each other - all this is done by means of message-ids. News servers
must therefore index stored articles by message-id, and message-ids
appear all the time in NNTP commands and responses. Therefore both the
no-whitespace and maximum length restrictions serve a necessary
purpose.

-- 
Andrew.

MBOX-Line: From [email-masked]  Sat Feb 22 08:51:24 2003
Date: Sat, 22 Feb 2003 11:37:08 -0500
From: Cyrus Daboo <[email-masked]>
Old-To: Andrew Gierth <[email-masked]>, Mark Crispin <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Status: O
To: [email-masked]
Sender: [email-masked]

Hi Andrew,

--On Saturday, February 22, 2003 10:05 AM +0000 Andrew Gierth 
<[email-masked]> wrote:

| Mail systems do not appear to make any actual use of the message-id
| header other than for logging (and internally in some odd systems).

Message-ID, In-Reply-To and References are an integral part of 'threading' 
email messages - its not just news. Email clients that do threading will 
typically use those. The IMAP THREAD extension also uses those.

-- 
Cyrus Daboo

MBOX-Line: From [email-masked]  Sat Feb 22 08:48:05 2003
Date: Sat, 22 Feb 2003 08:33:50 -0800 (PST)
From: [email-masked]
Subject: Re: mail vs. news ???
Old-To: Andrew Gierth <[email-masked]>
Cc: Mark Crispin <[email-masked]>, Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Status: RO
To: [email-masked]
Sender: [email-masked]

> >>>>> "Mark" == Mark Crispin <[email-masked]> writes:

>  Mark> Now we hear the claim that completely unnecessary restictions
>  Mark> in headers are necessary because of news software.

> who says those restrictions are unnecessary?

> Mail systems do not appear to make any actual use of the message-id
> header other than for logging (and internally in some odd systems).
> None of SMTP, POP3 or IMAP ever use message-ids as protocol
> parameters.  News, on the other hand, uses message-ids as protocol
> parameters _all the time_, both for readers accessing messages and for
> server-to-server transfer.

FWIW, draft-ietf-imapext-thread-12.txt is the obvious counterexample. But even
if what you say is true about the present email transport infrastructure, there
are plenty of email clients that support threading via message-id.

> (and, FWIW, I checked about a hundred thousand message-ids taken from
> mail messages here, and only one of those had whitespace in and that
> was a Chinese spam which had four spaces in place of the domain-part,
> almost certainly due to misconfigured or broken spamming software. So
> this clearly isn't a feature that anyone actually _needs_, or probably
> even wants.)

Which of course is illegal according to any of the relevant standards.

Mind you, I'm not saying that additional restrictions on message-ids in news
are inappropriate. At a minimum the restrictions of RFC 1036 have to be taken
into account. However, I do urge caution when attempting to cater to failure to
support the existing specifications in the installed base, however. This is a
game that has to be played very carefully.

				Ned

MBOX-Line: From [email-masked]  Sat Feb 22 02:20:59 2003
Old-To: Mark Crispin <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 10:05:34 +0000
Lines: 27
Status: O
To: [email-masked]
Sender: [email-masked]

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> Now we hear the claim that completely unnecessary restictions
 Mark> in headers are necessary because of news software.

who says those restrictions are unnecessary?

Mail systems do not appear to make any actual use of the message-id
header other than for logging (and internally in some odd systems).
None of SMTP, POP3 or IMAP ever use message-ids as protocol
parameters.  News, on the other hand, uses message-ids as protocol
parameters _all the time_, both for readers accessing messages and for
server-to-server transfer.

(and, FWIW, I checked about a hundred thousand message-ids taken from
mail messages here, and only one of those had whitespace in and that
was a Chinese spam which had four spaces in place of the domain-part,
almost certainly due to misconfigured or broken spamming software. So
this clearly isn't a feature that anyone actually _needs_, or probably
even wants.)

 Mark> And the IETF/IESG is supposed to respect this?

yes

-- 
Andrew.

MBOX-Line: From [email-masked]  Sat Feb 22 02:06:15 2003
Old-To: Mark Crispin <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 10:05:34 +0000
Lines: 27
Status: O
To: [email-masked]
Sender: [email-masked]

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> Now we hear the claim that completely unnecessary restictions
 Mark> in headers are necessary because of news software.

who says those restrictions are unnecessary?

Mail systems do not appear to make any actual use of the message-id
header other than for logging (and internally in some odd systems).
None of SMTP, POP3 or IMAP ever use message-ids as protocol
parameters.  News, on the other hand, uses message-ids as protocol
parameters _all the time_, both for readers accessing messages and for
server-to-server transfer.

(and, FWIW, I checked about a hundred thousand message-ids taken from
mail messages here, and only one of those had whitespace in and that
was a Chinese spam which had four spaces in place of the domain-part,
almost certainly due to misconfigured or broken spamming software. So
this clearly isn't a feature that anyone actually _needs_, or probably
even wants.)

 Mark> And the IETF/IESG is supposed to respect this?

yes

-- 
Andrew.

MBOX-Line: From [email-masked]  Fri Feb 21 22:15:08 2003
Date: Fri, 21 Feb 2003 22:03:31 -0800 (PST)
From: [email-masked]
Subject: Re: Unicode newsgroup name options
Cc: [email-masked]
Status: O
To: [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > I also note that Dan Kohn's alternative News Article Format draft uses
> > punycode, which already includes a stringprep profile.

> Actually, Punycode does not include a Stringprep profile, or depend on
> one in any way.  Punycode maps *any* Unicode string to an ASCII string,
> reversibly (without loss).

Point taken.

				Ned

MBOX-Line: From [email-masked]  Fri Feb 21 20:31:52 2003
Date: Fri, 21 Feb 2003 20:31:24 -0800 (Pacific Standard Time)
From: Mark Crispin <[email-masked]>
Old-To: Terje Bless <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Networks & Distributed Computing
Status: O
To: [email-masked]
Sender: [email-masked]

On Sat, 22 Feb 2003, Terje Bless wrote:
> I do not know the situation on other platforms, but on Mac OS X the system
> libraries -- including GUI frameworks -- operate on UTF-16 strings
> internally and support UNICODE internally. I would expect this to be common
> on most platforms today or in the very near future. IOW, this whole issue
> is likely to be allready solved by the host platform for the "GUI programs
> and screen editors".

I suggest that you read up on stringprep.  There's quite a lot that needs
to be done that isn't done by these libraries.

> There are other parts of the infrastructure where I would worry far more
> about the "width and length" of characters.

Even if the OS and libraries support Unicode, the application programs
still need to use those routines.  You can't expect to take an old program
and have it magically become Unicode-savvy.

All you have to do is look at Windows, which has been Unicode savvy for
years longer than Mac.  Most applications on Windows are still not Unicode
savvy.

A Unicode savvy application has to be designed to be Unicode savvy from
the ground up.  As soon as you get into GUIs, you have all sorts of other
wonderful details to worry about.  You actually have to alter the layout
of your GUI on a script-dependent basis.  No amount of OS libraries will
do this for you.

There are some simplifying assumptions that you may be able to make for
newsgroup names.  Maybe.  It won't work for text.

It ain't just a matter of wider characters and bigger fonts.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

MBOX-Line: From [email-masked]  Fri Feb 21 20:40:11 2003
Date: Fri, 21 Feb 2003 20:31:24 -0800 (Pacific Standard Time)
From: Mark Crispin <[email-masked]>
Old-To: Terje Bless <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Networks & Distributed Computing
Status: O
To: [email-masked]
Sender: [email-masked]

On Sat, 22 Feb 2003, Terje Bless wrote:
> I do not know the situation on other platforms, but on Mac OS X the system
> libraries -- including GUI frameworks -- operate on UTF-16 strings
> internally and support UNICODE internally. I would expect this to be common
> on most platforms today or in the very near future. IOW, this whole issue
> is likely to be allready solved by the host platform for the "GUI programs
> and screen editors".

I suggest that you read up on stringprep.  There's quite a lot that needs
to be done that isn't done by these libraries.

> There are other parts of the infrastructure where I would worry far more
> about the "width and length" of characters.

Even if the OS and libraries support Unicode, the application programs
still need to use those routines.  You can't expect to take an old program
and have it magically become Unicode-savvy.

All you have to do is look at Windows, which has been Unicode savvy for
years longer than Mac.  Most applications on Windows are still not Unicode
savvy.

A Unicode savvy application has to be designed to be Unicode savvy from
the ground up.  As soon as you get into GUIs, you have all sorts of other
wonderful details to worry about.  You actually have to alter the layout
of your GUI on a script-dependent basis.  No amount of OS libraries will
do this for you.

There are some simplifying assumptions that you may be able to make for
newsgroup names.  Maybe.  It won't work for text.

It ain't just a matter of wider characters and bigger fonts.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

MBOX-Line: From [email-masked]  Fri Feb 21 19:49:32 2003
Date: Sat, 22 Feb 2003 04:48:42 +0100
From: Terje Bless <[email-masked]>
Subject: Re: Unicode newsgroup name options
Old-To: [email-masked]
Cc: [email-masked], [email-masked], Mark Crispin <[email-masked]>
Status: O
To: [email-masked]
Sender: [email-masked]

Mark Crispin <[email-masked]> wrote:

>Step back from the tree to see the forest.  "Some tweaking" includes
>such small details as not assuming that one byte means one screen space,
>which has all sorts of implications in GUI programs and screen editors. 
>Don't forget that you don't just have variable length characters in
>UTF-8, but you also have variable width characters in Unicode, plus
>characters which interact with other characters in interesting ways.

I do not know the situation on other platforms, but on Mac OS X the system
libraries -- including GUI frameworks -- operate on UTF-16 strings
internally and support UNICODE internally. I would expect this to be common
on most platforms today or in the very near future. IOW, this whole issue
is likely to be allready solved by the host platform for the "GUI programs
and screen editors".

There are other parts of the infrastructure where I would worry far more
about the "width and length" of characters.

-- 
When I decide that the situation is unacceptable for me, I'll simply fork
the tree.   I do _not_ appreciate being enlisted into anyone's holy wars,
so unless you _really_ want to go _way_ up in my  personal shitlist don't
play politics in my vicinity.                   -- Alexander Viro on lkml

Date: Sun, 23 Feb 2003 01:33:35 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: usenet-format <[email-masked]>, [email-masked]
Organization: Bruce Lilly
To: Andrew Gierth <[email-masked]>
CC: usenet-format <[email-masked]>, [email-masked]
Subject: Message-IDs (Was Re: Kohn draft (was RE: USEFOR: Current situation and next steps))
Sender: [email-masked]

Andrew Gierth wrote:
> [adding ietf-822 back in as this is getting into their territory]

> 2822 did specifically make the message-id syntax more restrictive than
> 822 (822 allowed whitespace in quoted strings, 2822 only allows it in
> quoted-pair (i.e. preceeded by \)

If you look carefully, you'll see that not only must the whitespace
be backslash-escaped, that in turn must be within a quoted-string.
I.e.
   <foo [email-masked]>    illegal
   <foo\ [email-masked]>   illegal
   <"foo bar"@baz.com>  illegal
   <"foo\ bar"@baz.com> legal

> Message-ids that are valid under 2822 but not valid under 1036 (or are
> longer than the 250-char limit) seem not to occur in mail with any
> frequency (at least I don't see any in those mail collections I have
> searched). I would be interested in whether anyone with access to large
> amounts of real-world email can confirm this.

I have a small collection (22937 messages). I checked the msg-ids in
top-level header fields Message-ID, Received, In-Reply-To, References,
Resent-Message-ID, Content-ID, and Supersedes. There were none that
were simultaneously
1. legal under RFC 2822 generate rules
and
2a. longer than 250 octets including angle brackets
  or
2b. had backslash-escaped space or control characters in a quoted string.

Those were all fairly recent messages (within the past 9 months).

Errors that were present included:
1. no @ (the ones in Received fields were generated by MS
    "Internet Mail Service", by something called "ZMailer"
    and/or "ZMailer Server", and by whatever is running at
    maillennium.att.com, mtiwmhc11.worldnet.att.net,
    mtiwmhc12.worldnet.att.net, and mtiwmhc13.worldnet.att.net.
    Also apparently generated in Message-ID fields by
    MDaemon.v2.7.SP4.R (in spam).
2. whitespace (sometimes on the left, sometimes on the right, in
    both in one case), usually space characters, but in some cases
    tabs (often these appear in References fields, where the
    original Message-ID was compliant)
3. line folding (typically the result of some software mangling
    a msg-id with internal whitespace)
4. 8-bit cruft (1 case; spam) generated either by Microsoft
    MimeOLE V6.00.2600.0000 or by Microsoft SMTPSVC

Date: Sat, 22 Feb 2003 21:53:53 -0600
From: "Eric A. Hall" <[email-masked]>
To: Dan Kohn <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: yEnc (was RE: mail vs. news ???)
Sender: [email-masked]

on 2/21/2003 4:25 PM Dan Kohn wrote:

> Note that there's absolutely no reason why yEnc (perhaps with optional
> gzip functionality added in) can't be specified as a new
> Content-Transport-Encoding as an alternative to base64.  Further, if
> message/partial doesn't do the job, than a new application/news-partial
> could be specified.

Unfortunately there are loads of issues that make this whole thing very
complex. Its been tackled at least a few times, with the result typically
being that everybody is too exhausted to continue discussing it.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: "Usefor Mailing List" <[email-masked]>, "IMAP mailing list" <[email-masked]>, "IETF-822 Mailing List" <[email-masked]>
Subject: Re: mail vs. news ???
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sat, 22 Feb 2003 18:47:37 -0800
Lines: 41
Sender: [email-masked]

Dan Kohn <[email-masked]> writes:

> In general, I'm thinking about suggesting a different tact from RFC
> 2822, which said that Section 3 current syntax is MUST generate and
> Section 4 obsolete syntax is MUST accept.  Instead, I'm thinking of
> suggesting that agents SHOULD generate messages that honor current news
> restrictions (e.g., no Received headers, tighter Message IDs, SP after
> colon in all headers), while saying that agents compliant with the spec
> MUST accept RFC 2822 syntax.  That way, implmenters know what to do to
> maximize the likelihood that their messages will be processed, but
> there's also a push toward supporting a universal Internet Message
> Format syntax across all messaging protocols.

It depends on what portions of the protocol you're working with, but
tighter message IDs is a MUST, and acceptance of message IDs without those
limitations is not acceptable and will cause serious and significant
interoperability problems.

This is simply an area where Usenet cannot change.  There is too much
deployed software out there that assumes things about message IDs.  Trying
to write a standard that says that Usenet software should accept long
message IDs and message IDs with spaces is a waste of time.  It's like
inventing a new delimiter instead of colon for e-mail headers and then
saying that mail software must accept it; you'd have more luck inventing a
completely new protocol and getting people to switch to it.

For the other bits, I also think that you're wasting your time on some of
them (like spaces after colons in headers, or like allowing multiple From
or Subject headers in the same message) and the time would be better spent
moving a few more really marginal bits of syntax on the e-mail side into
the obsolete MUST accept, MUST NOT generate category.  The amount of
software that would have to change to support this on the news side is
rather large and I don't expect it to ever happen fully because those
changes offer no additional expressiveness or function.  But the changes
are at least easier, so I don't feel as strongly about it.

When it comes to accepting Received headers, sure, whatever.  I don't
really care.  They're meaningless in Usenet messages really.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

content-class: urn:content-classes:message
Subject: yEnc (was RE: mail vs. news ???)
Importance: normal
Date: Fri, 21 Feb 2003 14:25:21 -0800
From: "Dan Kohn" <[email-masked]>
To: "Usefor Mailing List" <[email-masked]>, "IMAP mailing list" <[email-masked]>, "IETF-822 Mailing List" <[email-masked]>
Sender: [email-masked]

Russ Albery wrote:

> - MIME never got very much uptake on Usenet for attachments.  The
> binary newsgroups are almost universally uuencode or, these days,
> yEnc.  Base64 is rather rare and not at all popular.  Binary readers
> and posters, for reasons that I admit to really not understanding,
> seem to have had no interest in the additional, cleaner semantics
> offered by MIME attachments and continue to be happy with uuencode,
> scanning the body for telltale lines, and doing guessing based on
> file names.  Part of this may be due to the fact that much of the
> Usenet binary traffic is split across multiple posts and the MIME
> message/partial content-type is ill-suited to Usenet for a wide
> variety of reasons.  Just splitting a uuencoded file into multiple
> parts is a lot easier (although it certainly makes decoding a pain).

Note that there's absolutely no reason why yEnc (perhaps with optional
gzip functionality added in) can't be specified as a new
Content-Transport-Encoding as an alternative to base64.  Further, if
message/partial doesn't do the job, than a new application/news-partial
could be specified.  The issue is that the news community has rarely
found it useful or necessary to standardize common practices.  I suggest
that this has led to a raft of interoperability problems (especially the
unlabeled 8-bit character sets).

The solution, I believe, is to get a base document published on the
standards track with no new functionality except for i18n and MIME
support, so that various implementers can start adding interoperable
features with new drafts (even if some are on the Experimental track).
The IMAP WG, though not perfect, is an example of how this can and
should be done.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

content-class: urn:content-classes:message
Subject: RE: mail vs. news ???
Importance: normal
Date: Fri, 21 Feb 2003 14:34:34 -0800
From: "Dan Kohn" <[email-masked]>
To: "Usefor Mailing List" <[email-masked]>, "IMAP mailing list" <[email-masked]>, "IETF-822 Mailing List" <[email-masked]>
Sender: [email-masked]

Russ Albery wrote:

>> Usenet's restrictions on the syntax of message ID headers are very
>> specific and very precise, and much stronger than those of RFC 2822,
>> in part because message IDs are used as part of the NNTP protocol.

>> Yes, although there are a variety of subtle things that fall into
this
>> category that one has to be careful of.  That's one of the
>> difficulties with Dan Kohn's existing draft; in dropping all of the
>> language that was completely unnecessarily copied from RFC 2822, he's
>> also dropped some language that's actually necessary.

Mark Crispin wrote:

> Golly gee, where's the chorus of "these are bugs that should be fixed"
> now?  First we hear the claim that 7-bit messaging restrictions in
> mail are a "bug that should be fixed" even though 7-bit was
> specifically in the standard.

> But rather than fix the disaster, they seem to want to inflict a new
> disaster upon the email community.

> The solution to interoperability is to stop claiming that news is
> special, and start playing ball with the rest of the messaging world.
> This means making compromises, including at times accepting what
> seems to be unnecessary limitations, in order to achieve
> interoperability. 

I'm working on a new version of Kohn draft that captures more of the
semantics from the Lindsey draft about news-specific restrictions.  I
believe laying them out against the context of RFC 2822 (rather than
expecting the implementer to compare the documents) will produce the
best interoperability.

In general, I'm thinking about suggesting a different tact from RFC
2822, which said that Section 3 current syntax is MUST generate and
Section 4 obsolete syntax is MUST accept.  Instead, I'm thinking of
suggesting that agents SHOULD generate messages that honor current news
restrictions (e.g., no Received headers, tighter Message IDs, SP after
colon in all headers), while saying that agents compliant with the spec
MUST accept RFC 2822 syntax.  That way, implmenters know what to do to
maximize the likelihood that their messages will be processed, but
there's also a push toward supporting a universal Internet Message
Format syntax across all messaging protocols.

Of course, once the uniquely news syntax is specified clearly (i.e.,
against the context of RFC 2822), the WG (assuming it's rechartered) can
hash out all the MAYs, SHOULDs, and MUSTs.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Sat, 22 Feb 2003 15:16:05 -0800 (PST)
From: [email-masked]
Subject: Re: Unicode newsgroup name options
To: [email-masked]
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> [email-masked]  wrote on 21.02.03 in <[email-masked]>:

> > Part of the problem with having a group take so long to complete its work is
> > that what's minimally acceptable to the IETF/IESG changes over time. It used
> > to be that protocol could just use some encoding of Unicode, end of story.
> > No more. Now that stringprep exists the IESG requires it be used when
> > Unicode protocol elements are involved. Several recent specifications have
> > been returned to their respective working groups because they didn't do
> > this.
> >
> > I note in passing that while the current News Article Format Draft talks
> > about normalization of group names, it does so without reference to
> > stringprep. This would need to change: Stringprep covers stuff besides
> > normalization, and more generally provides a checklist for Unicode usage
> > each protocol needs to consider.

> That's probably because stringprep wasn't even known to many people on
> USEFOR, and certainly that requirement wasn't.

I already pointed out that this is a new thing.

> Stringprep actually solves a number of problems that some people here
> didn't want to tackle, IIRC. That is, IMO, stringprep is a good thing.

> However, I note that the IANA stringprep profile registry is currently
> *empty* (except for reference to a certain RFC NNNN [that is, reference to
> an unnamed draft] for IDN).

That's to be expected given that the registry is brand new and no RFCs have
been published yet that define such a profile. 

> > I also note that Dan Kohn's alternative News Article Format draft uses
> > punycode, which already includes a stringprep profile.

> Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the
> current one, yes?). We obviously can't use a profile that says it must be
> used together with extra restrictions specific to IDN, but otherwise it
> seems that we could mostly just copy this one ... except shouldn't these
> profiles be reusable?

Some will be, some won't. If this one works for you great, if it needs tweaking 
use it as a starting point for your own.

There's also draft-ietf-ips-iscsi-string-prep-03.txt, which defines a
profile for iSCSI, if you want another one to look at.

> It's hard to figure this out by looking at current registrations (as
> recommended by the stringprep RFC, 2454) if there aren't any.

Unfortunately, somebody always has to go first.

				Ned

To: [email-masked] (Kai Henningsen)
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Simon Josefsson <[email-masked]>
Date: Sat, 22 Feb 2003 23:46:06 +0100
Sender: [email-masked]

[email-masked] (Kai Henningsen) writes:

>> I also note that Dan Kohn's alternative News Article Format draft uses
>> punycode, which already includes a stringprep profile.
>
> Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the  
> current one, yes?). We obviously can't use a profile that says it must be  
> used together with extra restrictions specific to IDN, but otherwise it  
> seems that we could mostly just copy this one ... except shouldn't these  
> profiles be reusable?

Yes.  Using nameprep for usefor is a bad idea, but the contents of
nameprep can be copied, renamed, polished, and then, IMHO, it would
turn into a good idea.

> It's hard to figure this out by looking at current registrations (as  
> recommended by the stringprep RFC, 2454) if there aren't any.

There are a few drafts with stringprep profile, here are links to
those I'm familiar with.  Note that most of them contain some minor
technical problems.

http://www.ietf.org/internet-drafts/draft-ietf-idn-nameprep-11.txt
http://www.ietf.org/internet-drafts/draft-ietf-xmpp-nodeprep-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-xmpp-resourceprep-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-sasl-anon-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-string-prep-03.txt

Date: 22 Feb 2003 23:10:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
cc: [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked]  wrote on 21.02.03 in <[email-masked]>:

> Part of the problem with having a group take so long to complete its work is
> that what's minimally acceptable to the IETF/IESG changes over time. It used
> to be that protocol could just use some encoding of Unicode, end of story.
> No more. Now that stringprep exists the IESG requires it be used when
> Unicode protocol elements are involved. Several recent specifications have
> been returned to their respective working groups because they didn't do
> this.
>
> I note in passing that while the current News Article Format Draft talks
> about normalization of group names, it does so without reference to
> stringprep. This would need to change: Stringprep covers stuff besides
> normalization, and more generally provides a checklist for Unicode usage
> each protocol needs to consider.

That's probably because stringprep wasn't even known to many people on  
USEFOR, and certainly that requirement wasn't.

Stringprep actually solves a number of problems that some people here  
didn't want to tackle, IIRC. That is, IMO, stringprep is a good thing.

However, I note that the IANA stringprep profile registry is currently  
*empty* (except for reference to a certain RFC NNNN [that is, reference to  
an unnamed draft] for IDN).

> I also note that Dan Kohn's alternative News Article Format draft uses
> punycode, which already includes a stringprep profile.

Some detective work finds draft-ietf-idn-nameprep-11.txt (that *is* the  
current one, yes?). We obviously can't use a profile that says it must be  
used together with extra restrictions specific to IDN, but otherwise it  
seems that we could mostly just copy this one ... except shouldn't these  
profiles be reusable?

It's hard to figure this out by looking at current registrations (as  
recommended by the stringprep RFC, 2454) if there aren't any.

MfG Kai

Date: 22 Feb 2003 22:28:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
cc: [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Lawrence Greenfield)  wrote on 21.02.03 in <[email-masked]>:

>    Date: 21 Feb 2003 22:11:00 +0200
>    From: [email-masked] (Kai Henningsen)
> [...]
>    UTF-7 has, however, one important problem: it uses + as the escape
>    character. My active file has over 200 groups with a + in their name.
>
> [... reinventing IMAP4's modified UTF-7 elided ...]
>
> You might want to read the IMAP archives where the decision to use
> modified UTF-7 for mailbox names was made and some of the feedback
> there. The IMAP mUTF-7 goals are close enough to yours that you could
> just adapt it and at least make one group (IMAP client/servers)
> easier. But the IMAP community in general seems like it has been less
> than thrilled about the mUTF-7 mailbox names.

Actually, it seems that mUTF-7 fails to address the unique encoding  
requirement just as UTF-7 does.

Apart from that, it is certainly concievable that omething in this  
direction could be done - there's a reason I called mine a strawman  
proposal. I certainly didn't expect it to be final - it was meant to  
demonstrate a possible way to solve the problem, nothing more.

MfG Kai

Date: 22 Feb 2003 22:08:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: mail vs. news ???
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Ken Murchison)  wrote on 21.02.03 in <[email-masked]>:

> I find it interesting, if not disturbing, that some members of the
> usenet community seem to think that mail messages and usenet articles
> are not the same thing.  AFAICT, from reading the relevant standards,
> writing server code for SMTP/LMTP/IMAP/POP3/NNTP, and everyday use, mail
> messages and news articles both conform to RFC 2822 (RFC 1036 states as
> much).  The only differences that I'm aware of are the following:

Well, obviously news and mail formats are very similar. I don't think  
anybody seriously argues against that.

On the other hand, news and mail do have *some* differing requirements,  
even in the format. Very few people seem to seriously argue with that,  
either.

Saying "they're the same", or "they're different", in this context, at  
least to me just means pointing at either the similarities or the  
differences and asking that they not be forgotten.

Or in other words, people who "find it disturbing that other people seem  
to think they're different" (or the same) are, IMO, misinterpreting what  
they read.

(It's somewhat similar to arguing if men and woman are the same, or  
different. Both are true, of course, depending on how you look at the  
question.)

MfG Kai

Date: Sat, 22 Feb 2003 16:53:39 -0500 (EST)
From: Seth Breidbart <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: mail vs. news ???
Sender: [email-masked]

> I have heard of mail transports that assume a 1-1 correspondence
> between message-ids and body content but I believe this to be highly
> unusual; please correct me if I am mistaken on this point.

I believe (or at least hope) there are mailing-list servers that avoid
duplication on messages sent to multiple lists that way, but that's
not a critical use.   There are certainly procmail files that do it on
the client side.

Seth

Date: Sat, 22 Feb 2003 16:37:23 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
To: Usefor Mailing List <[email-masked]>
CC: IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Andrew Gierth wrote:
> 
> >>>>> "Dave" == Dave Crocker <[email-masked]> writes:
> 
>  AG> Mail systems do not appear to make any actual use of the
>  AG> message-id header other than for logging (and internally in some
>  AG> odd systems).  None of SMTP, POP3 or IMAP ever use message-ids as
>  AG> protocol parameters.
> 
>  Dave> Message-ids are not nearly as deeply and usefully and broadly
>  Dave> used as we had expected them to be.  However, they ARE used, to
>  Dave> good effect, for duplicate detection and for thread
>  Dave> construction.
> 
> at the layer of actual message transport, or just in the clients or
> client-side services such as IMAP threading? I really wasn't referring
> to the clients when I mentioned "mail systems".
> 
> I have heard of mail transports that assume a 1-1 correspondence
> between message-ids and body content but I believe this to be highly
> unusual; please correct me if I am mistaken on this point.

FYI, the LMTP daemon in Cyrus IMAP uses the message-id for duplicate
suppression (and to prevent mail loops) at the time of delivery.  This
isn't done as part of the protocol as in NNTP, but its done internally
for every message before its committed to disk.

To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 21:12:08 +0000
Lines: 38
Sender: [email-masked]

>>>>> "Dave" == Dave Crocker <[email-masked]> writes:

 AG> Mail systems do not appear to make any actual use of the
 AG> message-id header other than for logging (and internally in some
 AG> odd systems).  None of SMTP, POP3 or IMAP ever use message-ids as
 AG> protocol parameters.

 Dave> Message-ids are not nearly as deeply and usefully and broadly
 Dave> used as we had expected them to be.  However, they ARE used, to
 Dave> good effect, for duplicate detection and for thread
 Dave> construction.

at the layer of actual message transport, or just in the clients or
client-side services such as IMAP threading? I really wasn't referring
to the clients when I mentioned "mail systems".

I have heard of mail transports that assume a 1-1 correspondence
between message-ids and body content but I believe this to be highly
unusual; please correct me if I am mistaken on this point.

There is absolutely no comparison between the extent to which
message-ids are used in mail and the extent to which they are used in
news. Their importance to the news transport is _in addition to_ uses
such as threading which are common with mail. Duplicate detection is
not something which is optionally performed by clients or user
delivery agents but something which is built right into the core of
the system. Every news server, regardless of the type of transport it
uses, has to keep a database of message-ids seen. A fullfeed transit
server with a lot of peers might receive over 200 million CHECK
commands per day (that's over 2300 per second), all containing a
message-id that must be looked up in the local history database. Some
news clients use the fetch-by-message-id commands of NNTP for _all_
their article retrieval. All NNTP article retrievals, whether by ID or
number, contain the message-id of the retrieved article in the command
response. And so on.

-- 
Andrew.

Date: Sat, 22 Feb 2003 13:04:48 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Russ Allbery <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: mail/news unification
Sender: [email-masked]

Russ,

Friday, February 21, 2003, 1:57:44 PM, you wrote:
RA> I personally am very strongly in favor of the unification of messaging
RA> formats,

Email "versus" Netnews debates have existed since the mid-70s.  (I count
the early spate of textual, messaging-oriented collaborations systems
that came out of the gas crisis of the early 70s as roughly equivalent
to netnews.)

I believe the basic problem with the debates has been a focus on
mechanism rather than human communication.  As soon as we focus on the
human functions, we can identify only a few, salient differences, I
believe.

To me, the only user-level difference that is major is one of
addressing, between explicitly sending to one or more individuals,
versus explicitly sending to a "group". In this regard, I believe that
that netnews is essentially the same as an email list.

Yes, one subscribes to a list but not to a newsgroup. However think
about the mode of posting to an open mailing list that you are not
subscribed to, and using the mailing list archive as the basis for
reading contributions.  It is darn close to the style of a newsgroup.

Imagine that Internet mail had a construct for labeling a mailing list.
(One might even hypothesize an additional addressing header, such as
"Follow-up" or... "Newsgroup", or more email-list styled as "List-To" or
the like.)

The infrastructure difference that is massive is, of course, the
relaying technology.  Netnews uses a flooding algorithm and mail is
multicast.   I have a suspicion there is some sort of opportunity for
convergence, there, but have not tried to think of how.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.866-358-5301

To: Mark Crispin <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 20:47:06 +0000
Lines: 109
Sender: [email-masked]

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 > On Sat, 22 Feb 2003, Andrew Gierth wrote:
 >> Mail systems do not appear to make any actual use of the
 >> message-id header other than for logging (and internally in some
 >> odd systems).

 Mark> Huh?  Message-ID, In-Reply-To, and References are heavily used
 Mark> in mail for threading.  IMAP even has a THREAD facility that
 Mark> stipulates how to use these.

perhaps I was insufficiently precise: I was referring specifically to
the transport & distribution of messages rather than the clients.

 >> None of SMTP, POP3 or IMAP ever use message-ids as protocol
 >> parameters.  News, on the other hand, uses message-ids as protocol
 >> parameters _all the time_, both for readers accessing messages and for
 >> server-to-server transfer.

 Mark> Let's try to translate your comment into something useful.  Is it your
 Mark> contention that:
 Mark> 	Some set of commands in NNTP use message-id as an argument.
 Mark> 	The syntax for that command precludes the use of space in
 Mark> 	that argument, and/or NNTP lacks the type of quoting mechanism
 Mark> 	found in IMAP and SMTP.

 Mark> If so, please provide the details.

the NNTP spec can be found in RFC 977 (and the "Common extensions"
document, RFC 2980 if I recall correctly)

Message-ids appear in NNTP as both command parameters and response
parameters (neither of which can contain spaces, though this isn't
spelled out for command parameters - it's just a consequence of the
fact that spaces separate parameters and there is no quoting or
escaping mechanism). For ARTICLE/HEAD/BODY/STAT, the command parameter
can be either a message-id or an article number (or omitted), but the
response string always contains the message-id as a parameter. For the
CHECK/TAKETHIS extension, the message-id must be given as a parameter
and is returned in the result, and the appearance of the id in the
result is always used by the sending server (to match up commands and
responses, since CHECK/TAKETHIS are used for pipelined transfer). For
IHAVE, the message-id is a parameter but does not necessarily appear
in the result.

Breakage on CHECK/TAKETHIS is particularly bad because the resulting
syntax error response does not allow the sending server to properly
identify the failing message; this usually halts the flow of articles
on the affected link as the sending server continually retries the bad
article (until the sending system's admin notices and manually deletes
the bad entry from the outgoing queue). This is why it's important for
servers to actively reject bad message-ids upfront rather than try and
propagate them.

The length limitation of 250 is from existing software rather than
from previous standards (NNTP restricts the total length of a
command-line to 512 including the CRLF). 250 is a better choice
because (a) values that long do not legitimately occur in practice,
(b) there is no particular reason why excessively long values should
be allowed, especially given the large number of CHECK commands, (c)
the shorter limit allows for extensions to add parameters to commands
such as CHECK without also having to extend the command-line limit.

 >> (and, FWIW, I checked about a hundred thousand message-ids taken from
 >> mail messages here, and only one of those had whitespace in and that
 >> was a Chinese spam which had four spaces in place of the domain-part,
 >> almost certainly due to misconfigured or broken spamming software.

 Mark> That isn't an example of a syntactically valid RFC 2822 message-id.

it wasn't intended to be. I found no examples of syntactically valid
RFC 2822 message-ids in my mail archives which were not also valid for
news.

 Mark> And the IETF/IESG is supposed to respect this?

 >> yes

 Mark> If NNTP has a limitation such as suggested above, then that
 Mark> would add a transport imposed requirement to a document that
 Mark> otherwise uses RFC 2822 and MIME as normative.  That is
 Mark> something that can be respected.

 Mark> That is different from duplicating all of RFC 2822 and MIME
 Mark> just to insert transport imposed requirements.  Doing so, which
 Mark> the Usefor document has done, is ludicrous.

I entirely agree (and you'll find a long string of comments to that
effect from me in the Usefor archive).

 Mark> What's worse, it has the highly undesirable effect of cloaking
 Mark> the transport-imposed requirements.

 Mark> A small document, such as Kohn's draft, which has a section of
 Mark> "NNTP transport requirements" would show these requirements in
 Mark> clear contrast.

it is not, in fact, merely an NNTP issue; message-ids are a key part
of _every_ news transport method (since they are used to detect
duplicates).  Every news server has to keep a history database (or the
moral equivalent) indexed by message-id, whether it uses NNTP or not.
Message-ids are passed around a lot internally, stored in queue files,
listed in ihave/sendme control messages for UUCP feeding, appear as
parameters in control messages, and generally pervade the entire
structure of news; whitespace or excessive length breaks things in
really quite a lot of places.

-- 
Andrew.

Date: Sat, 22 Feb 2003 11:31:03 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: Re: mail vs. news ???
To: Russ Allbery <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Sender: [email-masked]

On Sat, 22 Feb 2003 11:05:19 -0800, Russ Allbery wrote:
> > 	Some set of commands in NNTP use message-id as an argument.
> > 	The syntax for that command precludes the use of space in
> > 	that argument, and/or NNTP lacks the type of quoting mechanism
> > 	found in IMAP and SMTP.
> Your statement above is correct.

If this is the case, then we add a small section to the Kohn document of the
form:

NNTP Transport Requirements

   The BLURDYBLOOP command in NNTP uses the message-id as an argument.
   The syntax for this command does not provide any way for spaces to
   appear in the message-id.  Therefore, message-ids in the Message-ID,
   In-Reply-To, and References headers of a message MUST NOT contain
   the space character.

Implementation Recommendations

   Some widely-used NNTP implementations are known to core dump, reformat
   reformat the hard drive, and watch Joe Millionaire on TV if given a Date:
   header containing a symbolic timezone in a coment, e.g.
	Date: Sat, 22 Feb 2003 10:26:00 -0800 (PST)
   It is recommended that such comments be removed prior to transmitting
   a message via NNTP.

etc. etc. as needed.

To: Cyrus Daboo <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 19:07:01 +0000
Lines: 23
Sender: [email-masked]

>>>>> "Cyrus" == Cyrus Daboo <[email-masked]> writes:

 > | Mail systems do not appear to make any actual use of the message-id
 > | header other than for logging (and internally in some odd systems).

 Cyrus> Message-ID, In-Reply-To and References are an integral part of
 Cyrus> 'threading' email messages - its not just news. Email clients
 Cyrus> that do threading will typically use those. The IMAP THREAD
 Cyrus> extension also uses those.

that's primarily a client issue, or an issue of returning values from
the server (as, for example, XOVER does in news).

The more important use of message-ids in news is for clients to
request articles from the server, and for servers to offer articles to
each other - all this is done by means of message-ids. News servers
must therefore index stored articles by message-id, and message-ids
appear all the time in NNTP commands and responses. Therefore both the
no-whitespace and maximum length restrictions serve a necessary
purpose.

-- 
Andrew.

To: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Cc: Mark Crispin <[email-masked]>
Subject: Re: mail vs. news ???
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Sat, 22 Feb 2003 11:05:19 -0800
Lines: 28
Sender: [email-masked]

Mark Crispin <[email-masked]> writes:

> Let's try to translate your comment into something useful.  Is it your
> contention that:

> 	Some set of commands in NNTP use message-id as an argument.
> 	The syntax for that command precludes the use of space in
> 	that argument, and/or NNTP lacks the type of quoting mechanism
> 	found in IMAP and SMTP.

> If so, please provide the details.  If confirmed, that is useful and can
> be stated as a requirement.

Your statement above is correct.

> If NNTP has a limitation such as suggested above, then that would add a
> transport imposed requirement to a document that otherwise uses RFC 2822
> and MIME as normative.  That is something that can be respected.

Precisely.

> That is different from duplicating all of RFC 2822 and MIME just to
> insert transport imposed requirements.

Yes, I'm also not in favor of doing this.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sat, 22 Feb 2003 10:34:36 -0800
From: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: Andrew Gierth <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Andrew,

Saturday, February 22, 2003, 2:05:34 AM, you wrote:
AG> Mail systems do not appear to make any actual use of the message-id
AG> header other than for logging (and internally in some odd systems).
AG> None of SMTP, POP3 or IMAP ever use message-ids as protocol
AG> parameters.

Message-ids are not nearly as deeply and usefully and broadly used as we
had expected them to be.  However, they ARE used, to good effect, for
duplicate detection and for thread construction.

When looking for utility, please include the layer ABOVE the usual IETF
protocols.  Think of it as the User-to-User protocol.  It's the one that
users actually care about.  The rest is just support infrastructure.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 t +1.408.246.8253; f +1.866-358-5301

Date: Sat, 22 Feb 2003 09:41:54 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: Andrew Gierth <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Organization: Networks and Distributed Computing
Sender: [email-masked]

On Sat, 22 Feb 2003, Andrew Gierth wrote:
> Mail systems do not appear to make any actual use of the message-id
> header other than for logging (and internally in some odd systems).

Huh?  Message-ID, In-Reply-To, and References are heavily used in mail for
threading.  IMAP even has a THREAD facility that stipulates how to use
these.

> None of SMTP, POP3 or IMAP ever use message-ids as protocol
> parameters.  News, on the other hand, uses message-ids as protocol
> parameters _all the time_, both for readers accessing messages and for
> server-to-server transfer.

Let's try to translate your comment into something useful.  Is it your
contention that:
	Some set of commands in NNTP use message-id as an argument.
	The syntax for that command precludes the use of space in
	that argument, and/or NNTP lacks the type of quoting mechanism
	found in IMAP and SMTP.

If so, please provide the details.  If confirmed, that is useful and can
be stated as a requirement.  It isn't clear to me that it's a requirement
for header syntax; rather, it's a transport imposed requirement.  The
effect is the same (you can't use message-ids with spaces in news), but
it's stated in the right place.

Note that there is quite a bit in RFC 2822 that is not permissible in mail
due to transport imposed requirements.  There was even more in RFC 822.
That doesn't mean that mail can not use RFC 2822.

> (and, FWIW, I checked about a hundred thousand message-ids taken from
> mail messages here, and only one of those had whitespace in and that
> was a Chinese spam which had four spaces in place of the domain-part,
> almost certainly due to misconfigured or broken spamming software.

That isn't an example of a syntactically valid RFC 2822 message-id.

>  Mark> And the IETF/IESG is supposed to respect this?
> yes

If NNTP has a limitation such as suggested above, then that would add a
transport imposed requirement to a document that otherwise uses RFC 2822
and MIME as normative.  That is something that can be respected.

That is different from duplicating all of RFC 2822 and MIME just to insert
transport imposed requirements.  Doing so, which the Usefor document has
done, is ludicrous.  What's worse, it has the highly undesirable effect of
cloaking the transport-imposed requirements.

A small document, such as Kohn's draft, which has a section of "NNTP
transport requirements" would show these requirements in clear contrast.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: Sat, 22 Feb 2003 08:33:50 -0800 (PST)
From: [email-masked]
Subject: Re: mail vs. news ???
To: Andrew Gierth <[email-masked]>
Cc: Mark Crispin <[email-masked]>, Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Sender: [email-masked]

> >>>>> "Mark" == Mark Crispin <[email-masked]> writes:

>  Mark> Now we hear the claim that completely unnecessary restictions
>  Mark> in headers are necessary because of news software.

> who says those restrictions are unnecessary?

> Mail systems do not appear to make any actual use of the message-id
> header other than for logging (and internally in some odd systems).
> None of SMTP, POP3 or IMAP ever use message-ids as protocol
> parameters.  News, on the other hand, uses message-ids as protocol
> parameters _all the time_, both for readers accessing messages and for
> server-to-server transfer.

FWIW, draft-ietf-imapext-thread-12.txt is the obvious counterexample. But even
if what you say is true about the present email transport infrastructure, there
are plenty of email clients that support threading via message-id.

> (and, FWIW, I checked about a hundred thousand message-ids taken from
> mail messages here, and only one of those had whitespace in and that
> was a Chinese spam which had four spaces in place of the domain-part,
> almost certainly due to misconfigured or broken spamming software. So
> this clearly isn't a feature that anyone actually _needs_, or probably
> even wants.)

Which of course is illegal according to any of the relevant standards.

Mind you, I'm not saying that additional restrictions on message-ids in news
are inappropriate. At a minimum the restrictions of RFC 1036 have to be taken
into account. However, I do urge caution when attempting to cater to failure to
support the existing specifications in the installed base, however. This is a
game that has to be played very carefully.

				Ned

Date: Sat, 22 Feb 2003 11:37:08 -0500
From: Cyrus Daboo <[email-masked]>
To: Andrew Gierth <[email-masked]>, Mark Crispin <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Hi Andrew,

--On Saturday, February 22, 2003 10:05 AM +0000 Andrew Gierth 
<[email-masked]> wrote:

| Mail systems do not appear to make any actual use of the message-id
| header other than for logging (and internally in some odd systems).

Message-ID, In-Reply-To and References are an integral part of 'threading' 
email messages - its not just news. Email clients that do threading will 
typically use those. The IMAP THREAD extension also uses those.

-- 
Cyrus Daboo

To: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 13:21:36 +0000
Lines: 41
Sender: [email-masked]

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 > Andrew Gierth <[email-masked]> writes:
 >> but this proposal as it stands largely cripples the ability for
 >> clients to do "LIST ACTIVE <wildmat>" unless _both_ the client and
 >> the server are updated to support an extension for
 >> punycode-wildmats.  (i.e. without such an extension, the client
 >> can't do pattern matches in ways that are meaningful in terms of
 >> the human-readable name, only in terms of the encoded name)

 Russ> I'd thought about this, but the degree of crippling didn't seem
 Russ> severe enough to even mention.  Per-component wildmat patterns
 Russ> still work, and within-component matches of ASCII groups
 Russ> continue to work, so this only affects cases where the client
 Russ> is trying to use a wildmat to match a portion of a non-ASCII
 Russ> name.

 Russ> This is an extremely uncommon operation for a reader as I
 Russ> understand it,

I have no data on whether or not this is common. I know that one of
the clients I use does it when I ask it to (even though it
theoretically has a local copy of the list available), usually when
I'm looking for some group the name of which I'm not certain of.

To gather data on this I would have to add some additional logging
into my reader servers, which isn't something I can do in the short
term.

 Russ> How frequently do you see clients using wildmats other than *,
 Russ> the exact name of a group, or component-wise?  I thought that
 Russ> most clients that supported a newsgroup search feature did so
 Russ> by downloading the entire active file and then doing their
 Russ> searches locally.

well, if I were writing a client I would not necessarily do it that
way, if for no other reason than that the entire active file is often
uncomfortably large.

-- 
Andrew.

To: usenet-format <[email-masked]>, [email-masked]
Subject: Re: Kohn draft (was RE: USEFOR: Current situation and next steps)
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 10:45:45 +0000
Lines: 60
Sender: [email-masked]

[adding ietf-822 back in as this is getting into their territory]

>>>>> "Bruce" == Bruce Lilly <[email-masked]> writes:

 >> RFC1036 specifies the syntax for the Message-ID header as follows:
 >> 
 >> message-id-char = <any US-ASCII char in the range 0x21 - 0x7E except 0x3E)
 >> 
 >> message-id = "<" 1*message-id-char ">"
 >> 
 >> with the additional constraint that message-ids must match the syntax
 >> of RFC 822 _in addition_ to that of 1036.

 Bruce> That's not precisely what it says, but close enough for many
 Bruce> purposes.

it certainly implies (both by the comments in that section and by the
general statement at the start of the document) that the 822 syntax
must be followed in addition to the 1036 syntax, and what I wrote
above is just the combination of what 1036 does say about message-ids.

 Bruce> There are some subtlr differences, though; 822 rules out
 Bruce> specials except for quoted-strings, while 1036 says nothing at
 Bruce> all about specials or double-quotes. So apparently
 Bruce>    <<;:,"..[]<()@foo.com>
 Bruce> would be considered valid according to 1036, bur not 822.

correct, though several systems (such as INN) will reject it, and some
systems that would otherwise handle it correctly will reject it for
local posts (only).

 >> what transition plan?

 Bruce> If '/' is now to be unrestricted, there should be mention of
 Bruce> that as a warning to implementors of the change.

Since that warning was already in 1036 (which is what, 16 years old
now?), this seems superfluous.

 Bruce> Ideally, the differences which 1036 stipulated on top of 822
 Bruce> ought to be phased out,

 >> arguably they should instead be adopted by the mail standards.

 Bruce> Perhaps, if a convincing case can be made.

2822 did specifically make the message-id syntax more restrictive than
822 (822 allowed whitespace in quoted strings, 2822 only allows it in
quoted-pair (i.e. preceeded by \); I do not know if the latter was
intentional on the part of the DRUMS group, I certainly missed it on
several occasions)

Message-ids that are valid under 2822 but not valid under 1036 (or are
longer than the 250-char limit) seem not to occur in mail with any
frequency (at least I don't see any in those mail collections I have
searched). I would be interested in whether anyone with access to large
amounts of real-world email can confirm this.

-- 
Andrew.

To: Mark Crispin <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Andrew Gierth <[email-masked]>
Date: 22 Feb 2003 10:05:34 +0000
Lines: 27
Sender: [email-masked]

>>>>> "Mark" == Mark Crispin <[email-masked]> writes:

 Mark> Now we hear the claim that completely unnecessary restictions
 Mark> in headers are necessary because of news software.

who says those restrictions are unnecessary?

Mail systems do not appear to make any actual use of the message-id
header other than for logging (and internally in some odd systems).
None of SMTP, POP3 or IMAP ever use message-ids as protocol
parameters.  News, on the other hand, uses message-ids as protocol
parameters _all the time_, both for readers accessing messages and for
server-to-server transfer.

(and, FWIW, I checked about a hundred thousand message-ids taken from
mail messages here, and only one of those had whitespace in and that
was a Chinese spam which had four spaces in place of the domain-part,
almost certainly due to misconfigured or broken spamming software. So
this clearly isn't a feature that anyone actually _needs_, or probably
even wants.)

 Mark> And the IETF/IESG is supposed to respect this?

yes

-- 
Andrew.

Date: Fri, 21 Feb 2003 22:03:31 -0800 (PST)
From: [email-masked]
Subject: Re: Unicode newsgroup name options
Cc: [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > I also note that Dan Kohn's alternative News Article Format draft uses
> > punycode, which already includes a stringprep profile.

> Actually, Punycode does not include a Stringprep profile, or depend on
> one in any way.  Punycode maps *any* Unicode string to an ASCII string,
> reversibly (without loss).

Point taken.

				Ned

Date: Fri, 21 Feb 2003 20:31:24 -0800 (Pacific Standard Time)
From: Mark Crispin <[email-masked]>
To: Terje Bless <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Networks & Distributed Computing
Sender: [email-masked]

On Sat, 22 Feb 2003, Terje Bless wrote:
> I do not know the situation on other platforms, but on Mac OS X the system
> libraries -- including GUI frameworks -- operate on UTF-16 strings
> internally and support UNICODE internally. I would expect this to be common
> on most platforms today or in the very near future. IOW, this whole issue
> is likely to be allready solved by the host platform for the "GUI programs
> and screen editors".

I suggest that you read up on stringprep.  There's quite a lot that needs
to be done that isn't done by these libraries.

> There are other parts of the infrastructure where I would worry far more
> about the "width and length" of characters.

Even if the OS and libraries support Unicode, the application programs
still need to use those routines.  You can't expect to take an old program
and have it magically become Unicode-savvy.

All you have to do is look at Windows, which has been Unicode savvy for
years longer than Mac.  Most applications on Windows are still not Unicode
savvy.

A Unicode savvy application has to be designed to be Unicode savvy from
the ground up.  As soon as you get into GUIs, you have all sorts of other
wonderful details to worry about.  You actually have to alter the layout
of your GUI on a script-dependent basis.  No amount of OS libraries will
do this for you.

There are some simplifying assumptions that you may be able to make for
newsgroup names.  Maybe.  It won't work for text.

It ain't just a matter of wider characters and bigger fonts.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: Sat, 22 Feb 2003 02:02:23 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Unicode newsgroup name options
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

[email-masked] wrote:

> I also note that Dan Kohn's alternative News Article Format draft uses
> punycode, which already includes a stringprep profile.

Actually, Punycode does not include a Stringprep profile, or depend on
one in any way.  Punycode maps *any* Unicode string to an ASCII string,
reversibly (without loss).

The ASCII-compatible encoding (ACE) defined by IDNA is a combination of
Punycode, Nameprep (a profile of Stringprep), a prefix (xn--), and a few
other checks.  Unlike Punycode, ACE deliberately rejects some Unicode
strings and loses some information (to make comparisons behave better).

AMC

Date: Fri, 21 Feb 2003 17:13:00 -0800 (PST)
From: [email-masked]
Subject: Re: Unicode newsgroup name options
To: Mark Crispin <[email-masked]>
Cc: Russ Allbery <[email-masked]>, [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> The wildmat problem is a red herring.  Wildmat implementations need to be
> cognizant of Unicode in far more substantial ways than merely overcoming
> punycode issues.  A well-thought-out stringprep requirement will help
> some, but then the stringprep has to be implemented.

This last point of Mark's is worth noting. Regardless of whether you end up
with UTF-8, punycode, UTF-7, or some other encoding of Unicode, a stringprep
profile is going to be required in order to move forward. It can either be
specified directly or indirectly as part of the underlying encoding, but it has
to be there.

Part of the problem with having a group take so long to complete its work is
that what's minimally acceptable to the IETF/IESG changes over time. It used to
be that protocol could just use some encoding of Unicode, end of story. No
more. Now that stringprep exists the IESG requires it be used when Unicode
protocol elements are involved. Several recent specifications have been
returned to their respective working groups because they didn't do this.

I note in passing that while the current News Article Format Draft talks about
normalization of group names, it does so without reference to stringprep. This
would need to change: Stringprep covers stuff besides normalization, and more
generally provides a checklist for Unicode usage each protocol needs to
consider.

I also note that Dan Kohn's alternative News Article Format draft uses
punycode, which already includes a stringprep profile.

				Ned

To: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 21 Feb 2003 15:52:07 -0800
Lines: 23
Sender: [email-masked]

Ken Murchison <[email-masked]> writes:

> OK.  As I suspected there is nothing inherent in RFC [2]822 that makes
> it unsuitable for news.

Spaces in message IDs make them unsuitable for news.  This really, really
does break things, honest, I swear.  I'm not just making this up.  :)

In general, the statements in RFC 1036 putting additional limitations on
the article format *are* implemented on Usenet and *are* relied on in
practice by Usenet software.

But, rewriting your statement to say "there is nothing inherent in RFC
2822 with RFC 1036 limitations applied that makes it unsuitable for news,"
I agree.

> So, the claims that mail and news are not the same was either misleading
> and/or wishful thinking.

That is quite certainly my opinion.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 21 Feb 2003 18:47:45 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
To: Usefor Mailing List <[email-masked]>
CC: IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Sender: [email-masked]

Russ Allbery wrote:
> 
> Ken Murchison <[email-masked]> writes:
> 
> > I find it interesting, if not disturbing, that some members of the
> > usenet community seem to think that mail messages and usenet articles
> > are not the same thing.  AFAICT, from reading the relevant standards,
> > writing server code for SMTP/LMTP/IMAP/POP3/NNTP, and everyday use, mail
> > messages and news articles both conform to RFC 2822 (RFC 1036 states as
> > much).  The only differences that I'm aware of are the following:
> 
> > - usenet puts a greater restriction on the headers (although still being
> > RFC 2822 compliant)
> 
> Yes, although there are a variety of subtle things that fall into this
> category that one has to be careful of.  That's one of the difficulties

[...]

> - National 8-bit character sets are in widespread use in Usenet message
> headers, possibly more widespread than they are in (non-spam) mail
> messages.  Untagged 8-bit national character sets are widely used in

[...]

> - MIME never got very much uptake on Usenet for attachments.  The binary
> newsgroups are almost universally uuencode or, these days, yEnc.  Base64

[...]

OK.  As I suspected there is nothing inherent in RFC [2]822 that makes
it unsuitable for news.  Any incompatibilities are purely the result of
non-standardized stuff being implemented in news software.

So, the claims that mail and news are not the same was either misleading
and/or wishful thinking.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

To: Lawrence Greenfield <[email-masked]>
Cc: [email-masked], Kai Henningsen <[email-masked]>, [email-masked]
Subject: Re: Unicode newsgroup name options
From: Simon Josefsson <[email-masked]>
Date: Fri, 21 Feb 2003 23:49:04 +0100
Sender: [email-masked]

Lawrence Greenfield <[email-masked]> writes:

>    Date: 21 Feb 2003 22:11:00 +0200
>    From: [email-masked] (Kai Henningsen)
> [...]
>    UTF-7 has, however, one important problem: it uses + as the escape  
>    character. My active file has over 200 groups with a + in their name.
>
> [... reinventing IMAP4's modified UTF-7 elided ...]
>
> You might want to read the IMAP archives where the decision to use
> modified UTF-7 for mailbox names was made and some of the feedback
> there. The IMAP mUTF-7 goals are close enough to yours that you could
> just adapt it and at least make one group (IMAP client/servers)
> easier. But the IMAP community in general seems like it has been less
> than thrilled about the mUTF-7 mailbox names.

I wonder if they would have been more thrilled about using punycode
instead of UTF-7, had it been proposed at the time.

Date: Fri, 21 Feb 2003 14:42:05 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: Russ Allbery <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Networks and Distributed Computing
Sender: [email-masked]

On Fri, 21 Feb 2003, Russ Allbery wrote:
>    We know that some existing software will work with
>    UTF-8 newsgroup names out of the box without modification, although it
>    will require some tweaking for ideal operation.

Step back from the tree to see the forest.  "Some tweaking" includes such
small details as not assuming that one byte means one screen space, which
has all sorts of implications in GUI programs and screen editors.  Don't
forget that you don't just have variable length characters in UTF-8, but
you also have variable width characters in Unicode, plus characters which
interact with other characters in interesting ways.

A bit more than a "tweak".

Stepping back further is the realization that it won't work in any
reasonable way if the program doing the actual display and user input
(whether terminal emulator or the news reading application itself) is not
configured for UTF-8.

>    By comparison,
>    punycode (C) we know won't work correctly with *any* existing software;
>    the only reason why that column is a D instead of Y is that users can
>    use the funny-looking encoded names and still participate in the
>    groups.

Just as "some tweaking" was understated, "won't work" is overstated here.

Punycode names will work perfectly well as ASCII names.  If the program
doing the actual display and use input is not cognizant of punycode, it
will fall back on ASCII display/input that will work in the same way
around the world.

The wildmat problem is a red herring.  Wildmat implementations need to be
cognizant of Unicode in far more substantial ways than merely overcoming
punycode issues.  A well-thought-out stringprep requirement will help
some, but then the stringprep has to be implemented.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: 21 Feb 2003 22:14:19 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: mail vs. news ???
Sender: [email-masked]

Mark Crispin writes:
  [ USENET message-id restrictions ]
> What are those restrictions?

No comments; no spaces inside IDs; etc.

You should be happy about the restrictions. After all, your software has
never handled RFC 822 comments and spaces correctly. For example, I just
fired up Pine 4.44 on the RFC 822 example

   From: Muhammed.(I am the greatest) Ali @(the)Vegas.WBA

and received the error message ``INVALID_ADDRESS@.SYNTAX-ERROR.'' when I
tried to import that address into the Pine address book.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 21 Feb 2003 14:10:01 -0800
Lines: 53
Sender: [email-masked]

Mark Crispin <[email-masked]> writes:

> Now, let's factor out the items in which all three choices are
> equivalent, and the superiority of choice "C" becomes even more
> apparent.

>  | 1   2   3   4   5   6   10  12  13
> -+-----------------------------------
> A| D   C   N   N   D   Y    Y   D   Y
> B| Y   Y   C   Y   Y   N    N   N   C
> C| D   C   C   C   D   N    N   N   C

That was the conclusion that was jumping out at me as well, but here are a
few other things to keep in mind:

 * These columns don't really have equal weight, in that some of them
   represent small numbers of installations of relatively easily changed
   software and some of them represent very large installed bases or
   software that's difficult to change.  In particular, (1) and (5), the
   installed base of news readers, will take a very long time to change,
   and (2), (3), and (4), the news server software, we know is rarely
   updated and upgrades are difficult to motivate.  Usenet server software
   routinely runs for years on autopilot without any maintenance.

   By comparison, (6), the process that sends mail to moderated groups, is
   a small and easily changed component in most situations, and the number
   of moderators (10), news to mail gateways (12), and IMAP servers
   serving news messages (13) are all, while certainly significant, much
   smaller than the number of installations of the core Usenet software.

 * The single largest set of installed software, (1) and (5), is almost a
   C for proposal (A).  We know that some existing software will work with
   UTF-8 newsgroup names out of the box without modification, although it
   will require some tweaking for ideal operation.  By comparison,
   punycode (C) we know won't work correctly with *any* existing software;
   the only reason why that column is a D instead of Y is that users can
   use the funny-looking encoded names and still participate in the
   groups.

   This is one of the stronger arguments in favor of (A), namely that you
   can implement it to a surprising degree without changing any news
   software at all.

 * There are other issues not reflected on this matrix at all, such as
   complexity of implementation and compatibility with the existing
   messaging format, that weigh in various directions.

Again, this is not to disagree with your conclusion.  I just wanted to
point out that while I found the table helpful, it's a bit over-simplified
and hides the nature of some of the issues and tradeoffs.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 21 Feb 2003 13:57:44 -0800
Lines: 164
Sender: [email-masked]

Mark Crispin <[email-masked]> writes:
> On Fri, 21 Feb 2003, Russ Allbery wrote:

>> Usenet's restrictions on the syntax of message ID headers are very
>> specific and very precise, and much stronger than those of RFC 2822, in
>> part because message IDs are used as part of the NNTP protocol.

> What are those restrictions?

The primary ones are:

 * Absolutely no occurences of either whitespace or the ">" character,
   escaped or not, are permitted inside the message ID.  Either is known
   to break existing software in various ways.

 * Nothing is permitted in the Message-ID header other than the message ID
   itself.  Comments either preceding or following the message ID will
   cause the message to be rejected by many news servers.

 * The message ID must not be longer than about 500 characters.  The
   failure mode for violating this rule tends to be rather nasty for some
   existing NNTP software, including things like desynchronization of the
   protocol between the client and server.  NNTP (unfortunately) has a
   maximum command length defined as part of the protocol.  In practice,
   many news servers enforce a 250 octet limit (including the surrounding
   angle brackets).

Please note that I'm not arguing that these restrictions are desirable,
simply that violating them *will* break existing news software.  I also
don't think that fixing one and possibly two is really worth the effort,
since there isn't much in the way of useful purpose served by not
following those rules anyway.

>> Comments in various places that mail supports them are not
>> well-supported by currently deployed Usenet software (although it
>> certainly hurts nothing to support them when writing new code, other
>> than adding complexity).  The space after the colon in headers is not
>> optional on Usenet.  The syntax of the Date header is restricted in
>> ways somewhat similar to that of the Message-ID header.

> Golly gee, where's the chorus of "these are bugs that should be fixed"
> now?

Are you expecting me to serve as the chorus?  I certainly hope that you're
not expecting me to try to be consistent with statements made by other
people that I don't necessarily agree with.  I tend to hold my own
opinions and not necessarily agree with other people.  :)

> First we hear the claim that 7-bit messaging restrictions in mail are a
> "bug that should be fixed" even though 7-bit was specifically in the
> standard.

> Now we hear the claim that completely unnecessary restictions in headers
> are necessary because of news software.

These restrictions are published in RFC 1036, so I would not expect them
to be a surprise to news implementors.  Usenet has, since B news, used a
subset of the mail messaging format.

RFC 1036 is unfortunately imprecise about precisely what additional
restrictions it put on the message format, but at the least the space
after the colon in headers is quite explicit.  The message ID restrictions
are also fairly clear apart from the length limitation (which falls out of
the NNTP protocol instead).  (The bit in RFC 1036 about slashes being
strongly discouraged in message IDs is now completely obsolete.)

The Date specification in RFC 1036 is obnoxious, referring to a particular
software implementation that isn't documented as part of the standard.  In
practice, an RFC 2822 date that doesn't use any of the obsolete syntax is
fine provided that the header is not folded.

Issues surrounding comments are more complex.  Apart from Date and
Message-ID, which are the most sensitive headers that are also shared with
mail, comments in References headers are unlikely to cause catastrophic
problems but may show up as oddities in the thread tree in a news reader
and news software can be fairly picky about the From header (although one
is likely fine as long as one avoids the obsolete syntax rules).

> And the IETF/IESG is supposed to respect this?

My message was solely addressing the differences *in practice* that exist
right now on the wire.  I was not attempting to make any sort of statement
about what the future should like.

I personally am very strongly in favor of the unification of messaging
formats, and think that this is one of the most important things that
could come out of USEFOR.  I think that it's reasonable to simply require
that Usenet software going forward cope with comments in the References
header and with the full From syntax in RFC 2822 (possibly omitting the
obsolete rules, since they have never been supported on Usenet).

I'm ambivalent about folded dates.  The date parsing software that I've
written personally and that is used in the software I maintain supports
them.  I don't understand why anyone would generate a folded date, though,
so I can understand why people don't see what purpose is served in
supporting it.

I think that not requiring a space after the colon in headers (except for
compatibility with older messages) is silly, but I don't have a strong
opinion on it.  Changing news software to support this can be a rather
significant undertaking, however, given that this rule is clearly
specified in RFC 1036 and the assumption tends to be very widespread in
any code that parses headers in news messages.

The message ID restrictions hit the single hottest code path in every
Usenet transit server, and I really don't see any purpose served by
complicating the parsing algorithm for message IDs solely to support
rather questionable constructions that can be easily avoided.  Apart from
that personal opinion, I'll also note that removing those restrictions
would require extremely significant changes to the Usenet infrastructure
and would not be in any sense backwards-compatible; lots of software was
written on the basis of the guarantees provided by RFC 1036.

My primary consideration in the standards work that I do on Usenet article
formats is to support backward compatibility with existing software to the
degree that is feasible.  My secondary consideration is to support
unification of the messaging format in order to get rid of the various
places where gatewaying is difficult for silly and unnecessary reasons.  I
consider tighter integration of Usenet and e-mail to be obviously good, a
growing trend, and one of the more interesting applications for Usenet
technology going forward.  NNTP is an interesting alternative access
protocol for large public archives of mail messages because of its extreme
simplicity and very lightweight nature, although anonymous IMAP is
certainly a strong competitor with its much more advanced searching
support.  (Either is obviously utterly superior to converting all of the
messages to HTML and putting them behind a clumsy web page interface.)
NNTP also has some advantages when it comes to mass distribution of
messages.

> This is because portions of the news community listened to the siren
> song of "just send 8-bits" offered by those individuals who song was
> rejected in mail.  Now the news community has a non-interoperable
> disaster.

It actually works pretty well on Usenet in those hierarchies that have
standardized on a character set.  It breaks down very badly whenever those
messages move outside of a pure NNTP system, but I'm afraid that I can't
agree with your characterization given the number of people who are very
happily using untagged character sets in their own hierarchies.

However, I *do* agree with you that using random untagged 8-bit character
sets is obviously not a solution to the problem.  It is a bad hack that
works in certain limited and specific situations and actively interferes
with movement to a unified messaging format, and Usenet is already running
hard against its limitations.

> But rather than fix the disaster, they seem to want to inflict a new
> disaster upon the email community.

I would greatly appreciate it if you would be somewhat careful about who
you choose to include in sweeping pronouns like "they."

> The solution to interoperability is to stop claiming that news is
> special, and start playing ball with the rest of the messaging world.

And many news implementors have been doing this for years, so please don't
make blanket statements about what everyone on the Usenet side is doing.
The active members of the USEFOR working group are not a representative
sample of Usenet implementors or users, and many of us who strongly
believe in a unified messaging format gave up on USEFOR in disgust years
ago.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 21 Feb 2003 13:32:41 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: Russ Allbery <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Networks and Distributed Computing
Sender: [email-masked]

On Thu, 20 Feb 2003, Russ Allbery wrote:
>   | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
>  -+------------------------------------------------------
>  A| D   C   N   N   D   Y   N   N   N   Y   N   D   Y   D
>  B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   C   D
>  C| D   C   C   C   D   N   N   N   N   N   N   N   C   D

Choice "A" has three "Y"s, four "D"s, and one "C".

Choice "B" has with four "Y"s, one "D", and two "C"s.

Choice "C", the ugly duckling, has no "Y"s, three "D"s, and four "C"s.

Choice "A" has one fewer "N" than "B" or "C".

If you believe in the premise that a "Y" is more expensive than a "D", and
a "D" is more expensive than a "C", then choice "C" (punycode everywhere)
stands out as the preferable choice.  The more expensive that you rate a
"Y" compared to a "D" or "C", the better than choice "C" looks.

Now, let's factor out the items in which all three choices are equivalent,
and the superiority of choice "C" becomes even more apparent.

 | 1   2   3   4   5   6   10  12  13
-+-----------------------------------
A| D   C   N   N   D   Y    Y   D   Y
B| Y   Y   C   Y   Y   N    N   N   C
C| D   C   C   C   D   N    N   N   C

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: Fri, 21 Feb 2003 16:20:50 -0500
From: Lawrence Greenfield <[email-masked]>
To: [email-masked], Kai Henningsen <[email-masked]>
Cc: [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

   Date: 21 Feb 2003 22:11:00 +0200
   From: [email-masked] (Kai Henningsen)
[...]
   UTF-7 has, however, one important problem: it uses + as the escape  
   character. My active file has over 200 groups with a + in their name.

[... reinventing IMAP4's modified UTF-7 elided ...]

You might want to read the IMAP archives where the decision to use
modified UTF-7 for mailbox names was made and some of the feedback
there. The IMAP mUTF-7 goals are close enough to yours that you could
just adapt it and at least make one group (IMAP client/servers)
easier. But the IMAP community in general seems like it has been less
than thrilled about the mUTF-7 mailbox names.

Larry

Date: Fri, 21 Feb 2003 13:19:41 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: Russ Allbery <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Organization: Networks and Distributed Computing
Sender: [email-masked]

On Fri, 21 Feb 2003, Russ Allbery wrote:
> Usenet's restrictions on the syntax of message ID headers are very
> specific and very precise, and much stronger than those of RFC 2822, in
> part because message IDs are used as part of the NNTP protocol.

What are those restrictions?

> Comments
> in various places that mail supports them are not well-supported by
> currently deployed Usenet software (although it certainly hurts nothing to
> support them when writing new code, other than adding complexity).  The
> space after the colon in headers is not optional on Usenet.  The syntax of
> the Date header is restricted in ways somewhat similar to that of the
> Message-ID header.

Golly gee, where's the chorus of "these are bugs that should be fixed"
now?  First we hear the claim that 7-bit messaging restrictions in mail
are a "bug that should be fixed" even though 7-bit was specifically in the
standard.

Now we hear the claim that completely unnecessary restictions in
headers are necessary because of news software.

And the IETF/IESG is supposed to respect this?

> - National 8-bit character sets are in widespread use in Usenet message
> headers, possibly more widespread than they are in (non-spam) mail
> messages.  Untagged 8-bit national character sets are widely used in
> various non-English hierarchies in headers as the preferred way of
> including such content, and in some cases use of RFC 2047 is frowned on.

This is because portions of the news community listened to the siren song
of "just send 8-bits" offered by those individuals who song was rejected
in mail.  Now the news community has a non-interoperable disaster.

But rather than fix the disaster, they seem to want to inflict a new
disaster upon the email community.

The solution to interoperability is to stop claiming that news is special,
and start playing ball with the rest of the messaging world.  This means
making compromises, including at times accepting what seems to be
unnecessary limitations, in order to achieve interoperability.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: 21 Feb 2003 22:11:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
cc: [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Russ Allbery)  wrote on 21.02.03 in <[email-masked]>:

> Andrew Gierth <[email-masked]> writes:
>
> > there is a specific reason to avoid punycode in my proposal (proposal
> > A), which is that it's not easily generated in a simple script; the
> > easiest way to upgrade several existing news software packages to handle
> > moderated non-ASCII groups is to wrap the mail-to-moderator program in a
> > script to do the necessary header changes. (This works for INN, Diablo,
> > my server, and at least some versions of DNews on some platforms -
> > basically any server which uses the common approach of doing
> > mail-to-moderator as an external program call.)
>
> Good point.  That should have been mentioned in my summary.

I just had a look at the punycode draft. What a horrible encoding.

Some people were saying that we should use punycode to avoid using UTF-7,  
which nobody wants to see more of.

I certainly dislike UTF-7.

But if I had to choose between UTF-7 and punycode, I would not hesitate a  
second to use UTF-7. It is *much* less problematic.

As to speed, UTF-7 is essentially a slight variation of base64 for non- 
ASCII characters. Punycode - as has been mentioned - does a lot of  
shifting around during the encoding. I think there can be no argument that  
UTF-7 is significantly faster than punycode.

UTF-7 has, however, one important problem: it uses + as the escape  
character. My active file has over 200 groups with a + in their name.

I'm not particularly happy with the idea, but we _could_ define yet  
another variant of UTF-7 which uses a different escape character - say,  
the = that the UTF-7 RFC said would have been nicer except for 2047  
wanting to use it for themselves (for Q encoding, and why would you use  
UTF-7 with B encoding?).

Frankly, I think punycode stinks. Its only excuse is that the legal DNS  
hostname character set is *really* small (with - as the *only* non- 
alphanumeric); I don't see that we have to be quite as restricted with  
newsgroup names.

Ok, so let's see what else could be bad about UTF-7. Well, it allows  
different encodings for the same string (you can but need not encode  
specific characters). We'd obviously have to outlaw that. Also, it encodes  
UTF-16.

Hmm.

Ok, so here's a strawman encoding proposal based on the ideas of UTF-7 but  
not on its actual definition.

Looking at my active file again, all groups in there use the set
a-zA-Z0-9.+_- - that's 66 characters. For another reference, base64 uses
a-zA-Z0-9./=

/ is bad for people who convert newsgroup names to filenames and don't  
expect it.

So:

Anything in the set a-zA-Z0-9.+_- shall be unencoded.
Anything outside that set shall be encoded as follows:

* First, encode the character as UTF-32. Every such character is thus an  
Unicode character in the range 0-0x10ffff (actually some of those won't be  
used, such as the unencoded chars above and chars not legal in UTF-32.)

* Then, encode each character in base64 as if it were 24 bits (where at  
least the three most significant bits are zero), in network bit order  
(most signifiant bit first). Thus, each character gives four base64  
characters.

* Then replace any / character with a _ character.

* finally, whenever switching from unencoded characters to encoded  
characters or back, insert a = character.

Assuming I made no mistake, that makes our beloved test group name into

        dk.test.utf8-=AADmAAD4AADl

(only one = because it doesn't switch back).

As an alternate example, hypothetical alt.fan.claus-f�rber would become

        alt.fan.claus-f=AADk=rber

Note that this works just fine with *-only wildmat matching even inside a  
component! And it's just as readable as a punycode-encoded version -  
possibly more so because you can see where the encoded chars belong.

Sorry, no non-Latin1-covered example.

Yes, it does look ugly. So does every other encoding.

Yes, it could be compressed. I don't believe it's worth it. I prefer  
simple.

Oh, yes. We need one additional rule: handle the begin of the string as if  
it was preceded by an unencoded character, that is, start with a = if the  
first character of the top-level hierarchy isn't unencoded. (It's bound to  
happen.)

MfG Kai

Date: Fri, 21 Feb 2003 12:58:31 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: Russ Allbery <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Organization: Networks and Distributed Computing
Sender: [email-masked]

On Fri, 21 Feb 2003, Russ Allbery wrote:
> D J Bernstein <[email-masked]> writes:
> > Actually, there's very little opposition (especially among implementors)
> > to requiring all MTAs, MUAs, etc. to handle UTF-8 messages. Eventually
> > we will all be using UTF-8; all relevant bugs must be fixed. Only the
> > wildest ``7 bits forever!'' proponents, such as Keith Moore, disagree.

I disagree, in the very strongest terms, with both this categorization of
the situation and with the personal attack.

> > Many of us (especially implementors) believe that these short-term 7-bit
> > kludges have huge costs (as illustrated by your message) and miniscule
> > benefits. We believe that the 7-bit kludges should be dropped.

The definition of "many" and "we" is extremely subjective, particularly as
it is a minority opinion.

> One additional option that I didn't mention would be to decide that the
> IETF standards process is out of touch with the reality of what
> implementors want

The people who advocate this option are those individuals who have not
been able to get their way.  I strongly suggest that you plug your ears
when you hear this siren song, lest you cause a disaster for yourself and
wreak havoc on others.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

Date: Fri, 21 Feb 2003 12:36:09 -0800
From: Brad Templeton <[email-masked]>
To: Ken Murchison <[email-masked]>
Cc: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
Mail-Followup-To: Brad Templeton <[email-masked]>, Ken Murchison <[email-masked]>, Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: http://www.templetons.com/brad
Sender: [email-masked]

On Fri, Feb 21, 2003 at 09:46:35AM -0500, Ken Murchison wrote:
> - usenet puts a greater restriction on the headers (although still being
> RFC 2822 compliant)

And of course has its own extra headers.

> - mail messages are typically tranmitted over a 1-to-1 protocol (SMTP)
> and news articles are typically transmitted over a 1-to-many protocol
> (NNTP)

That is the key difference.   In E-mail, you have control over whom
you are sending messages to, even if you have a giant mailing list.
You have the ability to know who they are and possibly ask them
questions about what they want.   Many mailing lists (see yahoo groups)
let list members put in a setting saying if they want HTML or plain
text mail from the list for example.

USENET is a broadcast format.  You don't know who's reading, they have
no ability to set a preference, so you must transmit in a form that
all of them can handle, and all of them must be ready to handle
any form that is transmitted.

This makes migration of forms much harder, believe it or not, than
in mail.   Taken strictly, nobody can use a new form in news until
everybody is ready to receive it, which is impossible, so the net
does sometimes break into subnets to try out new things when it must.

Note that NNTP is not a one to many protocol itself.  It is USENET
itself that is the one to many, simulating broadcasting through the
use of a collection of 1 to 1 protocols.  Though in theory (and
briefly once in practice) USENET could be done via IP multicasting.

To: [email-masked]
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 21 Feb 2003 12:21:45 -0800
Lines: 42
Sender: [email-masked]

D J Bernstein <[email-masked]> writes:

> Actually, there's very little opposition (especially among implementors)
> to requiring all MTAs, MUAs, etc. to handle UTF-8 messages. Eventually
> we will all be using UTF-8; all relevant bugs must be fixed. Only the
> wildest ``7 bits forever!'' proponents, such as Keith Moore, disagree.

> The real controversy is over whether we should also do _other_ things
> before UTF-8 is working everywhere. For example, should we introduce
> some ad-hoc 7-bit character encoding for newsgroup names?

> Many of us (especially implementors) believe that these short-term 7-bit
> kludges have huge costs (as illustrated by your message) and miniscule
> benefits. We believe that the 7-bit kludges should be dropped.

> Our opponents are claiming that the IESG will demand a 7-bit solution.
> But they aren't opposing the requirement of UTF-8 support; they're
> opposing the reliance on UTF-8 as the sole solution.

I'm not sure that I agree with your summary of the positions, but I'm
certainly sympathetic to this viewpoint.  Just implementing UTF-8 feels a
lot cleaner to me too.

However, my main interest personally is to get something published by the
IETF documenting the Usenet article format that isn't as horribly obsolete
and out of date as RFC 1036 is.  I'm also most definitely not a mail
system implementor or an IMAP implementor and don't know what issues
implementors in those areas face.

One additional option that I didn't mention would be to decide that the
IETF standards process is out of touch with the reality of what
implementors want and to then simply punt on specifying a non-ASCII
encoding for newsgroups in the standard (so as not to get dragged into
these arguments) and encourage anyone who wants to use a non-ASCII
character set in practice to use UTF-8.  This will work for most news
server software.  I personally don't have a sufficient grasp on the issues
facing news client implementors, IMAP implementors, or mail system
implementors to know whether that's a viable solution outside of news, and
if so, whether it's the best solution available.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 21 Feb 2003 19:53:07 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Russ Allbery writes:
> I am also making the assumption that a standard requiring e-mail or IMAP
> to handle unencoded UTF-8 in message headers and in newsgroup names is not
> a viable option, due to strong oppposition from the e-mail community.

Actually, there's very little opposition (especially among implementors)
to requiring all MTAs, MUAs, etc. to handle UTF-8 messages. Eventually
we will all be using UTF-8; all relevant bugs must be fixed. Only the
wildest ``7 bits forever!'' proponents, such as Keith Moore, disagree.

The real controversy is over whether we should also do _other_ things
before UTF-8 is working everywhere. For example, should we introduce
some ad-hoc 7-bit character encoding for newsgroup names?

Many of us (especially implementors) believe that these short-term 7-bit
kludges have huge costs (as illustrated by your message) and miniscule
benefits. We believe that the 7-bit kludges should be dropped.

Our opponents are claiming that the IESG will demand a 7-bit solution.
But they aren't opposing the requirement of UTF-8 support; they're
opposing the reliance on UTF-8 as the sole solution.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: Usefor Mailing List <[email-masked]>, IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: mail vs. news ???
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 21 Feb 2003 10:11:47 -0800
Lines: 63
Sender: [email-masked]

Ken Murchison <[email-masked]> writes:

> I find it interesting, if not disturbing, that some members of the
> usenet community seem to think that mail messages and usenet articles
> are not the same thing.  AFAICT, from reading the relevant standards,
> writing server code for SMTP/LMTP/IMAP/POP3/NNTP, and everyday use, mail
> messages and news articles both conform to RFC 2822 (RFC 1036 states as
> much).  The only differences that I'm aware of are the following:

> - usenet puts a greater restriction on the headers (although still being
> RFC 2822 compliant)

Yes, although there are a variety of subtle things that fall into this
category that one has to be careful of.  That's one of the difficulties
with Dan Kohn's existing draft; in dropping all of the language that was
completely unnecessarily copied from RFC 2822, he's also dropped some
language that's actually necessary.

Usenet's restrictions on the syntax of message ID headers are very
specific and very precise, and much stronger than those of RFC 2822, in
part because message IDs are used as part of the NNTP protocol.  Comments
in various places that mail supports them are not well-supported by
currently deployed Usenet software (although it certainly hurts nothing to
support them when writing new code, other than adding complexity).  The
space after the colon in headers is not optional on Usenet.  The syntax of
the Date header is restricted in ways somewhat similar to that of the
Message-ID header.

> - mail messages are typically tranmitted over a 1-to-1 protocol (SMTP)
> and news articles are typically transmitted over a 1-to-many protocol
> (NNTP)

> Could somebody please enlighten me as to any others differences,
> perceived or otherwise?

- National 8-bit character sets are in widespread use in Usenet message
headers, possibly more widespread than they are in (non-spam) mail
messages.  Untagged 8-bit national character sets are widely used in
various non-English hierarchies in headers as the preferred way of
including such content, and in some cases use of RFC 2047 is frowned on.

Usenet was able to get away with that when e-mail couldn't because the
Usenet core software has always been 8-bit clean and because, due to the
more centralized nature of Usenet, it's possible to make determinations
like "every group in this hierarchy will be using this national character
set" and even encode them in news clients.  It's much harder to know what
character set to expect in the general case in e-mail software.

- MIME never got very much uptake on Usenet for attachments.  The binary
newsgroups are almost universally uuencode or, these days, yEnc.  Base64
is rather rare and not at all popular.  Binary readers and posters, for
reasons that I admit to really not understanding, seem to have had no
interest in the additional, cleaner semantics offered by MIME attachments
and continue to be happy with uuencode, scanning the body for telltale
lines, and doing guessing based on file names.  Part of this may be due to
the fact that much of the Usenet binary traffic is split across multiple
posts and the MIME message/partial content-type is ill-suited to Usenet
for a wide variety of reasons.  Just splitting a uuencoded file into
multiple parts is a lot easier (although it certainly makes decoding a
pain).

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 21 Feb 2003 09:44:06 -0800
Lines: 15
Sender: [email-masked]

Andrew Gierth <[email-masked]> writes:

> there is a specific reason to avoid punycode in my proposal (proposal
> A), which is that it's not easily generated in a simple script; the
> easiest way to upgrade several existing news software packages to handle
> moderated non-ASCII groups is to wrap the mail-to-moderator program in a
> script to do the necessary header changes. (This works for INN, Diablo,
> my server, and at least some versions of DNews on some platforms -
> basically any server which uses the common approach of doing
> mail-to-moderator as an external program call.)

Good point.  That should have been mentioned in my summary.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Fri, 21 Feb 2003 09:43:21 -0800
Lines: 27
Sender: [email-masked]

Andrew Gierth <[email-masked]> writes:

> but this proposal as it stands largely cripples the ability for
> clients to do "LIST ACTIVE <wildmat>" unless _both_ the client and the
> server are updated to support an extension for punycode-wildmats.
> (i.e. without such an extension, the client can't do pattern matches
> in ways that are meaningful in terms of the human-readable name, only
> in terms of the encoded name)

I'd thought about this, but the degree of crippling didn't seem severe
enough to even mention.  Per-component wildmat patterns still work, and
within-component matches of ASCII groups continue to work, so this only
affects cases where the client is trying to use a wildmat to match a
portion of a non-ASCII name.

This is an extremely uncommon operation for a reader as I understand it,
even when there are non-ASCII groups.  Am I out of touch there?
Essentially all of the use of those sorts of wildmats that I'm aware of is
in transit servers.

How frequently do you see clients using wildmats other than *, the exact
name of a group, or component-wise?  I thought that most clients that
supported a newsgroup search feature did so by downloading the entire
active file and then doing their searches locally.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 21 Feb 2003 09:46:35 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
To: Usefor Mailing List <[email-masked]>
CC: IMAP mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: mail vs. news ???
Sender: [email-masked]

I find it interesting, if not disturbing, that some members of the
usenet community seem to think that mail messages and usenet articles
are not the same thing.  AFAICT, from reading the relevant standards,
writing server code for SMTP/LMTP/IMAP/POP3/NNTP, and everyday use, mail
messages and news articles both conform to RFC 2822 (RFC 1036 states as
much).  The only differences that I'm aware of are the following:

- usenet puts a greater restriction on the headers (although still being
RFC 2822 compliant)
- mail messages are typically tranmitted over a 1-to-1 protocol (SMTP)
and news articles are typically transmitted over a 1-to-many protocol
(NNTP)

Could somebody please enlighten me as to any others differences,
perceived or otherwise?

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

To: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
From: Andrew Gierth <[email-masked]>
Date: 21 Feb 2003 07:26:51 +0000
Lines: 19
Sender: [email-masked]

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

 Russ> Note that in each case some other encoding system besides
 Russ> punycode could in theory be used.  I don't believe the choice
 Russ> of encoding changes the remainder of this analysis, however, so
 Russ> punycode is left as a placeholder (and what seems to currently
 Russ> be the most likely choice).

there is a specific reason to avoid punycode in my proposal (proposal
A), which is that it's not easily generated in a simple script; the
easiest way to upgrade several existing news software packages to
handle moderated non-ASCII groups is to wrap the mail-to-moderator
program in a script to do the necessary header changes. (This works
for INN, Diablo, my server, and at least some versions of DNews on
some platforms - basically any server which uses the common approach
of doing mail-to-moderator as an external program call.)

-- 
Andrew.

To: [email-masked], [email-masked], [email-masked]
Subject: Unicode newsgroup name options
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 20 Feb 2003 16:10:09 -0800
Lines: 280
Sender: [email-masked]

This is a summary of what I believe to be the options for handling Unicode
newsgroup names in an IETF standard.  So far, I see three separate viable
options for handling encoded newsgroup names through the entire protocol.
Those three options are:

  (A) UTF-8 in articles and NNTP, punycode in e-mail and IMAP
  (B) punycode in e-mail, IMAP, and articles, UTF-8 in NNTP
  (C) punycode everywhere

These are expanded in more detail below.  Note that in each case some
other encoding system besides punycode could in theory be used.  I don't
believe the choice of encoding changes the remainder of this analysis,
however, so punycode is left as a placeholder (and what seems to currently
be the most likely choice).

I am also making the assumption that a standard requiring e-mail or IMAP
to handle unencoded UTF-8 in message headers and in newsgroup names is not
a viable option, due to strong oppposition from the e-mail community.
Regardless of whether I agree with that opposition or not, I'm
uninterested in reopening that discussion, which went on both in
usenet-format and in ietf-822 at extended length.

This summary does not address internationalization issues in any other
headers or information besides Usenet newsgroup names.

In this analysis, I will be referring to the following components of the
Usenet messaging system:

  (1) A newsreader posting via NNTP.
  (2) The NNTP server accepting posts from a client.
  (3) The NNTP transit server relaying posts to other servers.
  (4) The NNTP server providing messages to a client.
  (5) A newsreader reading via NNTP.
  (6) The NNTP server relaying a message posted to a moderated group.
  (7) The local mail system of the NNTP server.
  (8) The mail system of the moderation relay site.
  (9) The local mail system of the moderator.
 (10) The software used by the moderator of a newsgroup.
 (11) A mail to news gateway.
 (12) A news to mail gateway.
 (13) An IMAP server serving Usenet messages to a client.
 (14) An IMAP client reading Usenet messages from an IMAP server.

One or another of these proposed options affect every single component of
this system except for (11).  In the case of (11), none of these three
proposals will affect any existing mail to news gateways.  Existing mail
to news gateways may not be able to handle new non-ASCII newsgroups
without modification, but all three proposals are backward-compatible in
the sense that all currently working gateways to currently existing groups
will continue to function as they do now.  Please note that I have
separated the moderation process into a separate component from a general
mail to news gateway.

For new mail to news gateways for new non-ASCII newsgroups, the issues are
essentially the same as for posting agents (1).

In the case of (14), I am making the assumption that changing the IMAP
protocol is not an option.  This means that messages served to (14) will
not contain unencoded UTF-8 in the headers, and newsgroup names in IMAP
will not have unencoded UTF-8 names.  All of the work (if any) required to
make the articles compatible with an IMAP environment would have to be
born by (13).  In all of these cases, it will therefore be desirable for
the IMAP client to be modified to understand punycode and display the
newsgroup names correctly.

Note that there is an additional component that is left unmentioned above,
namely encoding of newsgroup names in URLs.  I don't know enough about
this area to comment usefully, but I believe that it's somewhat orthogonal
to the remaining issues.

(A) UTF-8 in articles and NNTP, punycode in e-mail and IMAP
===========================================================

This is Andrew's original proposal.  The canonical name of the newsgroup
would be in UTF-8 without further encoding.  The Usenet article format
would be defined to carry UTF-8 newsgroup names without further encoding
in those headers that contain newsgroup names (Newsgroups, Followup-To,
Control, and Xref).  Similarly, the body of control messages for non-ASCII
newsgroups would be required to be in UTF-8 and would contain the UTF-8
newsgroup names.

NNTP commands would take UTF-8 arguments wherever newsgroup names are
referred to.  wildmat would be modified to match UTF-8 characters if the
server supported the ? or [] wildcards.

At every point where a Usenet article must be conveyed via e-mail,
specifically (6), (12), and (13), any non-ASCII content in Newsgroups and
Followup-To would be encoded in punycode (or some other suitable encoding
method).  (Control and Xref headers are generally not gatewayed.)  The
envelope recipient used when sending to the moderation relays (8) would
contain the encoded form of the newsgroup name.  A moderator (10) who
received a post to a non-ASCII newsgroup (either the newsgroup they
themselves are modifying or a newsgroup to which the message was
crossposted) would, in order to approve the message, have to either decode
the newsgroup name to its canonical UTF-8 form again or use an injector
(2) that will do this.  Otherwise, the article should be rejected.

This proposal requires no changes to (7), (8), or (9); in other words, the
existing mail transit systems are unaffected by this proposal.  It
requires only minimal changes to (2), (3), and (4), the existing news
transit system, to remove restrictions preventing creation of non-ASCII
newsgroups.  It is believed that essentially all existing news transit and
server systems still in active use can handle 8-bit newsgroup names
without difficulties.  It would be desirable for (2), the injection agent,
to be able to undo the mail encoding automatically.

Additionally, a news reader (5) may be able to read such groups without
modification if it already has support for 8-bit characters and can be
configured appropriately, and similarly a news posting agent (1) may also
be able to be used without modification.  Updates to (1) and (5) to
provide Unicode character entry, canonicalization, and display would of
course be extremely desirable.  (1) and (5) require no modifications to
deal with existing ASCII newsgroups except modifications for 8-bit
cleanliness to handle crossposted messages.

Moderation software (10) would have to change in order to handle any
non-ASCII groups, since the mail encoding would have to be decoded, or the
moderator would have to arrange to use an updated injecting agent (2).
Moderators of existing ASCII newsgroups who didn't want to deal with this
issue could simply reject all articles crossposted to non-ASCII
newsgroups.  There is some likelihood that messages crossposted between
moderated ASCII newsgroups and other (moderated or unmoderated) non-ASCII
newsgroups would end up under some circumstances being injected into the
news system with the non-ASCII newsgroup names encoded in the mail
encoding, with the only damage being that the articles would not show up
in the non-ASCII newsgroups that they were intended to be posted to.

Any news to mail gateway (12) would have to be modified if it received any
messages crossposted to non-ASCII newsgroups and wanted to preserve the
Newsgroups header in the e-mail message.  Failure to encode the headers
appropriately would result in unencoded 8-bit text in the headers of a
mail message, where it may be mangled or rejected by the mail system.

Any IMAP server processing Usenet messages (13) would have to perform the
same transformations, encoding newsgroup names in Newsgroups, Followup-To,
and Control (and Xref if the IMAP server wished to maintain it).  In
addition, the newsgroup name would have to be presented to the client in
an encoded form; UTF-7 may be preferrable in this case to punycode.

(B) punycode in e-mail, IMAP, and articles, UTF-8 in NNTP
=========================================================

This is the intermediate proposal, allowing use of UTF-8 directly in NNTP
where it's fairly uncontroversial and continuing to treat the canonical
name of the newsgroup as the unencoded UTF-8 form, but always encoding the
newsgroup name wherever it occurs in a news article.  This maintains
complete RFC 2822 compatibility in the article format, unlike (A), but
still allows use of UTF-8 in NNTP.

Any non-ASCII newsgroup names in Newsgroups, Followup-To, Control, and
Xref would be encoded using punycode.  For ease of processing and
consistency, that probably also means that newsgroup names in the bodies
of control messages should also be encoded in punycode.

All NNTP commands would take UTF-8 arguments for newsgroup names, and the
newsgroup names returned by LIST, GROUP, and similar commands would be in
UTF-8.

This means that the newsgroup name sent to the server in a GROUP command
and the newsgroup name in the Newsgroups and Xref headers would not be the
same.  While it may still be possible for an extremely sophisticated user
to use an unmodified news reader (5) or posting agent (1), it would
require the user to override the news client at a multitude of points and
would be at best a last-ditch sort of affair, far too clumsy to use for
any sustained period.  This proposal therefore mandates modifications to
(1) and (5) for any user who wants to use non-ASCII newsgroups.

If the user only wants to use existing ASCII newsgroups, their existing
client software can be used unmodified.  It must, however, be able to
handle 8-bit newsgroup names returned from the LIST command (but doesn't
have to be able to handle 8-bit content in the article headers).

NNTP servers (2) and (4) must be modified in order to carry non-ASCII
newsgroups to decode the newsgroup headers when receiving messages so as
to know what newsgroup into which to file them.  The active file would
also need to be kept in UTF-8.  As above, it is believed that the other
NNTP commands besides POST/IHAVE/TAKETHIS would work without modification
because existing NNTP software is already 8-bit clean.  If the NNTP
software is not modified, the newsgroups will show up in their punycode
encoded form, possibly confusing compliant news reading software.

Transit servers (3) do not need to be modified.  For the best support of
pattern-based feeds, transit servers will want to decode the newsgroup
header as it comes in and then apply wildmat patterns to the decoded form
so that wildmat patterns can be specified in UTF-8.  The transit servers
will continue to function correctly without this modification, however,
and news administrators could add additional appropriate patterns to catch
the punycode-encoded forms.  Presuming that ASCII newsgroup names are not
encoded (a reasonable assumption for any encoding format, I believe), the
only reason to add punycode support to transit servers would be for the
convenience of the administrator in expressing wildmat patterns for
non-ASCII newsgroups in an unencoded form.  (I believe that the likelihood
that a punycode-encoded name would happen to match one of the widely used
patterns like *sex* or *mp3* is fairly small, but I could be wrong as I've
not done a statistical analysis.)

This proposal requires no modifications to the moderation system of (6),
(7), (8), (9), and (10) whatsoever, including while handling non-ASCII
groups.  It similarly requires no modifications to news to mail gateways
(12).  IMAP servers (13) may want to recode the newsgroup names from
punycode to UTF-7, but would not need to make any transformations to the
articles themselves.

(C) punycode everywhere
=======================

The "most encoded" proposal, this proposal says to use punycode
everywhere.  All newsgroup names in the Usenet articles and via the NNTP
protocol would be encoded in punycode and the punycode-encoded version of
the newsgroup name would be the canonical one.  The name would only be
decoded for display purposes in the client software.  This maintains
complete RFC 2822 compatibility for the article format.

This proposal mandates modifications to the posting agents (1) and the
news readers (5) in order to properly display the names.  No modifications
must be made to news readers that are reading only ASCII newsgroups; they
will just see a bunch of additional oddly-named newsgroups.  Existing news
readers could still read and post to non-ASCII newsgroups if they didn't
mind the odd names.

No modifications are required to (2) or (4), the NNTP servers, although
without modifications the server administrator would have to work with
encoded group names.  It would provide a much better user interface if the
administrative tools implemented punycode encoding and decoding for easier
handling of non-ASCII newsgroup names.

The same issues as with (B) apply to transit servers (3), namely that it
would be convenient but not required for transit servers to decode the
newsgroup names before doing wildmat matching so that the wildmat patterns
could be specified in a convenient format.

No modifications are required for the moderation system of (6), (7), (8),
(9), and (10) whatsoever, including while handling non-ASCII groups.
Similarly, no modifications are required for news to mail gateways (12).
IMAP servers (13) may again wish to recode punycode to UTF-7 for newsgroup
names, but otherwise require no modification.

Summary
=======

The following chart summarizes the backward compatibility issues for each
proposal and each component of the news system.  For each portion of the
news system, N means no change required, Y means change is required to
correctly handle non-ASCII newsgroups, D means change is very desirable
but not absolutely necessary, and C means change would be convenient but
unmodified software is still fairly usable.

  | 1   2   3   4   5   6   7   8   9  10  11  12  13  14
 -+------------------------------------------------------
 A| D   C   N   N   D   Y   N   N   N   Y   N   D   Y   D
 B| Y   Y   C   Y   Y   N   N   N   N   N   N   N   C   D
 C| D   C   C   C   D   N   N   N   N   N   N   N   C   D

The above summary I believe correctly indicates that proposal (B) requires
the most changes to be made to the news system itself.  It's possible to
use existing software without modification with either proposal (A) or
proposal (C); under proposal (A), news readers that aren't 8-bit clean
will break, and some news readers may actually get display right without
having to make any modifications, but with other news readers it may be
impossible to access a non-ASCII group because no Unicode entry is
supported.  Under proposal (C), we're guaranteed that nothing will break
and that it will always be possible to access even non-ASCII groups, but
no existing client will display the names correctly.

Overall, it is somewhat less necessary to change client software with (A)
than with (C); exactly how much less necessary is something of an open
question.

Proposal (A) is the only proposal that requires changes to any system
outside of the news system (other than changes to an IMAP client to
understand the punycode newsgroup names, which are the same for all
proposals).  Both (B) and (C) work with the moderation, e-mail, and IMAP
infrastructure without any additional changes.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: 19 Feb 2003 19:23:59 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

sendmail is not 8-bit-clean. It destroys bytes 128 through 159 in
message headers. It has to be fixed (I explained the easy fix to the
sendmail author _four years ago_) or replaced with better MTAs (which
is gradually happening).

Lilly is completely missing the point when he says that sendmail doesn't
hurt any messages allowed by RFC 2822. The goal here is to allow _more_
messages---specifically, UTF-8 messages.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Tue, 18 Feb 2003 01:58:08 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Russ Allbery wrote:

>>In message bodies?
> 
> 
> No, in message headers.
> 
> 
>>And message header fields can only contain a subset of ASCII characters;
>>see RFC 2822 for details.
> 
> 
> This is a completely unhelpful comment, which I probably should just
> ignore, but which I'm going to instead rant about briefly.

Calm down.

I posted a follow-up to the list because your response to
Jacob Palme's question did not specify that *only* the
message header is affected.  To put it another way, your
response implied that there is a problem with message
bodies, and that is not the case.

Sendmail does not have a problem with any legal 8-bit
message, and my followup was intended to make that clear.

Now you might well point out that there may be a potential
problem *if* 8-bit content were to be permitted in the
message header.  But such content is currently not permitted,
and there has not been any discussion here (the ietf-822
mailing list) recently about changing that in the near
future.

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 17 Feb 2003 20:08:20 -0800
Lines: 37
Sender: [email-masked]

Bruce Lilly <[email-masked]> writes:
> Russ Allbery wrote:

>> It still has problems with characters in the ISO 8859 excluded range
>> (the characters that correspond to ASCII control characters with the
>> high bit set).

> In message bodies?

No, in message headers.

> And message header fields can only contain a subset of ASCII characters;
> see RFC 2822 for details.

This is a completely unhelpful comment, which I probably should just
ignore, but which I'm going to instead rant about briefly.

I realize that you're trying to make a pedantic point about standards, but
that isn't what the rest of this thread was talking about.  The rest of
this thread, as I understood the message to which I was replying, was
talking about 8-bit cleanliness, independent of what one is allowed to
send under RFC 2822 rules.  It remains the case that sendmail is not 8-bit
clean because it doesn't support some characters in headers.  This is a
true statement regardless of whether those characters should ever occur in
headers to begin with, and is of interest if one is discussing any
proposed possible plan (whether you personally like the other details of
the plan or not) in which such characters would be sent.

I know you think that introducing any sort of raw 8-bit character set into
headers is a bad idea, and I understand your position, but not everyone
else in the discussion is convinced of the correctness of that position.
People are likely going to continue to talk about the implications of such
a move no matter how much you say that you think it's a bad idea or
contrary to existing standards.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Mon, 17 Feb 2003 22:57:49 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2047 and gatewaying
Sender: [email-masked]

Russ Allbery wrote:

> It still has problems with characters in the ISO 8859 excluded range (the
> characters that correspond to ASCII control characters with the high bit
> set).

In message bodies?  I don't think so.  And message header fields
can only contain a subset of ASCII characters; see RFC 2822 for
details.

To: [email-masked]
Subject: Re: RFC 2047 and gatewaying
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 17 Feb 2003 15:20:49 -0800
Lines: 15
Sender: [email-masked]

Jacob Palme <[email-masked]> writes:
> At 21:01 -0800 03-01-03, Russ Allbery wrote:

>>  * sendmail still is not fully 8-bit clean in the latest analysis that
>>    I've seen, which means that 7-bit issues continue to plague the mail
>>    system.

> In what ways is sendmail not yet fully 8-bit clean?

It still has problems with characters in the ISO 8859 excluded range (the
characters that correspond to ASCII control characters with the high bit
set).

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Mon, 17 Feb 2003 10:39:23 +0000
To: [email-masked]
From: Graham Klyne <[email-masked]>
Subject: Thanks (was: Problems with all lowercase header field names?)
Sender: [email-masked]

Thanks to everyone who provided experience-based feedback.

#g

-------------------
Graham Klyne
<[email-masked]>

Date: Mon, 17 Feb 2003 13:07:41 +0100
To: Russ Allbery <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Re: RFC 2047 and gatewaying
Cc: [email-masked]
Sender: [email-masked]

At 21:01 -0800 03-01-03, Russ Allbery wrote:
>  * sendmail still is not fully 8-bit clean in the latest analysis that
>    I've seen, which means that 7-bit issues continue to plague the mail
>    system.

In what ways is sendmail not yet fully 8-bit clean?
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

Date: Mon, 17 Feb 2003 08:00:28 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Graham Klyne <[email-masked]>
Cc: [email-masked]
Subject: Re: Problems with all lowercase header field names?
Sender: [email-masked]

Graham Klyne <[email-masked]>
> Is anyone aware if using all lowercase header field names in email messages 
> is ever known to cause any problems?

Pine generates "cc:", IIRC, so any problems would have been seen by now.

--Arnt

Date: 14 Feb 2003 19:44:06 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Review of the UTF-8 transition
Sender: [email-masked]

Bruce Lilly writes:
> if there's a desire to move to utf-8, use of untagged 8-bit content
> will have to cease first so that when generation of untagged utf-8 is
> eventually permitted, one can be absolutely assured that such untagged
> 8-bit content *is* utf-8

No. Here, once again, is the (blazingly obvious) transition plan.

The first step in the transition is to change message handlers to allow
UTF-8 messages, if they don't already. In particular, message readers
have to display messages as UTF-8 if they look like UTF-8. Impact:

   * In the IETF universe, this is a perfectly safe step, since 8-bit
     messages have no previous use.

   * In the real world, there are many messages in (e.g.) 8859-1.
     However, those messages generally don't follow the UTF-8
     byte-sequence restrictions, so they aren't affected by the change.
     Practically all messages following the UTF-8 restrictions are, in
     fact, UTF-8; the change means that they are displayed correctly.

The second step is to allow message generators to use UTF-8. Impact:

   * In the IETF universe, this can wait until after the first step is
     done, because users are all perfectly happy with ASCII right now
     and would never consider blatantly violating the standards. Using
     UTF-8 is perfectly safe once the first step is done.

   * In the real world, message generators will naturally switch to
     UTF-8 from (e.g.) 8859-1, once that's beneficial for the users.
     This will happen once the (shrinking) UTF-8 failure rate drops
     below the (fairly small but never shrinking) 8859-1 failure rate.

In the end, the IETF universe and the real world will end up at the same
place. The current mess of incompatible character encodings, and its
associated failure rate, will disappear. UTF-8 will be supported and
used everywhere.

Your silly notion of being ``absolutely assured that ... untagged 8-bit
content is utf-8'' has no relevance to the transition. It won't be true
in the real world until long after the transition is complete.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Fri, 14 Feb 2003 14:07:20 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: FW: I-D ACTION:draft-kohn-news-article-00.txt
Sender: [email-masked]

Charles Lindsey wrote:

> Those who are opposing us seem to be from the IETF "establishment".
> I reckon that we are nearer to the "grass roots". However, ultimately it
> is the IETF that get to decide :-( .

This has to win the prize for unintentional humor.  This is an IETF
working group, ostensibly working on a Standards Track IETF RFC,
the draft of which even has "ietf" in its name.  The IETF isn't
opposing the IETF WG, it's telling Charles that his attempts to
waylay the Group and ignore IETF procees are ill-advised.

Hmm, "grass roots" -- yep, that fits with "lower than a snake's belly",
but of course that doesn't apply to the *entire* WG.

Date: Fri, 14 Feb 2003 12:07:31 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Charles Lindsey wrote:

>>Before utf-8 can be adopted, there needs to be a transition
>>period where there is a moratorium on *all* untagged 8-bit
>>header field content as a prerequisite to a state where
>>the only untagged 8-bit content is utf-8.  The current
>>Usefor draft lacks such a transition plan.
> 
> 
> And that just shows how out of touch with the Real World (TM) you are.
> 
> How do you propose to introduce such a moratorium (that behaviour is
> _already_ non-compliant with the standards). More importantly, how do you
> propose to enforce it?

The standard can simply state the requirement, viz. no unencoded
8-bit content.  That is a necessary prerequisite from the current
chaos to use of a single 8-bit untagged charset.  Enforcement
isn't necessary -- if there's a desire to move to utf-8, use of
untagged 8-bit content will have to cease first so that when
generation of untagged utf-8 is eventually permitted, one can
be absolutely assured that such untagged 8-bit content *is* utf-8
and not any of a hundred other charsets. That's the carrot.

> I can tell you that it will _never_ happen, and so if you wait for it you
> will wait for ever, and UTF-8 will never get introduced into the
> nstandards.

That's the stick.

If it doesn't happen, at least the time-tested, standards-compliant,
backwards-compatible, widely-implemented MIME (RFC 2047/2231) methods
can be used, and untagged 8-bit content will still be officially
non-compliant.  And if it doesn't happen, that means there is no
real desire for it to happen.  If it does happen, there are still
issues that need to be addressed, but that's jumping ahead quite a
bit.

> OTOH, if it is made clear from the outset that UTF-8 is the one and true
> way, then at least some people will take heed, and the situation will get
> better (still not perfect, but better).

No the situation will remain as intolerably bad as it is now; a
proliferation of untagged 8-bit use, where it is not possible to
determine the charset in use. Or, in the words of RFC 2978:
    Use of a large number of charsets in a given protocol may hamper
    interoperability.  However, the use of a large number of undocumented
    and/or unlabeled charsets hampers interoperability even more.
And note that RFC 2277, BCP for internationalization, states:
    Negotiating a charset may be regarded as an interim mechanism that is
    to be supported until support for interchange of UTF-8 is prevalent;
    however, the timeframe of "interim" may be at least 50 years, so
    there is every reason to think of it as permanent in practice.
It is entirely up to Usenet users and software authors how long it
will take to reach such a condition; if use of multiple untagged
8-bit charsets ceases quickly, the transition to utf-8 can happen quickly.
If not, then not.  Requiring the cessation of untagged 8-bit content
is a necessary step in that process, not only so that it will
eventually be possible to have a single official untagged charset,
but also to provide for interoperability with existing news infradtructure,
including SMTP (used for moderated postings and with gateways) and
IMAP (used by news and mail UAs).  It also provides time for the
upgrading of that infrastructure.

To: [email-masked]
Xref: clerew local.mime:2408 local.usefor:15783
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Transformation of Non-ASCII headers
Date: Thu, 13 Feb 2003 17:21:05 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Sam Roberts wrote:

>> Backwards compatibility with standards compliant messages, that I
>> understand, but backwards compatiblity with invalidly encoded messages?

>Before utf-8 can be adopted, there needs to be a transition
>period where there is a moratorium on *all* untagged 8-bit
>header field content as a prerequisite to a state where
>the only untagged 8-bit content is utf-8.  The current
>Usefor draft lacks such a transition plan.

And that just shows how out of touch with the Real World (TM) you are.

How do you propose to introduce such a moratorium (that behaviour is
_already_ non-compliant with the standards). More importantly, how do you
propose to enforce it?

I can tell you that it will _never_ happen, and so if you wait for it you
will wait for ever, and UTF-8 will never get introduced into the
nstandards. Which will actually make the situation worse, because people
will start using even more outlandish charsets without tagging.

OTOH, if it is made clear from the outset that UTF-8 is the one and true
way, then at least some people will take heed, and the situation will get
better (still not perfect, but better). In fact one can possibly get to a
state where everybody except the Chinese uses it (sadly, the Chinese are a
law unto themselves :-( ).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: "Shmuel (Seymour J.) Metz" <[email-masked]>
Date: Wed, 12 Feb 2003 14:10:25 -0500
To: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Organization: Atid/2
Sender: [email-masked]

In <[email-masked]>, on 02/12/2003
   at 10:44 AM, Bruce Lilly <[email-masked]> said:

>Before utf-8 can be adopted, there needs to be a transition period
>where there is a moratorium on *all* untagged 8-bit header field
>content as a prerequisite to a state where the only untagged 8-bit
>content is utf-8.

I see nothing in the charter about playing King Canute telling the
tide not to come in. The avalanche has already started; it is too late
for the pebbles to vote.

The best that we can do is to convince the people who will post 8-bit
headers regardless to at least use an agreed upon representation.
Anything beyond that wouldn't happen even if the entire WG, the AD and
IETF all agreed with you.
 
-- 
-----------------------------------------------------------
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2
     Team OS/2
     Team PL/I
-----------------------------------------------------------

Date: Wed, 12 Feb 2003 22:38:03 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> But again, not for ever. Remember that we are talking particularly of RFC
> 2047 here. If in 50 years time you come across an old email with RFC 2047
> in its headers, and it is 25 years since the last MUA that understood it
> had died, all is still not lost because you can still read the ascii and
> decode it by hand.

I honestly don't know whether 25 years is enough.  I think document format
obsolesence is a big problem - along with media format obsolesence.

Yes, I can read WordStar documents that I wrote in 1978, but only because
I can write code and WordStar used a very simple format.  Most people don't
have tools that would let them read such documents effectively.

Keith

To: [email-masked]
Xref: clerew local.mime:2402 local.usefor:15748
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Upgrading to UTF-8
Date: Wed, 12 Feb 2003 15:19:49 GMT
Lines: 75
Sender: [email-masked]

In <[email-masked]> Chris Newman <[email-masked]> writes:

>Guide to removing cruft from standard protocols:

>1. Write a standard extension to negotiate use of the less-crufty protocol, 
>and explain how to downconvert to the fully-interoperable crufty protocol.

For sure downconversion is needed at some point in the process, but there
are several points at which you can introduce it:

a) Negotiate it, as you say. Indeed, I have a negotiation suggestion for
email that I have touted on the Usefor list. Very tentative, hairy and
incomplete at the moment, so I would rather not bring it to the ietf-822
list just yet.

b) But sometimes negotiation is impossible (e.g on Usenet the sender cannot
negotiate with all the intended recipients). So you designate boundary
points where downconversion MUST be done (usually at the point where you
are changing from one protocol to another). Later on, when less-crufty
facilities are available on the other side of the boundary, you may be able
to negotiate insteand of downconverting regardless. This is essentially
the approach Usefor is proposing in the present case.

c) Return it to the sender, and get him to downconvert it and send it
again. A bit draconian, you may say, but probably the best way to educate
senders who insist on sending stuff in unidentified charsets.  Corollary:
it you think it is spam, omit the "return it" stage :-) .

d) There must be other possibilities too ...

>2. Deploy standards-compliant servers and clients that accept the 
>less-crufty protocol, down-convert if necessary, and will generate the 
>less-crufty protocol only if negotiated.  These interoperate with all 
>software.

There is a chicken-and-egg problem here. Deployment will not happen until
there is pressure from users. Users cannot use it until it is deployed. So
expect some angry users.  Nothing like a bit on anger to get the
implementors moving.

But yes, the implementors need clear specs to work to, so you need at
least an experimental protocol, if not a standard.

>3. Deploy non-standard software that generates only the less-crufty 
>protocol.  This software isn't guaranteed to interoperate with everything, 
>but plays an important role as it creates pressure to upgrade to software 
>from step 2.

The difficult part is ensuring that stage 2 happens before stage 3. Users
are a pushy lot with a tendency to deploy non-standard stuff even before
the standards exist, let alone are deployed (and Yenc is a good example of
what happens if the standard solutions are not provided in a timely
manner).

The only way to stop that is to retrofit "drop this non-standard stuff on
the floor" features into existing software, and if you are going to that
sort of trouble, you may as well implement the new standard instead.

>4. Over time, more and more developers will not bother to implement the 
>downconversion and cruft-support.  Eventually the amount of software 
>requiring the downconversion is close enough to zero that the standards can 
>be changed to drop the requirement.

For some suitable value of "eventually" :-) .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2403 local.usefor:15749
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Upgrading to UTF-8
Date: Wed, 12 Feb 2003 15:27:44 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>and note that I didn't say 'indefinitely'.  but lots of people keep old mail
>around.   while it's hard to argue that people should keep buying 5.25" disk
>drives for the sake of reading their old floppies (partially because the 
>medium is essentially useless for storage of new things), it's a bit easier
>to argue that MUAs should contain a bit of extra code to handle slight
>differences in message format that were once widely used.

But again, not for ever. Remember that we are talking particularly of RFC
2047 here. If in 50 years time you come across an old email with RFC 2047
in its headers, and it is 25 years since the last MUA that understood it
had died, all is still not lost because you can still read the ascii and
decode it by hand. And if Google is still keeping articles that old
around, it will probably provide downcoding as part of its service.

But I would hate to be faced with converting punycode by hand :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: 12 Feb 2003 23:11:23 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Russ Allbery writes:
> Standardization of the zone file format was and remains a horrible idea. 

Furthermore, the zone-file format is not actually standardized. Every
major BIND release has a slightly different zone-file format. The RFCs
do not accurately describe the zone-file format in any version of BIND.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 12 Feb 2003 22:58:47 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Andrew Gierth writes:
> compatibility with the real world is important too.

In the real world, practically all messages that look like UTF-8 (i.e.,
that follow the UTF-8 byte-sequence restrictions) are, in fact, UTF-8.

In the real world, displaying a message as UTF-8 if it looks like UTF-8
does not lead to any user complaints, and produces immediately visible
benefits for some users.

In the real world, many users are acutely aware of the variability of
character encodings, and are quite happy to hear about software settling
on a universal character encoding.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

To: [email-masked]
Subject: Re: Transformation of Non-ASCII headers
From: Andrew Gierth <[email-masked]>
Date: 12 Feb 2003 20:58:20 +0000
Lines: 23
Sender: [email-masked]

>>>>> "Sam" == Sam Roberts <[email-masked]> writes:

 >> No, in fact Usenet (and mail) abounds with a large variety of
 >> untagged 8-bit charsets.

 Sam> Sorry, none of that is validly encoded, and has specifically NO
 Sam> meaning, unless assigned one.

that doesn't stop untagged 8-bit in either From or Subject from
accounting for something like 30% of all Usenet text traffic outside
the primary English-language hierarchies.

RFC2047 is much less common; only a handful of hierarchies (mostly
Japanese) have any significant amount of it.

 Sam> Backwards compatibility with standards compliant messages, that
 Sam> I understand, but backwards compatiblity with invalidly encoded
 Sam> messages?

compatibility with the real world is important too.

-- 
Andrew.

Date: Wed, 12 Feb 2003 10:44:38 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Sam Roberts <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Sam Roberts wrote:
> Quoteing [email-masked], on Tue, Feb 11, 2003 at 09:09:16AM -0500:
> 
>>Sam Roberts wrote:
>>
>>>Quoteing [email-masked], on Mon, Feb 10, 2003 at 05:39:26PM -0500:
>>>
>>>>The draft permits a UA to generate raw utf-8. That is then passed to
>>>>an injection agent, which determines that one or more newsgroups are
>>>>moderated.  Existing injection agents do not transform raw-utf-8,
>>>>and no existing or future injection agent can transform any untagged
>>>>8-bit content without charset and language information.
>>>
>>>
>>>Why not?
>>>
>>>The charset seems clearly to be utf-8!
>>
>>No, in fact Usenet (and mail) abounds with a large variety
>>of untagged 8-bit charsets.
> 
> 
> Sorry, none of that is validly encoded, and has specifically NO meaning,
> unless assigned one.

Incorrectly tagging something other than utf-8 *as* utf-8
makes it worse; at least the untagged cruft is clearly
illegal -- incorrectly labelling it doesn't fix it, it
only compunds the error.

> Backwards compatibility with standards compliant messages, that I
> understand, but backwards compatiblity with invalidly encoded messages?

Before utf-8 can be adopted, there needs to be a transition
period where there is a moratorium on *all* untagged 8-bit
header field content as a prerequisite to a state where
the only untagged 8-bit content is utf-8.  The current
Usefor draft lacks such a transition plan.

>>>And a langugae tag is only allowed for paramaters, and even there is
>>>optional, is it not?
>>
>>No, language-tagging is provided by MIME for RFC 2047
>>encoded-words also.
> 
> 
> And is still optional, so does nothing to explain why you would make the
> statement "no existing or future injection agent can transform any
> untagged 8-bit content without charset and language information".

Incorrect; it is "optional" only in the sense that the *user*
need not specify a language. If the user wishes to specify a
language, the protocols MUST provide for preservation of that
language information. See RFC 2277 section 4.

> Apparently there are problems, but lack of charset information and
> language tagging doesn't seem to be.

There indeed are problems, and the lack of tagging are among them.
Lack of charset tagging could possibly be overcome by a suitable
transition plan, but language tagging provision in the protocol
which carries end-to-end is an absolute requirement which can
only be currently met in a compatible manner via RFC 2047/2231
methods.

Date: Wed, 12 Feb 2003 12:03:53 +0100
From: Marc Mutz <[email-masked]>
Subject: Re: Upgrading to UTF-8
To: [email-masked], [email-masked]
Organization: KDE
Sender: [email-masked]

--Boundary-02=_ooiS+UKZg59le4I
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Tuesday 11 February 2003 21:16, Brad Templeton wrote:
<snip>
> Forwarding is not an issue, as
> tools can be told to map the old 7 bit encoding to the new one in
> that case.
<snip>

=2E..breaking digital signatures on the occasion and failing to reach into=
=20
opaque mime parts such as CMS and OpenPGP encrypted content...

Marc

=2D-=20
Mutig warf sich die kleine =DCberwachungskamera zwischen T=E4ter und Opfer!
                                        --Rena Tangens / FoeBuD e.V.

--Boundary-02=_ooiS+UKZg59le4I
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_ooiS+UKZg59le4I--

Date: Wed, 12 Feb 2003 00:27:40 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Tue, Feb 11, 2003 at 09:09:16AM -0500:
> Sam Roberts wrote:
> >Quoteing [email-masked], on Mon, Feb 10, 2003 at 05:39:26PM -0500:
> >>The draft permits a UA to generate raw utf-8. That is then passed to
> >>an injection agent, which determines that one or more newsgroups are
> >>moderated.  Existing injection agents do not transform raw-utf-8,
> >>and no existing or future injection agent can transform any untagged
> >>8-bit content without charset and language information.
> >
> >
> >Why not?
> >
> >The charset seems clearly to be utf-8!
> 
> No, in fact Usenet (and mail) abounds with a large variety
> of untagged 8-bit charsets.

Sorry, none of that is validly encoded, and has specifically NO meaning,
unless assigned one.

Backwards compatibility with standards compliant messages, that I
understand, but backwards compatiblity with invalidly encoded messages?

> >And a langugae tag is only allowed for paramaters, and even there is
> >optional, is it not?
> 
> No, language-tagging is provided by MIME for RFC 2047
> encoded-words also.

And is still optional, so does nothing to explain why you would make the
statement "no existing or future injection agent can transform any
untagged 8-bit content without charset and language information".

Apparently there are problems, but lack of charset information and
language tagging doesn't seem to be.

Cheers,
Sam

To: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 11 Feb 2003 15:19:32 -0800
Lines: 18
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> DNS zone files are an excellent example of a bad example.  IETF has no
>> business standardizing those storage formats.

> strongly disagree.  believe it or not, there are protocols useful to the
> network that don't necessarily involve sending bits over the network.

I really can't agree on this point.  Standardization of the zone file
format was and remains a horrible idea.  It's resulted in a ton of
BIND-specific crap creeping into the DNS standards that has interfered
with real interoperability with other implementations based on different
underlying concepts, because the zone file format embeds assumptions about
how the servers work that aren't actually necessary portions of the
protocol.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 11 Feb 2003 17:21:21 -0500
From: Keith Moore <[email-masked]>
To: [email-masked] (WJCarpenter)
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> km> nor is this the only IETF protocol that specifies a file format.
> km> DNS zone files and MIBs are other examples.
> 
> DNS zone files are an excellent example of a bad example.  IETF has no
> business standardizing those storage formats. 

strongly disagree.  believe it or not, there are protocols useful to the
network that don't necessarily involve sending bits over the network.

Date: Tue, 11 Feb 2003 17:20:15 -0500
From: Keith Moore <[email-masked]>
To: [email-masked] (Claus =?ISO-8859-1?Q?F?=�rber)
Cc: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> It does not have to stay that way. The format of known headers is known.

the format of header fields that are published in RFCs is known.
there are many header fields that have never been formally defined, or if they
are defined, their definitions are not widely known.
maybe we can pretend that they're unknown and that will be good enough,
especially if new 'self-describing' fields can be distinguished somehow
from legacy fields.

Keith

Date: 11 Feb 2003 13:42:00 +0100
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Sam Roberts <[email-masked]> schrieb/wrote:
> - objections to utf-8 as a standard/"priviledged" character set

>   Fair enough. It has some problems, like any possible charset probably
>   might.

Well, there are still contenders to UTF-8, such as GB18030, byText, to
only name a few.

>  However, a number of IETF protocol families, like PKIX, are going
>  utf-8, rather than have to deal with multiple national language
>  encodings.

Given then wide range of upcoming protocols using UTF-8 (or another
Unicode encoding) as the standard charset, the contenders IMO only have
a chance as an ``internal encoding of Unicode'' (where they might be
superior).

Future users will maybe wonder why all networks protocols use such a
funny ``Unicode-UTF-8'' encoding of the charset that is in widespread
use then, converting to and from which includes so strange combining and
bidirectinality rules. But I believe that Unicode will continue to be
able to represent all character sequences (UTF-8 can theoretically be
extended up to 42 bits).

> - It is incompatible with RFC[2]822

The problem here seems to be that UTF8 messages might leak out of a
confined UTF8 environment. This does happen with binary messages, too.

> The interesting questions seem to be:

> 1 - does this mean that it can't be standardized?

It could be standardised as an SMTP extension, similar to 8BITMIME.

> 2 - can a utf-8 encoded message be down-coded during transport?

This is the real problem and IMO the only argument against UTF-8 (but
also a very strong one). It seems impossible due to the problems that
you describe here:

>   This is the real problem, it seems, and it seems to be a fundamental
>   property of the RFC822 format: the header field formats aren't
>   self-describing. Its not possible to know whether a header field is
>   unstructured, structured, and if structured, whether words are allowed
>   to be encoded.

At least, it has looked like this to me and others here. But now that
you spell out the problem so clearly, I see a possible solution:

It does not have to stay that way. The format of known headers is known.
The format of new headers can be made self-describing to the extent
where it's possible to encode them without losses. We just have to
define a new general format and encoding to be used for all headers
defined in the future. (Actually, ``self-describing'' could also mean
that only an encoding for characters is defined and that all user agents
have to support it).

A gateway could then encode the header the ``old'' way if it is one from
the set of the headers known at day X and the ``new'' way otherwise.

But remember that the use of Unicode in ``names'' (addresses, newsgroup
names) is a completly different issue. Names need to have a canonical
encoding and have to be copyable even if fonts are missing.

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: Tue, 11 Feb 2003 16:46:52 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Brad Templeton <[email-masked]>
CC: [email-masked], [email-masked], WJCarpenter <[email-masked]>
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Brad Templeton wrote:
> On Tue, Feb 11, 2003 at 03:01:31PM -0500, Bruce Lilly wrote:

>>To name just one case; forwarding of old messages, either via the
>>infrequently used 822/2822 Resent- header fields, or by MIME
>>message/rfc-822 encapsulation, or by cut-and-paste.
> 
> 
> Ok, so assume for a moment that while we can (someday) prohibit the
> use of the older format on usenet, reading tools probably have to
> be told that they must decode the older 7 bit format forever, in case
> they need to read an old article.    Forwarding is not an issue, as
> tools can be told to map the old 7 bit encoding to the new one in that
> case.

No, that's then not forwarding a message, it's sending a modified
copy of a message.  The obvious case where that won't work is
where the original message in question is being sent as an example
of a message with certain header content.

A Usenet-specific example is the periodic posting of FAQs and similar
articles, some of which haven't changed in years.

Date: Tue, 11 Feb 2003 12:16:00 -0800
From: Brad Templeton <[email-masked]>
To: [email-masked], [email-masked]
Cc: WJCarpenter <[email-masked]>
Subject: Re: Upgrading to UTF-8
Mail-Followup-To: Brad Templeton <[email-masked]>, [email-masked], [email-masked], WJCarpenter <[email-masked]>
Organization: http://www.templetons.com/brad
Sender: [email-masked]

On Tue, Feb 11, 2003 at 03:01:31PM -0500, Bruce Lilly wrote:
> WJCarpenter wrote:
> >km> it takes a very long time before support is no longer required -
> >km> people still want to be able to read old email or news messages.
> >
> >I'm a little confused by this line of reasoning.  We're talking about
> >wire formats, not storage formats, aren't we?  What has reading old
> >anything got to do with protocols?
> 
> To name just one case; forwarding of old messages, either via the
> infrequently used 822/2822 Resent- header fields, or by MIME
> message/rfc-822 encapsulation, or by cut-and-paste.

Ok, so assume for a moment that while we can (someday) prohibit the
use of the older format on usenet, reading tools probably have to
be told that they must decode the older 7 bit format forever, in case
they need to read an old article.    Forwarding is not an issue, as
tools can be told to map the old 7 bit encoding to the new one in that
case.  Even though that has risks they are fairly minor, and this will
be an increasingly uncommon event.

Note as well, that actions like these are why I proposed the new
semantics for "Distribution."  Once those semantics are in place, you
can be assured that no article with a specific Distribution will be
processed (at least by conformant software) at a site or gateway that
does not subscribe to that distribution.

It thus becomes possible, very quickly, to define subsets of the nets,
which, early on, embrace the 8 bit format.  This requires that people
understand in writing mail gateways that they must not forward to mail
any article that comes with a distribution not explicitly subscribed to,
in this case it may not even be safe to do so.

Because distributions are expected to leak in relaying, just not in
local data storage and reading, all relay servers must be ready to handle
the new format before this can be done, however.

Date: Tue, 11 Feb 2003 15:01:31 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: WJCarpenter <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

WJCarpenter wrote:
> km> it takes a very long time before support is no longer required -
> km> people still want to be able to read old email or news messages.
> 
> I'm a little confused by this line of reasoning.  We're talking about
> wire formats, not storage formats, aren't we?  What has reading old
> anything got to do with protocols?

To name just one case; forwarding of old messages, either via the
infrequently used 822/2822 Resent- header fields, or by MIME
message/rfc-822 encapsulation, or by cut-and-paste.

Date: Tue, 11 Feb 2003 11:56:31 -0800
From: [email-masked] (WJCarpenter)
To: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

km> nor is this the only IETF protocol that specifies a file format.
km> DNS zone files and MIBs are other examples.

DNS zone files are an excellent example of a bad example.  IETF has no
business standardizing those storage formats.  Zone transfer formats,
sure; storage formats, no.
-- 
[email-masked] (WJCarpenter)    PGP 0x91865119
38 95 1B 69 C9 C6 3D 25    73 46 32 04 69 D6 ED F3

Date: Tue, 11 Feb 2003 14:01:10 -0500
From: Keith Moore <[email-masked]>
To: [email-masked] (WJCarpenter)
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> Yes, but that's an MUA issue.  I repeat: what has this got to do with
> IETF wire protocols?

the email protocol is not just used on wires (or fiber);
interoperability in other contexts is also important.

nor is this the only IETF protocol that specifies a file format.  DNS
zone files and MIBs are other examples.

> Earlier, someone said "on most systems, the wire format and the
> storage format are essentially thesame.  (modulo line-ending
> differences)".
> 
> In the first place, so what?  Extending an IETF wire protocol will not
> erase your MUA from your disk drive. 

nor will it upgrade your SMTP server.

> In the second place, though, I think the claim "on most systems" is
> not correct.  Unix-centric minds think that way, but I'm pretty
> confident that the vast majority of individual users are not in that
> situation. 

I suspect you're wrong about "vast majority".  Significant numbers, yes.
But even proprietary systems are sometimes based on off-the-shelf
tools and standard message formats.  Why pay for the inefficency of
conversion when it's not needed? But even if you're right, it doesn't
matter. The number of mail messages that are stored in wire format is
still significant.  We don't need to break compatibility of folks using
them even if there are large numbers of folks not using them.

I don't think IETF wants to encourage obsolence of data formats in
general, much less email messages in particular.  One reason we still
use ASCII in RFCs is that it's the best archival format for text that
we've found.

Keith

From: "Shmuel (Seymour J.) Metz" <[email-masked]>
Date: Tue, 11 Feb 2003 13:36:24 -0500
To: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Organization: Atid/2
Sender: [email-masked]

In <[email-masked]>, on 02/11/2003
   at 11:49 AM, Bruce Lilly <[email-masked]> said:

>One fundamental problem is that language-tagging, which is an
>essential part of internationalization, is ignored.  Another problem
>is that text in an untagged charset conveys no information about its
>charset

Those are currently noncompliant. Any article complying with RFC 1036
will not have that problem, because it will be ASCII. Any article
complying with the draft will not have that problem, because the draft
specifies the charset. So your complaint resolves to saying that under
this draft you wouldn't be able to identify the character set of a
noncompliant article.

-- 
-----------------------------------------------------------
     Shmuel (Seymour J.) Metz, SysProg and JOAT
     Atid/2
     Team OS/2
     Team PL/I
-----------------------------------------------------------

Date: Tue, 11 Feb 2003 13:50:27 -0500
From: Keith Moore <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> > and note that I didn't say 'indefinitely'.  but lots of people keep
> > old mail around.  while it's hard to argue that people should keep
> > buying 5.25" disk drives for the sake of reading their old floppies
> > (partially because the medium is essentially useless for storage of
> > new things), it's a bit easier to argue that MUAs should contain a
> > bit of extra code to handle slight differences in message format
> > that were once widely used.
> 
> That doesn't, however, imply that a standard should specify that
> behavior.

no it doesn't imply that.  however I believe it is appropriate for a
standard to do so, at least for several more years.

Keith

Date: Tue, 11 Feb 2003 10:47:16 -0800
From: [email-masked] (WJCarpenter)
To: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

km> and note that I didn't say 'indefinitely'.  but lots of people
km> keep old mail around.  while it's hard to argue that people should
km> keep buying 5.25" disk drives for the sake of reading their old
km> floppies (partially because the medium is essentially useless for
km> storage of new things), it's a bit easier to argue that MUAs
km> should contain a bit of extra code to handle slight differences in
km> message format that were once widely used.

Yes, but that's an MUA issue.  I repeat: what has this got to do with
IETF wire protocols?

Earlier, someone said "on most systems, the wire format and the
storage format are essentially thesame.  (modulo line-ending
differences)".

In the first place, so what?  Extending an IETF wire protocol will not
erase your MUA from your disk drive.  You can still use it to read old
email.

In the second place, though, I think the claim "on most systems" is
not correct.  Unix-centric minds think that way, but I'm pretty
confident that the vast majority of individual users are not in that
situation.  Their email is gatewayed into some proprietary format
either by their service provider or by their MUA (think AOL, think MS
Outlook).  Email geeks like us get asked on a regular basis "how do I
rescue my old email which is trapped by XZY program in some
proprietary format?"  For those users, one email client switcheroo is
all it takes to wipe the slate clean of the "old email"
consideration.

In the third place, I'd be willing to bet that people who are in the
situation of having vast archives of old email stored in a format
essentially the same as the wire format also fall into one or both of
these buckets:

  * The format is one of 3 or 4 extremely popular storage formats, and
    handy converters will spring up like tampopos in springtime.

  * The MUA in question is one of 10 or 12 extremely popular MUAs,
    probably with readily available source code, and their popularity
    will make them among the first to be converted to anything new.

Hey, I'm completely sympathetic to people who have vast email
archives.  I'm one of those packrats.  But I consider that *my*
problem to keep up with, not the IETF's.
-- 
[email-masked] (WJCarpenter)    PGP 0x91865119
38 95 1B 69 C9 C6 3D 25    73 46 32 04 69 D6 ED F3

To: [email-masked]
Subject: Re: Upgrading to UTF-8
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 11 Feb 2003 10:38:46 -0800
Lines: 33
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

> and note that I didn't say 'indefinitely'.  but lots of people keep old
> mail around.  while it's hard to argue that people should keep buying
> 5.25" disk drives for the sake of reading their old floppies (partially
> because the medium is essentially useless for storage of new things),
> it's a bit easier to argue that MUAs should contain a bit of extra code
> to handle slight differences in message format that were once widely
> used.

That doesn't, however, imply that a standard should specify that behavior.

Consider this analogy:  a few years ago, a new version of the C standard
was released.  It has some incompatibilities with the previous version of
the C standard, although not a whole lot.  But it is possible for
previously conforming programs to not compile under the new standard in
certain edge cases.

Does the new C standard mandate that all compilers be able to handle both
the new and old standard?  No, of course not.

Do most compilers implementing C99 have multiple modes so that you can
tell them to process a source file as C89 or C99?  Yes, in my experience.

Vendors understand the need for backward compatibility with their existing
install base all by themselves.  There's no need to mandate that in a
standard.  (Now, mandating the ability to understand *new* e-mail messages
that are still seen on the network, generated by old software, is another
matter and a proper focus of an IETF standard IMO.  But mandating the
ability to understand old mail archives is not, again IMO.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 11 Feb 2003 12:22:18 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: [email-masked], [email-masked]
CC: Sam Roberts <[email-masked]>
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Bruce Lilly wrote:

> The rules for encoded-words in header fields are simple and clear:
> 1. never anywhere in any Received field (it's not clear *why*, but
>    that's the rule).  Of course. that's not a news issue.
> 2. never in a MIME parameter (2231 extended parameters may be used there)
> 3. any "word" in a "phrase" (RFC 822 definitions), separated from
>    adjacent graphic characters by whitespace
> 4. in a comment, adjacent to an unquoted parenthesis or separated
>    from other adjacent graphic characters by whitespace
> 5. in unstructured text ("*text" per RFC 822 definitions)
> The last three rules can be compressed to one; they comprise all of
> the places where human-readable text (vs. protocol elements) are
> located.

Small corrigenda:
for item 2 (which is merely a clarification of 3-5), substitute:

2. never in a quoted-string

change "three" to "four" in the last sentence,
and add "except quoted-strings in a phrase" to the last sentence.

To: [email-masked]
Xref: clerew local.mime:2386
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Upgrading to UTF-8
Date: Tue, 11 Feb 2003 15:54:56 GMT
Lines: 127
Sender: [email-masked]

In <20030211024241.GB15342@debian> Sam Roberts <[email-masked]> writes:

>All currently valid email is 7bit/ASCII. Its meaning will not change if
>future email defines a meaning to 8bit message headers, and assigns that
>meaning to be some character set, such as utf-8.

>So, it is backwards compatible in this sense, is it not?

Yes.

>In theory, it is not backwards compatible with the SMTP transport, since
>that expects messages to be 7bit ASCII.

I prefer to use the term "forwards compatible" to describe the situation
where some newly introduced feature (e.g. UTF-8) can interwork
successfully with the existing installed base. So no, UTF-8 is not
forwards compatible with SMTP (though in practice most current
implementations pass it unchanged, the principal exception being sendmail,
and that will change with the next major upgrade, if not before).

However, Usefor does not permit sending raw UTF-8 in email, so the
question SHOULD NOT arise (though it probably will :-( ).

>In practice, I get 8bit messages (mostly spam, but some from native
>French speakers) very frequently.

The French Usenet hierarchy took a unilateral decision to use raw ISO8859-1
in headers, because available implementations of RFC 2047 were so buggy. I
think they have been persuaded of the error of their ways, at least to the
extent of switching to UTF-8.

>So, a sender of a message with utf-8 in the headers may find it not
>delivered. This doesn't sound like a catastrophic break in the current
>messaging system. It actually sounds like the only people who will
>notice are the senders and receivers, and nobody else.

Ssshhhhhh! You are not supposed to utter heresies like that on this list :-(.
But yes, that is precisely the position taken by the Usefor draft. The
main problem seems to be that if a news2mail gateway (e.g. from a
particular newsgroup to a mailing list) is not upgraded, then all the
people on the mailing list will notice; but at least then the fix needs
only be applied at the one place and, for an existing English-speaking
newsgroup, which most of them are, it is not likely to be an issue
anyway).

>The main objections to utf-8 becoming the "native" charset of internet
>messages seems to be:

>- objections to utf-8 as a standard/"priviledged" character set

There really is bo other viable candidate for the privilege. The Chinese
might make claims for GB18030 but there is zilch chance of the rest of the
world agreeing to that.

>- It is incompatible with RFC[2]822

>  In my mind, it is a "compatible" extension of RFC2822. It does not
>  change the meaning of any currently valid messages.

It is backwards compatible. Forwards compatibility is an issue for the
transport and user agents.

>- It is incompatible with SMTP.

Yes. See above.

>The interesting questions seem to be:

>1 - does this mean that it can't be standardized?

>  It WILL be transported by some SMTP implementations, and by all NNTP
>  ones. 

Implementations will certainly catch up within the next few years, but ...

>  But, I can see a strong objection to allowing a message format
>  that "may or may not be" transportable.

I have some ideas on that. A header that says "this messages uses 8-bit"
might be necessary.

>2 - can a utf-8 encoded message be down-coded during transport?

>  This is the real problem, it seems, and it seems to be a fundamental
>  property of the RFC822 format: the header field formats aren't
>  self-describing. Its not possible to know whether a header field is
>  unstructured, structured, and if structured, whether words are allowed
>  to be encoded. Because of that, its not possible to encode/decode
>  without knowing the field definition, and an automated grep of
>  all RFCs to determine it would be a little much to ask.

This is essentially an RFC 2047 problem, because it requires that an
implementation already knows whether a header is srtuctured or not, and if
so how. The position that Usefor now takes is that gateways MUST be able
to recognize all the official News headers (and the usual mail ones as
well) but that any header not recognized should have an "X-" stuck on the
front of it and then be treated as unstructured. Since most 'strange'
headers likely to be found in Usenet articles are for human consumption
only, this is not likely to result in any breaches of protocol.

>  So, you can only transform some fields. Like the ones that you know
>  are allowed to containt utf-8, because they are in the USEFOR draft.

Exactly so.

>  What about the others? What about throwing experimental headers that
>  have binary in them away? Or leaving them, at the gateway admins
>  option, raw.

There are no such binary headers defined in Usefor, so no problem. The
nearest is the Newgroups-header, and there is a special encoding defined
for that, for use in such emergencies.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 11 Feb 2003 12:10:16 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> > but legacy UAs that did support RFC 2047 would not necessarily be
> > able to parse utf-8.
> 
> Ditto for legacy UAs that don't support RFC 2047.
>

I wish I could say that there were none of those left, but I'd be wrong.

Date: Tue, 11 Feb 2003 12:07:59 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: Sam Roberts <[email-masked]>, [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Keith Moore wrote:

> but legacy UAs that did support RFC 2047 would not necessarily be able to
> parse utf-8.

Ditto for legacy UAs that don't support RFC 2047.

Date: Tue, 11 Feb 2003 11:49:22 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Sam Roberts <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Sam Roberts wrote:
> Quoteing [email-masked], on Mon, Feb 10, 2003 at 02:57:26PM -0500:
> 
>>Because there are old messages, all of the existing methods need to
>>continue to be supported indefinitely so that those old messages can
>>still be read.  A transition requires backward compatibility so that
>>the infrastructure doesn't suddenly break (as that would not constitute
>>a transition), and it requires a feasible plan.  The Usefor draft
>>breaks backward compatibility and provides no feasible transition plan.
>>That's not "working on solutions", that's *compounding* the real problem.
> 
> 
> All currently valid email is 7bit/ASCII. Its meaning will not change if
> future email defines a meaning to 8bit message headers, and assigns that
> meaning to be some character set, such as utf-8.
> 
> So, it is backwards compatible in this sense, is it not?

No, because the existing infrastructure is designed according to
the current standards, where 8-bit content in message and MIME-part
header fields is illegal.  Presenting that infrastructure with
illegal content is not backwards compatible.  A transition plan
could address change in stages. The first stage must include the
proviso that content generators only generate content which is
backwards compatible, i.e. compliant with the prior specification
to which the existing infrastructure has been designed.  What you
describe is non-obsolescence of existing legal content, which is
an often important, but different, issue.

> In theory, it is not backwards compatible with the SMTP transport, since
> that expects messages to be 7bit ASCII.

No theory about it; it's not compliant with the current (or prior)
SMTP standard.

> In practice, I get 8bit messages (mostly spam, but some from native
> French speakers) very frequently.

It's not standards-compliant, is it?  Good way to identify spam.

> So, a sender of a message with utf-8 in the headers may find it not
> delivered. This doesn't sound like a catastrophic break in the current
> messaging system. It actually sounds like the only people who will
> notice are the senders and receivers, and nobody else.

Standards compliance doesn't have a Richter scale; compliance,
like pregnancy, is a binary condition -- either content presented
to the infrastructure is standards-compliant or it is not.
Compliance is not measured by delivery failure or software crashes,
etc.; it is measured by the totality of the requirements presented
in the relevant standard (which might include, among other
requirements, reliable delivery and/or no crashes on illegal input).

> The main objections to utf-8 becoming the "native" charset of internet
> messages seems to be:
> 
[...]
> - It is incompatible with RFC[2]822
> 
>   In my mind, it is a "compatible" extension of RFC2822. It does not
>   change the meaning of any currently valid messages.

But it presents illegal content to 2822-compliant software and
networks.

>   A utf-8 message, of course, does NOT have a defined meaning to an
>   RFC822 UA. One could argue that neither do RFC2047 encoded messages.
>   Seeing =?iso-8859-1?b?45;lakdfj322lkdkd?= as a subject isn't much
>   better than how my UA displays Korean.

[ignoring the errors in what I assume to be intended as an encoded-word]
It is better in a number of respects:
1. it is fully standards-compliant
2. the charset is clearly indicated. Not all systems support all
    charsets, and at least with an indication of charset, software
    can determine if the requested charset is supported, and if it
    is not, the encoded form can be displayed.
3. the encoded form, if and when displayed, at least is legible. It
    is so precisely because it consists entirely of characters which
    are common to (essentially) all systems, viz. a specific subset
    of the graphic characters of ANSI X3.4.
4. it passes the "telephone test"; if you need to read it aloud
    (or transcribe it to paper, etc.) you can do so unambiguously
    (see RFC 2396 for a description of these issues, which were also
    considered in the design of URI syntax).
5. it can convey language information:
      Subject: =?ibm367*en-us?boot?=
      Subject: =?ibm367*en-uk?boot?=
    and
      Subject: =?ibm367*de?boot?=
    mean different things.

Returning to your first point in the above quote, some clarification
is in order.  It is true that neither text strings nor protocol
elements containing any 8th-bit-high octect have any meaning
(other than illegal content) to any software (not only UAs) compliant
with RFCs 561, 680, 724, 733, 821, 822, 2821, or 2822 (and all of
the MIME RFCs as far as MIME-part header fields and many media
types' body content are concerned).  RFC 2047 encoded-words *never*
apply to protocol elements; they are used solely for human-readable
text, and as such have meaning only to humans, never to software.
But the protocol elements in a text message *do* have prescribed
meaning to software, whether or not encoded-words are used (in
legal context) anywhere in the same message, whether or not those
protocol elements happen to appear to be similar to the form of an
encoded-word. E.g.
    Message-ID: (=?ibm367*en?q?encoded_comment?=) <=?us?b?c===?=@foo.net>
has a legal left-hand side in the msg-id which is *not* an
encoded-word, and that msg-id is used exactly like any other msg-id.

[...]
> The interesting questions seem to be:
> 
> 1 - does this mean that it can't be standardized?
> 
>   It WILL be transported by some SMTP implementations, and by all NNTP
>   ones. But, I can see a strong objection to allowing a message format
>   that "may or may not be" transportable.

"some" isn't adequate; a sender expects his message to be delivered.
That can be handled with negotiation and fallback  And don't forget IMAP,
which is used for both news and mail.

> 2 - can a utf-8 encoded message be down-coded during transport?
> 
>   This is the real problem, it seems, and it seems to be a fundamental
>   property of the RFC822 format: the header field formats aren't
>   self-describing. Its not possible to know whether a header field is
>   unstructured, structured

It is possible for all standard fields because the standard
indicates whether or not the field is structured, and if structured,
what the specific structure is.  It is true that user-defined
("x-" prefixed) fields, non-standard fields, and extension fields
designed after a given piece of software was implemented cannot be
identified as structured or unstructured.  Note that that is not a
problem for the intended purpose, viz. display, but is is a problem
with any attempt to bend, fold, spindle, or otherwise mutilate
messages in transport.

 > and if structured, whether words are allowed
 >   to be encoded.

The rules for encoded-words in header fields are simple and clear:
1. never anywhere in any Received field (it's not clear *why*, but
    that's the rule).  Of course. that's not a news issue.
2. never in a MIME parameter (2231 extended parameters may be used there)
3. any "word" in a "phrase" (RFC 822 definitions), separated from
    adjacent graphic characters by whitespace
4. in a comment, adjacent to an unquoted parenthesis or separated
    from other adjacent graphic characters by whitespace
5. in unstructured text ("*text" per RFC 822 definitions)
The last three rules can be compressed to one; they comprise all of
the places where human-readable text (vs. protocol elements) are
located.

 > Because of that, its not possible to encode/decode
>   without knowing the field definition, and an automated grep of
>   all RFCs to determine it would be a little much to ask.
[...]
>   So, you can only transform some fields. Like the ones that you know
>   are allowed to containt utf-8, because they are in the USEFOR draft.
> 
>   What about the others? What about throwing experimental headers that
>   have binary in them away? Or leaving them, at the gateway admins
>   option, raw.
> 
>   What are the problems with this approach, operationally?

One fundamental problem is that language-tagging, which is an
essential part of internationalization, is ignored.  Another problem
is that text in an untagged charset conveys no information about its
charset (it is not self-describing, to use your words; how many of
those messages with 8-bit content that you have received were in
utf-8?).  An encoded-word (or extended parameter) has provision for
both charset and language and that information can't be pulled out of
thin air.  Note that charset is not optional in an encoded-word, and
as detailed elsewhere, language information MUST be carried if
desired by the originating user.  Throwing away message content is
undesirable; transports shouldn't do that (and header fields,
experimental or not, are part of the message content). Passing
8-bit data to SMTP or IMAP is not standards-compliant.  Finally,
current gateways, injection agents, etc. are not expected to
handle 8-bit content and do not perform any such transformations --
so even if it were possible to perform such a transformation in
the absence of charset or language information and without knowledge
of the field syntax, a transition period would be required *before*
any content with raw utf-8 was generated so that all of the injection
agents, gateways, etc. could be modified.

One of the design principles (if I may temporarily jump ahead to
another topic you raised) is that of end-to-end communications.
So if a message-originating user chooses a valid charset and
specifies a language for some text, the charset and language
information should be carried from the point of origination,
maintained intact through transport, and presented to the
receiving users in an appropriate way (by using the specified
charset or indicating that it is unavailable, and by using the
language information if appropriate, as for a screen reader for
the visually impaired).  It should not be immediately discarded
by the UA and then a half-assed attept made to recreate it from
thin air in mid-transit.  That is a fundamental issue; charset
and language information must be carried with the message from
the moment of origination, and that precludes any transformation
approach (barring some explicit means to preserve charset and
language information).  In any event, with a proper transition
plan such a transformation would not be necessary.

> This seems to be a really important issue, and speaking as an
> implementor, if the mail standards HAVE to be as baroque and difficult
> as they are, fine, I can deal keep dealing with them, but, I would
> really, really, like to know the design rationale, because utf-8 sure
> does seem like it would solve a whole lot of problems.

UTF-8 per se doesn't really solve any problems. As a charset, it is
subject to the same limitations as all charsets (e.g. support is
not necessarily universal; while a sender may have an implementation
of a particular charset, his recipient(s) might not).  Considered as
an (8-bit) encoding of (32-bit) ISO 10646, there is another, not
entirely unrelated, issue, namely that there are different versions
of the underlying character set (in the ISO 10646/Unicode sense,
which is not the same as charset).  One can, of course, use utf-8
as the charset specified with MIME (RFC 2047/2231) methods, with
all of the advantages of those methods (standards-compliance, clear
indication of charset, legible fallback display, transcribability,
language-tagging, etc.) and with exactly the same characteristics
inherent in utf-8 noted in the earlier part of this paragraph.

> The RFCs are fairly lacking in any "design and architecture of the IETF
> text messaging system" section,

It's mostly about interoperability. While not specific to text
messaging, see RFC 1958.  Text messaging specific information does
appear in the various relevant RFCs, often the older, obsoleted
ones more so than the current ones -- see RFCs 724 and 2045-7 for
examples of old and current ones which do discuss design choices.

Date: Tue, 11 Feb 2003 10:31:56 -0500
From: Keith Moore <[email-masked]>
To: Sam Roberts <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> > > Won't UAs that support raw utf-8 in headers but don't support decoding
> > > 2047 tagging in header fields work exactly as "well" as the UAs that
> > > existed when RFC 2047 was first implemented?
> > 
> > no.  because legacy UAs could still parse, and correctly reply to,
> > messages with RFC 2047 (and its predecessors) encoded-words.
> 
> Which is exactly what a future, hypothetical, UA that did not
> support RFC 2047 encoded-words would do.

but legacy UAs that did support RFC 2047 would not necessarily be able to
parse utf-8.

Date: Tue, 11 Feb 2003 09:52:03 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Upgrading to UTF-8
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Tue, Feb 11, 2003 at 05:52:06AM +0000:
> On Feb 10,  9:42pm, Sam Roberts wrote:
> }
> } This seems to be a really important issue, and speaking as an
> } implementor, if the mail standards HAVE to be as baroque and difficult
> } as they are, fine, I can deal keep dealing with them, but, I would
> } really, really, like to know the design rationale, because utf-8 sure
> } does seem like it would solve a whole lot of problems.
> 
> Somebody must have made a similar speech about HDTV a few years ago.
> How likely is it, really, that all stations in the U.S. will broadcast
> exclusively in digital by Dec. 31, 2006?  If they actually do, what
> proportion of television sets will be able to receive it?

Broken analogy:

My analog TV does NOTHING with HDTV signals.

My UA and incoming SMTP server delivers email with 8bit chars routinely.

TV is broadcast, email is point-to-(small number of points).

Sam

Date: Tue, 11 Feb 2003 09:48:49 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Upgrading to UTF-8
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Tue, Feb 11, 2003 at 08:26:12AM -0500:
> > > it takes a very long time before support is no longer required - people 
> > > still want to be able to read old email or news messages.
> > 
> > Won't UAs that support raw utf-8 in headers but don't support decoding
> > 2047 tagging in header fields work exactly as "well" as the UAs that
> > existed when RFC 2047 was first implemented?
> 
> no.  because legacy UAs could still parse, and correctly reply to,
> messages with RFC 2047 (and its predecessors) encoded-words.

Which is exactly what a future, hypothetical, UA that did not
support RFC 2047 encoded-words would do.

Sam

Date: Tue, 11 Feb 2003 09:19:36 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
CC: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

D. J. Bernstein wrote:
> Bruce Lilly writes:
> 
>>The current Usefor draft requires UAs to support raw utf-8, which is
>>an encoding that is currently not required to be supported by text
>>message UAs.  That's not "working on solutions", that's *compounding*
>>the real problem.
> 
> 
> Please explain. Precisely what problem do you believe is created by the
> requirement that all message handlers accept UTF-8?

N.B. I did not say it "created" a problem, I clearly stated that
it *compounded* the problem.

One problem is the one posited by Dave Barr, to which the quoted
text is a response. I.e. the requirement to support N encodings
becomes a requirement to support N+1 encodings.

Date: Tue, 11 Feb 2003 09:09:16 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Sam Roberts <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Sam Roberts wrote:
> Quoteing [email-masked], on Mon, Feb 10, 2003 at 05:39:26PM -0500:
> 
>>>Yawn! The draft does not require, or even allow, raw UTF-8 to be passed to
>>>moderators. If you think it does, then please put up or shut up.
>>
>>The draft permits a UA to generate raw utf-8. That is then passed to
>>an injection agent, which determines that one or more newsgroups are
>>moderated.  Existing injection agents do not transform raw-utf-8,
>>and no existing or future injection agent can transform any untagged
>>8-bit content without charset and language information.
> 
> 
> Why not?
> 
> The charset seems clearly to be utf-8!

No, in fact Usenet (and mail) abounds with a large variety
of untagged 8-bit charsets.

> And a langugae tag is only allowed for paramaters, and even there is
> optional, is it not?

No, language-tagging is provided by MIME for RFC 2047
encoded-words also. See RFC 2231 and the errata page at
http://www.rfc-editor.org/errata.html.  Along with the
Content-Language field for body text, MIME provides full
compliance with RFCs 1958 and 2277 for internationalization
including language-tagging.  And 2277 is quite clear that
there must be provision for language-tagging if the *user*
chooses to tag text; it is not acceptable for software to
drop language information if supplied by the user. The
"optional" part is for user discretion, not for lazy
programmers.

Date: Tue, 11 Feb 2003 08:30:22 -0500
From: Keith Moore <[email-masked]>
To: Philip Hazel <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> > Because there are old messages, all of the existing methods need to
> > continue to be supported indefinitely so that those old messages can
> > still be read.
> 
> Analogy: because there is still data on old 5.25" floppy discs, all
> computers need to be able to read them indefinitely.
> 
> Of course not. If you want to read an old 5.25" floppy, you find a
> conversion service.
> 
> *Something* needs to be able to read old messages indefinitely, but I
> don't see why it has to be *everything*.

and note that I didn't say 'indefinitely'.  but lots of people keep old mail
around.   while it's hard to argue that people should keep buying 5.25" disk
drives for the sake of reading their old floppies (partially because the 
medium is essentially useless for storage of new things), it's a bit easier
to argue that MUAs should contain a bit of extra code to handle slight
differences in message format that were once widely used.

Date: Tue, 11 Feb 2003 08:27:01 -0500
From: Keith Moore <[email-masked]>
To: [email-masked] (WJCarpenter)
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> km> it takes a very long time before support is no longer required -
> km> people still want to be able to read old email or news messages.
> 
> I'm a little confused by this line of reasoning.  We're talking about
> wire formats, not storage formats, aren't we?  What has reading old
> anything got to do with protocols?

on most systems, the wire format and the storage format are essentially the
same.  (modulo line-ending differences).

Date: Tue, 11 Feb 2003 08:26:12 -0500
From: Keith Moore <[email-masked]>
To: Sam Roberts <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> > it takes a very long time before support is no longer required - people 
> > still want to be able to read old email or news messages.
> 
> Won't UAs that support raw utf-8 in headers but don't support decoding
> 2047 tagging in header fields work exactly as "well" as the UAs that
> existed when RFC 2047 was first implemented?

no.  because legacy UAs could still parse, and correctly reply to,
messages with RFC 2047 (and its predecessors) encoded-words.

Date: Tue, 11 Feb 2003 09:27:18 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: [email-masked], <[email-masked]>
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

On Mon, 10 Feb 2003, Bruce Lilly wrote:

> Because there are old messages, all of the existing methods need to
> continue to be supported indefinitely so that those old messages can
> still be read.

Analogy: because there is still data on old 5.25" floppy discs, all
computers need to be able to read them indefinitely.

Of course not. If you want to read an old 5.25" floppy, you find a
conversion service.

*Something* needs to be able to read old messages indefinitely, but I
don't see why it has to be *everything*.

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Mon, 10 Feb 2003 21:09:32 -0800
From: [email-masked] (WJCarpenter)
To: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

km> it takes a very long time before support is no longer required -
km> people still want to be able to read old email or news messages.

I'm a little confused by this line of reasoning.  We're talking about
wire formats, not storage formats, aren't we?  What has reading old
anything got to do with protocols?
-- 
[email-masked] (WJCarpenter)    PGP 0x91865119
38 95 1B 69 C9 C6 3D 25    73 46 32 04 69 D6 ED F3

Date: Mon, 10 Feb 2003 23:47:21 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Upgrading to UTF-8
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Mon, Feb 10, 2003 at 11:22:05PM -0500:
> > e) At some point tools that generate the 7 bit format are all but gone.
> > Use of it can be forbidden, and support for it no longer required.
> 
> it takes a very long time before support is no longer required - people 
> still want to be able to read old email or news messages.

Won't UAs that support raw utf-8 in headers but don't support decoding
2047 tagging in header fields work exactly as "well" as the UAs that
existed when RFC 2047 was first implemented?

But I would agree, newsreaders are probably the only large class of
UA that never had to understand MIME charset encoding, so won't have
to implement it to understand old news articles, they could just extend
to understanding utf-8, if USEFOR is allowed their way.

Cheers,
Sam

Date: Mon, 10 Feb 2003 23:22:05 -0500
From: Keith Moore <[email-masked]>
To: Brad Templeton <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> e) At some point tools that generate the 7 bit format are all but gone.
> Use of it can be forbidden, and support for it no longer required.

it takes a very long time before support is no longer required - people 
still want to be able to read old email or news messages.

Date: 11 Feb 2003 04:18:35 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Bruce Lilly writes:
> The current Usefor draft requires UAs to support raw utf-8, which is
> an encoding that is currently not required to be supported by text
> message UAs.  That's not "working on solutions", that's *compounding*
> the real problem.

Please explain. Precisely what problem do you believe is created by the
requirement that all message handlers accept UTF-8?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Mon, 10 Feb 2003 17:33:31 -0800
From: Chris Newman <[email-masked]>
Subject: RE: Upgrading to UTF-8
To: David Barr <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

The IETF has a simple policy: Never break standards-compliant deployed 
software.  We also try not to break non-standards-compliant deployed 
software when reasonable, but that is a second priority.  This policy 
results in cruft and junk in protocols.  It's ugly and unpleasant to 
implement as you correctly point out.  So here's how to succeed at cruft 
removal:
----
Guide to removing cruft from standard protocols:

1. Write a standard extension to negotiate use of the less-crufty protocol, 
and explain how to downconvert to the fully-interoperable crufty protocol.

2. Deploy standards-compliant servers and clients that accept the 
less-crufty protocol, down-convert if necessary, and will generate the 
less-crufty protocol only if negotiated.  These interoperate with all 
software.

3. Deploy non-standard software that generates only the less-crufty 
protocol.  This software isn't guaranteed to interoperate with everything, 
but plays an important role as it creates pressure to upgrade to software 
from step 2.

4. Over time, more and more developers will not bother to implement the 
downconversion and cruft-support.  Eventually the amount of software 
requiring the downconversion is close enough to zero that the standards can 
be changed to drop the requirement.
----

The 8BITMIME SMTP extension did steps 1-3 and is in the middle of step 4. 
Most of the delay in deploying 8BITMIME was fighting between the 
"interoperability with deployed standards" and "cruft removal" camps.  As 
soon as the latter camp gave in, both camps started to get what they wanted.

Bottom line: if you want UTF-8 in headers, write the spec for negotiated 
UTF-8 in headers as SMTP, IMAP and POP extensions with downconversion to 
2047/2231.  When that's written and both camps agree, then we can move 
forward to steps 2-4.

If we continue to argue, we will be stuck with the cruft indefinitely.

                - Chris

begin quotation by David Barr on 2003/2/10 12:54 -0600:

>
>> And if you had a shred of competency as an engineer or
>> protocol designer you'd be working on solutions to the very
>> real problems rather than trying to pretend that they don't exist.
>
> The fact that there are almost a dozen different (and in many cases
> horribly broken) encoding formats that are required now to implement an
> email client/web browser is not a real problem?  The fact that messages
> floating around often have to go through transformation after
> transformation, encoding after encoding, just to get from one place to
> another, and that *any* bugs or gaps with encoding along the way result
> in real breakage? That is not a real problem?  The fact that because of
> typical
> desgin-by-committee and generational effects we now have a series of
> protocols layered on top of each other that are horribly complex, vague,
> and broken?  And the fact that the vast majority of these problems would
> be solved simply by having the balls to once and for all say "we shalt
> transition all this shit to UTF-8"?
>
> Rather than actually solve problems, the IETF would rather add more duct
> tape to an already structurally unsound system.  This disturbs me.
>
> --Dave
>
>

Date: Mon, 10 Feb 2003 21:42:42 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Upgrading to UTF-8
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Mon, Feb 10, 2003 at 02:57:26PM -0500:
> Because there are old messages, all of the existing methods need to
> continue to be supported indefinitely so that those old messages can
> still be read.  A transition requires backward compatibility so that
> the infrastructure doesn't suddenly break (as that would not constitute
> a transition), and it requires a feasible plan.  The Usefor draft
> breaks backward compatibility and provides no feasible transition plan.
> That's not "working on solutions", that's *compounding* the real problem.

All currently valid email is 7bit/ASCII. Its meaning will not change if
future email defines a meaning to 8bit message headers, and assigns that
meaning to be some character set, such as utf-8.

So, it is backwards compatible in this sense, is it not?

In theory, it is not backwards compatible with the SMTP transport, since
that expects messages to be 7bit ASCII.

In practice, I get 8bit messages (mostly spam, but some from native
French speakers) very frequently.

So, a sender of a message with utf-8 in the headers may find it not
delivered. This doesn't sound like a catastrophic break in the current
messaging system. It actually sounds like the only people who will
notice are the senders and receivers, and nobody else.

> >This disturbs me.
> 
> If you genuinely believe that the Usefor draft solves problems rather
> than compounding them, you are indeed seriously disturbed.

The lack of technical content and incredibly personal comments in this
debate disturbs me! I don't understand why its like that.

The main objections to utf-8 becoming the "native" charset of internet
messages seems to be:

- objections to utf-8 as a standard/"priviledged" character set

  Fair enough. It has some problems, like any possible charset probably
  might. The lack of a standard charset has its own set of pretty
  serious drawbacks, no distinguished encoding possible for X.509
  certificates, for example.  However, a number of IETF protocol
  families, like PKIX, are going utf-8, rather than have to deal with
  multiple national language encodings.

- It is incompatible with RFC[2]822

  In my mind, it is a "compatible" extension of RFC2822. It does not
  change the meaning of any currently valid messages.

  A utf-8 message, of course, does NOT have a defined meaning to an
  RFC822 UA. One could argue that neither do RFC2047 encoded messages.
  Seeing =?iso-8859-1?b?45;lakdfj322lkdkd?= as a subject isn't much
  better than how my UA displays Korean.

  Anyhow, there are different shades of backwards compatible. S/MIMEv3
  messages, for example, can fail to be handled by S/MIME agents that
  used to be valid S/MIME implementations. Of course, they should have
  made it through the transport, at least, leading to...

- It is incompatible with SMTP.

  This is true. A valid SMTP implementation does not have to transfer
  messages that aren't pure ASCII. However, they seem to do so fairly
  frequently!

The interesting questions seem to be:

1 - does this mean that it can't be standardized?

  It WILL be transported by some SMTP implementations, and by all NNTP
  ones. But, I can see a strong objection to allowing a message format
  that "may or may not be" transportable.

2 - can a utf-8 encoded message be down-coded during transport?

  This is the real problem, it seems, and it seems to be a fundamental
  property of the RFC822 format: the header field formats aren't
  self-describing. Its not possible to know whether a header field is
  unstructured, structured, and if structured, whether words are allowed
  to be encoded. Because of that, its not possible to encode/decode
  without knowing the field definition, and an automated grep of
  all RFCs to determine it would be a little much to ask.

  Much as I dislike writing BER codecs, I have to admit that ASN.1 and
  XML are better this way.

  So, you can only transform some fields. Like the ones that you know
  are allowed to containt utf-8, because they are in the USEFOR draft.

  What about the others? What about throwing experimental headers that
  have binary in them away? Or leaving them, at the gateway admins
  option, raw.

  What are the problems with this approach, operationally?

This seems to be a really important issue, and speaking as an
implementor, if the mail standards HAVE to be as baroque and difficult
as they are, fine, I can deal keep dealing with them, but, I would
really, really, like to know the design rationale, because utf-8 sure
does seem like it would solve a whole lot of problems.

The RFCs are fairly lacking in any "design and architecture of the IETF
text messaging system" section, "read the mailing list archives" seems
to be the standard cop-out, but the flame to signal ratio is starting to
make my cheeks burn!

Cheers,
Sam

Date: Mon, 10 Feb 2003 21:16:48 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Mon, Feb 10, 2003 at 05:39:26PM -0500:
> >Yawn! The draft does not require, or even allow, raw UTF-8 to be passed to
> >moderators. If you think it does, then please put up or shut up.
> 
> The draft permits a UA to generate raw utf-8. That is then passed to
> an injection agent, which determines that one or more newsgroups are
> moderated.  Existing injection agents do not transform raw-utf-8,
> and no existing or future injection agent can transform any untagged
> 8-bit content without charset and language information.

Why not?

The charset seems clearly to be utf-8!

And a langugae tag is only allowed for paramaters, and even there is
optional, is it not?

Sam

Date: Mon, 10 Feb 2003 18:10:03 -0800
From: Brad Templeton <[email-masked]>
To: Chris Newman <[email-masked]>
Cc: David Barr <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Mail-Followup-To: Brad Templeton <[email-masked]>, Chris Newman <[email-masked]>, David Barr <[email-masked]>, [email-masked], [email-masked]
Organization: http://www.templetons.com/brad
Sender: [email-masked]

Actually, there is another procedure which could be followed when you
don't have the ability to negotiate (as a news poster does not have with
a receiving mail client.)

a) Declare that all new tools be able to accept the new format, both
USENET and MAIL.   Make it clear that if a tool decodes neither the 7
bit and 8 bit formats, it must not simply implement the 7 bit because
that is coming first, it must handle both.

b) At first, however, do not generate the new format.  Generate the
backwards compatible format (7 bit)

c) As a sector of the net (USENET) moves to being able to declare
use of the new format (8 bit) possible because at this point all clients
are required to handle it, it can do so.  All gateways must also be able
to map to the older format if passing material to it.

d) Eventually all tools handle the new format.  Use of the backwards
compatible format can be deprecated.

e) At some point tools that generate the 7 bit format are all but gone.
Use of it can be forbidden, and support for it no longer required.

Date: Mon, 10 Feb 2003 17:39:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
>>
>>>In <[email-masked]> Bruce Lilly <[email-masked]> writes:
>>>
>>>
>>>
>>>>Charles Lindsey wrote:
>>>
>>>
>>>>>In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>>>>>3. There is no backwards compatibility ...
>>>
>>>If there is a feature in some existing protocol which becomes illegal in
>>>the new version, then that is a breech of "backwards compatibility".
> 
> 
>>Sorry, no -- you have that wrong.
> 
> 
>>>If there is a feature in some new protocol that will not work with
>>>implementations written for the old protocol, then that is a breech of
>>>"forwards compatibility".
> 
> 
>>Nope, wrong again.
> 
> 
>>>It seems you want to define these terms the other way around. Does anyone
>>>else here agree with your view? The weight of evidence is surely against
>>>you.
> 
> 
>>Don't take my word for it. Dave Crocker has clearly and unambiguously
>>explained to you what backwards compatibility does and does not
>>mean to the IETF, which is what counts....
> 
> 
>>>Just to check, I grepped through my small collection of online RFCs, and
>>>found usage of "backwards compatibility" in my sense in RFCs 1945, 2060 and
>>>2156. I found only one (RFC 2646) that seemed to use it your way, so the
>>>majority is against you. You are. however, welcome to produce further
>>>examples.
> 
> 
> I have quoted you three examples from IETF Standars-track documents that
> use the term "backwards compatibility" as I have defined it. And yet you
> claim that is not what it means to the IETF, and quote Dave Crocker in
> support of your view.

No, you have not quoted examples, you have listed RFC numbers.  You are
wrong about RFC 2156; here's a quote:

"   There are three places where an order is specified:

    1.   The text encoding (std-or-address) of MTS.ORAddress as used
         in the local-part of an RFC 822 address.  An order is needed
         for those components which may have multiple values
         (Organizational Unit, and Domain Defined Attributes). When
         generating an 822.std-or-address, components of a given type
         shall be in hierarchical order with the most significant
         component on the RHS (right hand side or domain part).  If
         there is an Organization Attribute, it shall be to the right
         of any Organizational Unit attributes.  These requirements
         are for the following reasons:

    -         Alignment to the hierarchy of other components in RFC
              822 addresses (thus, Organizational Units will appear
              in the same order, whether encoded on the RHS or LHS).

    -         Backwards compatibility with RFC 987/1026.
"
That is, as Dave has painstakingly pointed out to you, content
generated by a new implementation is compatible with the existing
installed base which was developed in compliance with earlier
standards.  I haven't checked 1945 or 2060; I assume you're also
wrong about them unless and until you do provide a quote.

You're right about RFC 2646.  And there are also RFCs 939, 1138, 1148,
1327, 1349, 1494, 1554, 1812, 1911, 1995, 1996, 2077, 2157, 2311, 2396,
2421, 2423, 2425, 2440, 2616, 2633, 2652, 2831, 2985, 3183, 3195, 3261.
and 3454, all of which use backwards compatibility in the same sense
as Dave has explained.

No matter; the real issue is what "backwards compatibility" means in
terms of the Usefor WG charter, and it's clear that that means the same
thing as Dave has described.  It's also the same sense in which the term
has previously been used on tthe usenet-format list, e.g. last August.

Lame attempts to redefine the term at this late date in order to try
to wriggle out of the requirement in the charter to pay particular
attention to backwards compatibility simply won't fly.

>>Now, if you want to propose such
>>an incompatible format as part of a separate, parallel implementation,
>>that's another matter.  That may well suffer the same fate as the
>>incompatible multimedia mail implementations mentioned by Dave Crocker.
>>Perhaps not. But existing protocols, software installations, etc. could
>>either continue to support RFC 1036 as is or adopt the hypothetical
>>new format or both, where practical.
> 
> 
> Yes, that is exactly what we are proposing. I am glad that you understand
> at last.

No, that is not what the Usefor draft is proposing; "a separate, parallel
implementation" is inconsistent with "obsoleting RFC 1036".  It's also
incompatible with the WG charter.

>>Indeed we did. And what I actually said is "The draft provides no
>>means of moderation of extended newsgroups which is both acceptable
>>to moderators and compatible with the existing installed base of
>>RFC 1036-compliant UAs and gateways.", and that remains the case.
>>Encapsulation is unacceptable to moderators, and either passing
>>raw utf-8 or attempting to second-guess the UA in a gateway is
>>incompatible with the installed base (for that matter, encapsulation
>>is also incompatible with the existing installed base)..
> 
> Yawn! The draft does not require, or even allow, raw UTF-8 to be passed to
> moderators. If you think it does, then please put up or shut up.

The draft permits a UA to generate raw utf-8. That is then passed to
an injection agent, which determines that one or more newsgroups are
moderated.  Existing injection agents do not transform raw-utf-8,
and no existing or future injection agent can transform any untagged
8-bit content without charset and language information.  If you still
don't understand that the UA must only generate compatible (i.e.
MIME 2047/2231) content because the existing infrastructure,
including injection agents and gateways, can't properly handle raw
untagged 8-bit content, then you may as well take a long walk on a
short pier.

Date: Mon, 10 Feb 2003 17:06:51 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> "D. J. Bernstein" <[email-masked]> writes:
> 
> 
>>Keith Moore writes:
>>
>>>The burden on those who want to see raw utf-8 in message headers is to
>>>convincingly demonstrate how to make it work well, without breaking the
>>>installed base.
> 
> 
>>It works just like other mandatory protocol extensions. Readers are
>>upgraded to support the extended format; once that's complete, writers
>>can safely use the format. Examples:
> 
> 
> It is even easier than that. For a charset change, they just have to wait
> until people who are interested in that charset have changed.

Provided the charset is properly tagged and the content properly encoded.

But that's besides the point. Dan Berstein's point about the process
is valid -- but is not what the current Usefor draft proposes. That
draft proposes writers generating an "extended" (a.k.a. illegal)
format *before* all of the installed base of readers (gateways, etc.)
are prepared for it.

> Anyway, here is a wicked thought.
<hare-brained scheme which is non-compliant with 822/2822 and MIME snipped>

Date: Mon, 10 Feb 2003 15:35:24 -0500
From: Keith Moore <[email-masked]>
To: "David Barr" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> The fact that there are almost a dozen different (and in many cases
> horribly broken) encoding formats that are required now to implement
> an email client/web browser is not a real problem?  

of course it's a real problem.  but adding more encoding formats doesn't
reduce that problem.  nor does handwaving about the additional
complexity involved in doing so.

> The fact that messages floating around often have to go through
> transformation after transformation, encoding after encoding, just to
> get from one place to another, and that*any* bugs or gaps with
> encoding along the way result in real breakage? 

uh, no they don't.  and if they are implemented that way, the
implementations are broken.

> The fact that because of typical desgin-by-committee and
> generational effects we now have a series of protocols layered on top
> of each other that are horribly complex, vague, and broken?  

and exactly how is one person (who hasn't bothered to look into the
details, and who couldn't get consensus on a proposal to save his life),
specifying a different and incompatible way to do the same thing 
going to help this situation?

> And the fact that the vast majority of these problems would be solved
> simply by having the balls to once and for all say "we shalt
> transition all this shit to UTF-8"?

nope, that's bullshit.  or at best, wishful thinking.

Keith

Date: Mon, 10 Feb 2003 14:57:26 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
CC: [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

David Barr wrote:
[Keith Moore wrote]
>>And if you had a shred of competency as an engineer or 
>>protocol designer you'd be working on solutions to the very 
>>real problems rather than trying to pretend that they don't exist.
> 
> 
> The fact that there are almost a dozen different (and in many cases horribly
> broken) encoding formats that are required now to implement an email
> client/web browser is not a real problem?

The current Usefor draft requires UAs to support raw utf-8, which is an
encoding that is currently not required to be supported by text
message UAs.  That's not "working on solutions", that's *compounding*
the real problem.

 > The fact that messages floating
> around often have to go through transformation after transformation,
> encoding after encoding, just to get from one place to another, and that
> *any* bugs or gaps with encoding along the way result in real breakage?
> That is not a real problem?

The current Usefor draft would require gateways to perform
transformations which are not now required and which the
existing installed base of gateways does not now perform. That's
not "working on solutions", that's *compounding* the real problem.

 > The fact that because of typical
> desgin-by-committee and generational effects we now have a series of
> protocols layered on top of each other that are horribly complex, vague, and
> broken?  And the fact that the vast majority of these problems would be
> solved simply by having the balls to once and for all say "we shalt
> transition all this shit to UTF-8"?

Because there are old messages, all of the existing methods need to
continue to be supported indefinitely so that those old messages can
still be read.  A transition requires backward compatibility so that
the infrastructure doesn't suddenly break (as that would not constitute
a transition), and it requires a feasible plan.  The Usefor draft
breaks backward compatibility and provides no feasible transition plan.
That's not "working on solutions", that's *compounding* the real problem.

> Rather than actually solve problems, the IETF would rather add more duct
> tape to an already structurally unsound system.

Text messages (822/2822 + MIME) as used by email and by Usenet under the
current standard (RFC 1036) works.  Most of the core issues requiring urgent
attention that the Usefor WG was formed to address are already addressed
by MIME; all that would have been required is a relatively minor update
to RFC 1036.  822/2822 + MIME is structurally sound; what the Usefor draft
proposes would result in an unsound system due to the dead weight of the
Usefor draft "duct tape" (adding *additional* encodings in UAs and requiring
transformations in gateways where currently none are required).

> This disturbs me.

If you genuinely believe that the Usefor draft solves problems rather
than compounding them, you are indeed seriously disturbed.

From: "David Barr" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: RE: Upgrading to UTF-8
Date: Mon, 10 Feb 2003 12:54:11 -0600
Importance: Normal
Sender: [email-masked]

> And if you had a shred of competency as an engineer or 
> protocol designer you'd be working on solutions to the very 
> real problems rather than trying to pretend that they don't exist.

The fact that there are almost a dozen different (and in many cases horribly
broken) encoding formats that are required now to implement an email
client/web browser is not a real problem?  The fact that messages floating
around often have to go through transformation after transformation,
encoding after encoding, just to get from one place to another, and that
*any* bugs or gaps with encoding along the way result in real breakage?
That is not a real problem?  The fact that because of typical
desgin-by-committee and generational effects we now have a series of
protocols layered on top of each other that are horribly complex, vague, and
broken?  And the fact that the vast majority of these problems would be
solved simply by having the balls to once and for all say "we shalt
transition all this shit to UTF-8"?

Rather than actually solve problems, the IETF would rather add more duct
tape to an already structurally unsound system.  This disturbs me.

--Dave

Date: Mon, 10 Feb 2003 12:41:42 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: ietf-822 Non-sub: Re (fwd)
Sender: [email-masked]

> >However I strongly disagree about the use of process.   IESG should
> >not accept WG output that doesn't meet the quality required for
> >standardization (which includes breaking things).
> 
> Yes, but the proper way is for the WG to prepare its proposals and
> then to discuss them with the IESG, usually via the Area Director in
> the first instance.

the WG needs to make a good faith effort to try to make its
proposals conform to RFC 2026.  a WG that fails to do this should be
summarily disbanded.

this is not specific to usefor. I'm just sick of groups insisting that
they have a right to publish whatever brain-damage the isolated,
narrowly foucsed WG thinks it has consensus on.

To: [email-masked]
Xref: clerew local.mime:2381 local.usefor:15703
Newsgroups: local.usefor,local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: ietf-822 Non-sub: Re (fwd)
Date: Mon, 10 Feb 2003 11:07:46 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>However I strongly disagree about the use of process.   IESG should not
>accept WG output that doesn't meet the quality required for
>standardization (which includes breaking things).

Yes, but the proper way is for the WG to prepare its proposals and then to
discuss them with the IESG, usually via the Area Director in the first
instance.

That is now being done. Our chairman has made preliminary contact with the
AD, and we are now preparing a more detailed submission.

>The burden on those who want to see raw utf-8 in message headers is to
>convincingly demonstrate how to make it work well, without breaking the
>installed base.

Yes, that is what we intend to demonstrate.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 10 Feb 2003 09:15:15 +0000
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: pronunciation of MIME
Sender: [email-masked]

At 21:54 09/02/2003 -0500, Keith Moore wrote:

> > In that case, I suspect that in RFC 2047 section 3
> >
> >     the character set names allowed in an MIME "charset" parameter of a
> >
> > should be
> >
> >     the character set names allowed in a MIME "charset" parameter of a
>
>egad.  That means we'll have to revise the document and reset to Proposed 
>Standard.

Missing smiley I presume...

:-)

Paul				VPOP3 - Internet Email Server/Gateway
[email-masked]			http://www.pscs.co.uk/

Date: Sun, 9 Feb 2003 21:54:33 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: pronunciation of MIME
Sender: [email-masked]

> In that case, I suspect that in RFC 2047 section 3
> 
>     the character set names allowed in an MIME "charset" parameter of a
> 
> should be
> 
>     the character set names allowed in a MIME "charset" parameter of a
> 

egad.  That means we'll have to revise the document and reset to Proposed Standard.

Keith

Date: Sun, 09 Feb 2003 21:51:27 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked], [email-masked]
Subject: Re: pronunciation of MIME
Sender: [email-masked]

Keith Moore wrote:
>>>What is the official pronunciation of MIME
>>>a) as the English word "mime"
>>>b) as spelled out, i.e. emm eye emm eee
>>>c) other?
>>
>>Nathaniel thought it up and used (a) from the outset. And so have I. I've
>>heard it spelled out a couple of times, but only a couple.
> 
> 
> The MIME acronym was specifically chosen to be pronouncable.  
> 
> Keith

In that case, I suspect that in RFC 2047 section 3

    the character set names allowed in an MIME "charset" parameter of a

should be

    the character set names allowed in a MIME "charset" parameter of a

Date: Sun, 9 Feb 2003 19:25:00 -0500
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: pronunciation of MIME
Sender: [email-masked]

> > What is the official pronunciation of MIME
> > a) as the English word "mime"
> > b) as spelled out, i.e. emm eye emm eee
> > c) other?
> 
> Nathaniel thought it up and used (a) from the outset. And so have I. I've
> heard it spelled out a couple of times, but only a couple.

The MIME acronym was specifically chosen to be pronouncable.  

Keith

Date: Sun, 09 Feb 2003 16:05:48 -0800 (PST)
From: [email-masked]
Subject: Re: pronunciation of MIME
To: Bruce Lilly <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> What is the official pronunciation of MIME
> a) as the English word "mime"
> b) as spelled out, i.e. emm eye emm eee
> c) other?

Nathaniel thought it up and used (a) from the outset. And so have I. I've heard
it spelled out a couple of times, but only a couple.

				Ned

Date: Sun, 09 Feb 2003 16:53:42 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: pronunciation of MIME
Sender: [email-masked]

What is the official pronunciation of MIME
a) as the English word "mime"
b) as spelled out, i.e. emm eye emm eee
c) other?

Date: Sun, 9 Feb 2003 01:33:42 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> Do you write any software, Keith? Does your software have problems with
> UTF-8? Are you having trouble figuring out how to fix it? If so, tell us
> the details, and we'll explain the solution to you.

Apparently, Dan, you can't explain anything - which is why you so often
resort to insult, ridicule, and ad hominem attacks.

And if you had a shred of competency as an engineer or protocol designer you'd
be working on solutions to the very real problems rather than trying to
pretend that they don't exist.

Date: 8 Feb 2003 23:49:33 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

Keith Moore writes:
> I said 'convincingly demonstrate'

Were you ever an EBCDIC user, Keith? Did you demand that the ASCII
proponents ``convincingly demonstrate how to make ASCII work well''?

Do you have trouble understanding how ASCII text today is successfully
copied from place to place and eventually displayed on your screen?

UTF-8 will work the same way as ASCII. Yes, the fonts take more space;
yes, some crude programming techniques (such as counting bytes to align
text in a fixed-width font) have to be abandoned; but all the details
have been clear for years. We already have many working UTF-8 programs.

Do you write any software, Keith? Does your software have problems with
UTF-8? Are you having trouble figuring out how to fix it? If so, tell us
the details, and we'll explain the solution to you.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sat, 8 Feb 2003 01:34:21 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Upgrading to UTF-8
Sender: [email-masked]

> Keith Moore writes:
> > The burden on those who want to see raw utf-8 in message headers is to
> > convincingly demonstrate how to make it work well, without breaking the
> > installed base.
> 
> It works just like other mandatory protocol extensions. 

I said 'convincingly demonstrate' not waving of hands and calling names.
Those are the trademark of incompetence.

Keith

Date: 8 Feb 2003 02:50:28 -0000
Mail-Followup-To: [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked]
Subject: Upgrading to UTF-8
Sender: [email-masked]

Keith Moore writes:
> The burden on those who want to see raw utf-8 in message headers is to
> convincingly demonstrate how to make it work well, without breaking the
> installed base.

It works just like other mandatory protocol extensions. Readers are
upgraded to support the extended format; once that's complete, writers
can safely use the format. Examples:

   * SMTP clients looking at DNS records were upgraded to understand MX
     records. Once that upgrade was complete, SMTP server administrators
     were able to safely use MX records.

   * The extension under discussion: Message handlers that have trouble
     with UTF-8 will be fixed. Once that's complete, message generators
     will be able to safely use UTF-8.

   * A different extension aimed at the same goal: Message handlers
     could be taught some archaic 7-bit GoofyCode format, with certain
     complicated rules for figuring out which parts of a message header
     are subject to GoofyCode processing. If that were ever completed,
     message writers would be able to safely use GoofyCode.

If you're complaining about the possibility that someone uses a format
before it is universally supported---perhaps causing failures---then
you're a hypocrite. Your own RFC 1342, a predecessor to GoofyCode, has
caused tremendous damage in exactly this way for the past ten years,
yet you continue to support GoofyCode-style proposals such as IDNA.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: 08 Feb 2003 00:08:00 +0100
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Charles Lindsey <[email-masked]> schrieb/wrote:
> And it is actually hard work - you need an arbitrary lookahead
> faciltiy, or full backtracking capability.

Only if you don't manage to just take the last two <'>s as delimiters.

man 3 strrchr

Claus
-- 
------------------------ http://www.faerber.muc.de/ ------------------------
OpenPGP: DSS 1024/639680F0 E7A8 AADB 6C8A 2450 67EA AF68 48A5 0E63 6396 80F0

Date: Fri, 7 Feb 2003 16:25:34 -0500
From: Keith Moore <[email-masked]>
To: Benjamin Franz <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: ietf-822 Non-sub: Re (fwd)
Sender: [email-masked]

thanks for resending this.

However I strongly disagree about the use of process.   IESG should not
accept WG output that doesn't meet the quality required for
standardization (which includes breaking things). 

Furthermore, rechartering of WGs is fully and appropriately within
IESG's power -- particularly when such WGs produce poor-quality output
and/or don't meet their deadlines for deliverables.

I wish they'd push back on such WGs a tad more often.

The burden on those who want to see raw utf-8 in message headers is to
convincingly demonstrate how to make it work well, without breaking the
installed base.

Keith

On Fri, 7 Feb 2003 11:58:50 -0800 (PST)
Benjamin Franz <[email-masked]> wrote:

> 
> This is being resent for the IETF-822 wg's benefit because the
> original was only 'half-delivered' due to that list blocking initial
> mails by non-subscribers (didn't usefor get chewed out by the IETF for
> doing that a couple of years ago?). My apologies to those on usefor
> who have already seen it
> 
> ---------- Forwarded message ----------
> Date: Thu, 6 Feb 2003 12:06:24 -0800 (PST)
> >From: Benjamin Franz <[email-masked]>
> To: [email-masked]
> cc: [email-masked], <[email-masked]>
> Subject: Re: FW: I-D ACTION:draft-kohn-news-article-00.txt
> In-Reply-To:
> <[email-masked]>
> Message-ID:
> <[email-masked]>
> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII
> 
> On Tue, 4 Feb 2003, Dan Kohn wrote:
> 
> >  This will require 7-bit encoding of i18n characters, likely using
> >  RFC 2047/2231 and/or nameprep+punycode (as with IDNA).  The general
> >  approach will be the one laid out in
> >  <http://www.ietf.org/internet-drafts/draft-kohn-news-article-00.tx
> >  t>.
> 
> Stop.
> 
> Really.
> 
> It is simply an attempt to hijack a working group via manipulation of
> IETF procedure. It would set a precedent that *everyone* in the IETF
> would live to regret.
> 
> "Don't like what someone's working group is producing? Recharter them
> to force the decisions you want in the charter! Fun and games for the
> whole family!"
> 
> --
> Benjamin Franz
> 
> "Hey! I have an idea! Let's recharter IETF-822 to require binary
> transparent operation!"
> 

Date: Fri, 7 Feb 2003 16:20:39 -0500
From: Keith Moore <[email-masked]>
To: Graham Klyne <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Problems with all lowercase header field names?
Sender: [email-masked]

> Is anyone aware if using all lowercase header field names in email
> messages is ever known to cause any problems?

this is one problem that I've never observed, and I've written programs
that generated all lower-case field names on many occasions.  (IIRC some
MTAs will "canonicalize" the casing, but this didn't bother me).

for a few field names there's no obviously correct casing anyway
(is it "Content-type" or "Content-Type"?)
so my guess is that most UAs that couldn't deal with variant casing have
found these bugs pretty quickly.

Keith

p.s. however I do know of at least one UA (sylpheed) that can't deal
with variant casing on month names in dates.  it's just lazy coding.

Date: Fri, 07 Feb 2003 13:06:31 -0800 (PST)
From: [email-masked]
Subject: Re: Problems with all lowercase header field names?
To: Graham Klyne <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Is anyone aware if using all lowercase header field names in email messages
> is ever known to cause any problems?

I don't know of a case where all lowercase fields was the proximate cause of a
problem, but I've seen cases where specific capitalization had to be used for
something to recognize a header field. I've also seen cases where fields had to
be in a specific order to work.

I've also seen problems with folding, especially when encoded words
were involved.

Of course all of this egregiously violates the relevant standards, and as much
pressure as possible needs to be brought to bear to fix the broken software
rather than attempt to work around it somewhere else.

				Ned

Date: Fri, 07 Feb 2003 14:38:15 -0600
From: "Eric A. Hall" <[email-masked]>
To: Graham Klyne <[email-masked]>
CC: [email-masked]
Subject: Re: Problems with all lowercase header field names?
Sender: [email-masked]

on 2/7/2003 12:05 PM Graham Klyne wrote:
> Is anyone aware if using all lowercase header field names in email
> messages is ever known to cause any problems?
> 
> It shouldn't, but...

I've seen similar. There were a couple of versions of DNews that wouldn't
list lowercased netnews header field names in overview output.

In answer to your question, "no", although I wouldn't be surprised.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Fri, 07 Feb 2003 15:14:00 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: Problems with all lowercase header field names?
Sender: [email-masked]

never had a problem with them, nor with all uppercase.

	Tony Hansen
	[email-masked]

Graham Klyne wrote:
> 
> Is anyone aware if using all lowercase header field names in email 
> messages is ever known to cause any problems?
> It shouldn't, but...

Date: Fri, 7 Feb 2003 11:58:50 -0800 (PST)
From: Benjamin Franz <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: ietf-822 Non-sub: Re (fwd)
Sender: [email-masked]

This is being resent for the IETF-822 wg's benefit because the original
was only 'half-delivered' due to that list blocking initial mails by
non-subscribers (didn't usefor get chewed out by the IETF for doing that a
couple of years ago?). My apologies to those on usefor who have already
seen it

---------- Forwarded message ----------
Date: Thu, 6 Feb 2003 12:06:24 -0800 (PST)
>From: Benjamin Franz <[email-masked]>
To: [email-masked]
cc: [email-masked], <[email-masked]>
Subject: Re: FW: I-D ACTION:draft-kohn-news-article-00.txt
In-Reply-To: <[email-masked]>
Message-ID: <[email-masked]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Tue, 4 Feb 2003, Dan Kohn wrote:

>  This will require 7-bit encoding of i18n characters, likely using RFC
>  2047/2231 and/or nameprep+punycode (as with IDNA).  The general approach
>  will be the one laid out in
>  <http://www.ietf.org/internet-drafts/draft-kohn-news-article-00.txt>.

Stop.

Really.

It is simply an attempt to hijack a working group via manipulation of IETF
procedure. It would set a precedent that *everyone* in the IETF would live
to regret.

"Don't like what someone's working group is producing? Recharter them to
force the decisions you want in the charter! Fun and games for the whole
family!"

--
Benjamin Franz

"Hey! I have an idea! Let's recharter IETF-822 to require binary
transparent operation!"

Date: Fri, 07 Feb 2003 18:05:35 +0000
To: [email-masked]
From: Graham Klyne <[email-masked]>
Subject: Problems with all lowercase header field names?
Sender: [email-masked]

Is anyone aware if using all lowercase header field names in email messages 
is ever known to cause any problems?

It shouldn't, but...

#g

-------------------
Graham Klyne
<[email-masked]>

Date: Fri, 07 Feb 2003 11:21:55 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Charles Lindsey wrote:
> In <HM7/[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:
> 
> 
>>>None of the registered names has *any* single quotes or percent signs.
> 
> 
>>And do _you_ think that will change? Does anyone here?
> 
> 
> No, practically speaking it won't. But according to the RFCs it _could_.
> Therefore anybody who wants to write an anal-retentively correct parser
> for RFC 2231 parameters has to allow for that possibility. And it is
> actually hard work - you need an arbitrary lookahead faciltiy, or full
> backtracking capability.

Would you care to provide a proof to support your claim that arbitrary
lookahead or full backtracking are required?

I suppose we can expect that about the same time as you provide
citations for the email RFCs which you claimed encourage reordering
of header fields...

Date: Fri, 07 Feb 2003 11:16:02 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Charles Lindsey wrote:

> Yes, but a hypothetical charset could also contain 56 single quotes.

So what?  A C expression could contain 56 minus signs (some unary,
some binary) -- that doesn't matter.

Date: Thu, 6 Feb 2003 15:53:02 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: David Barr <[email-masked]>
cc: [email-masked], [email-masked]
Subject: RE: I-D ACTION:draft-kohn-news-article-00.txt
Sender: [email-masked]

On Thu, 6 Feb 2003, David Barr wrote:
> ...I think now most of us realize
> that Usenet is dying almost entirely because of its innate nature, not of
> any feature or improvement now on the table.

Some of us don't think it's dying at all.  Changing, yes, and becoming
less important than it was, but not dying. 

> (Aren't mailing lists dying at a rate at least as high as Usenet, too?)

"Why look, yet another technology dying out on an upward curve."  Sure
doesn't look that way to me.

> I also think your Milestones are almost an order of magnitude too
> aggressive.  I've never heard of any group spitting out a finished draft in
> a month, especially not one as complex and subjective as that for Usenet.

As I've said elsewhere, the hidden agenda here is clearly to just adopt
the Usefor draft, change the things that the proposers don't like, and do
a bit of straightforward editing to cover up the scene of the crime.

                                                          Henry Spencer
                                                       [email-masked]

From: "David Barr" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: RE: I-D ACTION:draft-kohn-news-article-00.txt
Date: Thu, 6 Feb 2003 13:47:14 -0600
Importance: Normal
Sender: [email-masked]

> Both demand for and use of newsgroups and mailing lists has 
> been growing here at Stanford, mailing lists more so than 
> newsgroups.  The attempts that have been made to convert 
> popular local newsgroups to web-based fora have failed miserably.

Having worked at large universities for a number of years, I understand the
unique environment that universities afford.  Usenet and mailing lists grew
up with universities, and I don't see that ever changing.  However the
Internet has become a far far bigger place now, and the proportion of Usenet
and mailing lists in the grand scheme of things is going down.  When I say
"dying" I don't really mean literally decreasing in size, merely not growing
nearly to the degree everything else is and becoming more or less stagnant
(in both size and mindshare).  Growth of the Big-8 newsgroups is essentially
flat.  I don't know what stats like are for article flows over the last 6
years.  Regionals are growing in some areas, shrinking in others.  It seems
that for what corporations used to turn to email or Usenet communication
internally, they are turning to proprietary or web-based technologies. 

> blog/LiveJournal strikes me more as a completely new 
> phenomenon, not a replacement for any other existing 
> communications medium.

I don't agree.  Now that blogs are around, they are being used in places
where Usenet or email would have been before.  That's a replacement.

> But if you want to see the future of Usenet technology in my 
> view of things, go to <http://www.gmane.org/>.

Fun stuff.

--Dave

To: <[email-masked]>, <[email-masked]>
Subject: Re: I-D ACTION:draft-kohn-news-article-00.txt
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 06 Feb 2003 11:19:31 -0800
Lines: 31
Sender: [email-masked]

David Barr <[email-masked]> writes:

> Six years ago we all thought Usenet would die unless we did something
> quickly to bring it up to modern standards.  I think now most of us
> realize that Usenet is dying almost entirely because of its innate
> nature, not of any feature or improvement now on the table.  The web
> offers much more control, flexibility, and interactivity, one which
> Usenet by it's nature can't.  Usenet is great, but it's not a blog and
> it shouldn't try to be.  (Aren't mailing lists dying at a rate at least
> as high as Usenet, too?)

Both demand for and use of newsgroups and mailing lists has been growing
here at Stanford, mailing lists more so than newsgroups.  The attempts
that have been made to convert popular local newsgroups to web-based fora
have failed miserably.

YMMV.

blog/LiveJournal strikes me more as a completely new phenomenon, not a
replacement for any other existing communications medium.  Both suffer
from the standard problem that /. also has, namely that maintaining any
sort of extended and thoughtful discussion is essentially impossible,
which means that both have their best niche in presentation of "news"
(whether of general interest or only about a particular person), with
possibly some scattered, mostly superficial comments.

But if you want to see the future of Usenet technology in my view of
things, go to <http://www.gmane.org/>.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 6 Feb 2003 10:23:02 -0800
To: (Many IETF lists)
From: Paul Hoffman / IMC <[email-masked]>
Subject: New draft on internationalizing email addresses
Sender: [email-masked]

Greetings. This message is Bcc'd to many IETF-related lists that are 
discussing internationalization, Internet mail and Usenet. With the 
imminent release of the RFCs for IDNA, Adam Costello and I have a 
proposal for allowing internationalized characters in email addresses 
(on both sides of the @). Instead of having it discussed on a bunch 
of different lists, there is a new list for discussing the draft and 
other ideas related to internationalizing email address.

If you are interested, please see <http://www.imc.org/ietf-imaa/> for 
information on the Internet Draft and the mailing list.

--Paul Hoffman, Director
--Internet Mail Consortium

From: "David Barr" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Subject: RE: I-D ACTION:draft-kohn-news-article-00.txt
Date: Thu, 6 Feb 2003 10:20:13 -0600
Importance: Normal
Sender: [email-masked]

The presumtion here is that USEFOR has gone astray because our charter was
too broad.  I think that argument, while true on paper, does not hold up to
closer scrutiny.

For approximately the last 2-3 years, we have been operating under a
substantially reduced scope.  In the first few years of the WG, virtually
everything was fair game, and it was plain nuts.  We literally talked
ourselves in circles four and five times over the course of 3 years.  At
that time a major factor was indeed that our charter was too broad.

But we haven't been working under that broad goal for 2-3 years now.  For
the last 2+ years, all we have been doing is going over the current draft,
chopping out things that we can't get consensus on, and revising things we
can get consensus on.  We've made real progress, just not progress that is
popular with some.

The real problems with the group have nothing to do with the charter or with
the draft, and proposing a new charter and a new draft will do little to
help, IMHO, at least not one that includes such narrow preconceptions such
as the requirement of 7-bit encoding.  The real problem is simple burnout
and loss of mandate.

Six years ago we all thought Usenet would die unless we did something
quickly to bring it up to modern standards.  I think now most of us realize
that Usenet is dying almost entirely because of its innate nature, not of
any feature or improvement now on the table.  The web offers much more
control, flexibility, and interactivity, one which Usenet by it's nature
can't.  Usenet is great, but it's not a blog and it shouldn't try to be.
(Aren't mailing lists dying at a rate at least as high as Usenet, too?)

I also think your Milestones are almost an order of magnitude too
aggressive.  I've never heard of any group spitting out a finished draft in
a month, especially not one as complex and subjective as that for Usenet.

Having said that, however, I believe I have said before that once we pound
out this current draft, we need to seriously sit back and decide where to go
next (i.e. recharter).

--Dave

To: [email-masked]
Xref: clerew local.mime:2367
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Date: Wed, 5 Feb 2003 11:31:28 GMT
Lines: 24
Sender: [email-masked]

In <HM7/[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>> None of the registered names has *any* single quotes or percent signs.

>And do _you_ think that will change? Does anyone here?

No, practically speaking it won't. But according to the RFCs it _could_.
Therefore anybody who wants to write an anal-retentively correct parser
for RFC 2231 parameters has to allow for that possibility. And it is
actually hard work - you need an arbitrary lookahead faciltiy, or full
backtracking capability. I doubt anyone will bother, but it would have
been nicer for the RFCs to have excluded those characters in the first
place.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 04 Feb 2003 13:11:48 -0500
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Arnt Gulbrandsen wrote:
> Bruce Lilly writes:

>> None of the registered names has *any* single quotes or percent signs.
> 
> 
> And do _you_ think that will change?

No.

From: Arnt Gulbrandsen <[email-masked]>
To: Bruce Lilly <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Cc: [email-masked]
Date: Tue, 4 Feb 2003 19:00:53 +0100
Sender: [email-masked]

Bruce Lilly writes:
> According to my understanding, the 40-character limit is for charset
> names used with RFC 1759 Printer MIBs and does not apply in general.

Seems wider: "These are the official names for character sets that may 
be used in the Internet and may be referred to in Internet 
documentation." And a few lines down: "The character set names may be 
up to 40 characters taken from the printable characters of US-ASCII.  
However, no distinction is made between use of upper and lower case 
letters."

> RFC 2978 doesn't specify a limit for mime-charset, but obviously
> anything longer than 68 chars won't be useful for a 2047 encoded-word.
> There are a couple of registered charsets with names a bit longer than
> 40 chars; the longest is 45 chars.

I suppose I'd avoid those, personally speaking. Inconsistencies are a PITA.

> None of the registered names has *any* single quotes or percent signs.

And do _you_ think that will change? Does anyone here?

--Arnt

Date: Tue, 04 Feb 2003 12:45:43 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Arnt Gulbrandsen wrote:
> 
> Charles Lindsey writes:
> 
>> Yes, but a hypothetical charset could also contain 56 single quotes.
> 
> 
> There's a long list of allowed charset names somewhere in Assigned 
> Numbers. It was on pages 101-118 of RFC 1700 (I'm behind the times, I 
> guess). I confidently expect that no charset name containing 56 single 
> quotes will be allowed onto that list, and not because of the 
> 40-character length limit.

RFC 1700 has been replaced by an on-line set of documents at
http://www.iana.org/numbers.html

The charsets are at
http://www.iana.org/assignments/character-sets

According to my understanding, the 40-character limit is for charset
names used with RFC 1759 Printer MIBs and does not apply in general.
RFC 2978 doesn't specify a limit for mime-charset, but obviously
anything longer than 68 chars won't be useful for a 2047 encoded-word.
There are a couple of registered charsets with names a bit longer than
40 chars; the longest is 45 chars.

None of the registered names has *any* single quotes or percent signs.

content-class: urn:content-classes:message
Importance: normal
Subject: FW: I-D ACTION:draft-kohn-news-article-00.txt
Date: Tue, 4 Feb 2003 08:39:36 -0800
From: "Dan Kohn" <[email-masked]>
To: <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

This I-D is intended to show an alternative way forward for usefor, if
the IESG were to recharter the working group.  I should be clear that
it's mainly an outline, and that if this direction were pursued, there
would be a lot of useful to work to pull out of Charles' draft and into
this one and a separate policy draft.

General discussions about the direction of usefor still presumably
belong on that list.  If you'd like to improve this I-D, please join the
group [email-masked] (by sending "subscribe" to
[email-masked]) or go to
<http://www.dankohn.com/mailman/listinfo/news_dankohn.com>.  There's no
reason for follow-ups to ietf-822, as they can join usefor or
[email-masked] if they're interested.

Here is a proposed recharter:

USENET news remains one of the largest sources of Internet traffic,
although its primary specification, RFC 1036, is long out-of-date.
Usefor was chartered 6 years ago, but the group is heading with tenuous
consensus toward an approach unlikely to be approved by the IESG.

This rechartered working group will update this news article
specification, using a normative reference to Section 3 of RFC 2822 (the
legal to generate email syntax) as the basis for the news message
standard going forward.  It will further define the applicability of
MIME (RFC 2045, 2046, and 2049).  It will also define the appropriate
news-specific headers from RFC 1036, and deprecate those no longer in
use.

The group will enable the i18n of user-visible header text in such a way
as to preserve backward compatibility with existing clients, servers,
and gateways for mail and news.  This will require 7-bit encoding of
i18n characters, likely using RFC 2047/2231 and/or nameprep+punycode (as
with IDNA).  The general approach will be the one laid out in
<http://www.ietf.org/internet-drafts/draft-kohn-news-article-00.txt>.

The group will also publish a standards-track policy document describing
responsibilities and recommendations for news agents, servers, and
gateways.  This document will also consider how clients, servers, and
gateways should deal with the existing unstable situation of 8-bit
newsgroup names in various unlabeled character sets, and specify a
transition plan for such newsgroups, if feasible.

An explicit goal of the group is to ensure cross-compatibility with mail
gateways.  The group's charter is to create a stable base document
defining the news message format, which can act as a foundation for new
innovations going forward.  The group will not standardize any
technology other than i18n that is not already in wide use, such as
strong verification of control messages, raw UTF-8 headers, cancel
locks, etc.  Work on such areas would take place in a separate working
group, if such a group were later chartered.

Goals and Milestones

Apr 2003
+ Produce a first Internet-Draft describing the news article format in a
form compatible with the charter

May 2003
+ Produce a first Internet-Draft describing policy for news agents,
servers, and gateways.

Jun 2003
+ Produce a second article Internet-Draft

September 2003
+ Produce a second policy Internet-Draft

November 2003
+ Submit article and policy documents to IESG for approval and close
working group.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, February 04, 2003 03:45
Subject: I-D ACTION:draft-kohn-news-article-00.txt

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

	Title		: News Article Format
	Author(s)	: D. Kohn
	Filename	: draft-kohn-news-article-00.txt
	Pages		: 13
	Date		: 2003-2-3
	
This document defines the format and procedures for interchange of
network news articles. It updates and obsoletes RFC 1036, in
particular adding support for internationalization of headers and
message bodies and multimedia support in message bodies.  It does
this in a manner designed to maximize backward compatibility with
mail and news servers, gateways, and user agents.
Network news articles resemble mail messages but are broadcast to
potentially-large audiences, using a flooding algorithm that
propagates one copy to each interested host (or group thereof),
typically stores only one copy per host, and does not require any
central administration or systematic registration of interested
users. Network news originated as the medium of communication for
Usenet, circa 1980. Since then Usenet has grown explosively, and many
Internet sites participate in it. In addition, the news technology is
now in widespread use for other purposes, on the Internet and
elsewhere.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-kohn-news-article-00.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the
message.

Internet-Drafts are also available by anonymous FTP. Login with the
username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-kohn-news-article-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-kohn-news-article-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail
readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

From: Arnt Gulbrandsen <[email-masked]>
To: Charles Lindsey <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Cc: [email-masked]
Date: Tue, 4 Feb 2003 13:55:24 +0100
Sender: [email-masked]

Charles Lindsey writes:
> Yes, but a hypothetical charset could also contain 56 single quotes.

There's a long list of allowed charset names somewhere in Assigned 
Numbers. It was on pages 101-118 of RFC 1700 (I'm behind the times, I 
guess). I confidently expect that no charset name containing 56 single 
quotes will be allowed onto that list, and not because of the 
40-character length limit.

Isn't it great when the ridiculous warts can be simplified out of existence?

--Arnt

To: [email-masked]
Xref: clerew local.mime:2362
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RFC 2231 and CFWS
Date: Tue, 4 Feb 2003 10:26:57 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles Lindsey wrote:

>> Thanks for the pointer to RFC 2978. I see that '%' is also allowed, but
>> not '*'. So writing a foolproof parser for extended-initial-value would be
>> tricky (you have to scan from R to L until you have identified the two
>> rightmost "'"s), but I doubt implementors will bother to go so far.

>That is not the only way to skin that particular cat; a grammar-based
>parser can easily handle extended parameters including a hypothetical
>charset name containing a single quote and/or percent sign without
>resorting to reverse scanning -- it's handled by the normal parser
>state machine stack.  It's not even particularly "tricky" -- one
>simply writes the appropriate grammar rules and it just works.

Yes, but a hypothetical charset could also contain 56 single quotes.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 4 Feb 2003 00:00:40 -0500
From: Keith Moore <[email-masked]>
To: [email-masked], [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

> Keith Moore wrote:
> 
> > Actually, no.  There was a difference in intent between the two.  1123 was
> > 
> > imposing new requirements on existing protocols, 1341 and 1342 were not
> > doing so.  MIME implementation was not mandatory (as a matter of standards
> > compliance) for existing user agents.
> 
> Yes, in theory 1123 is mandatory and MIME compliance is not. In theory,
> SMTP MTAs had to be compliant with RFC 821 as reaffirmed by 1123.  In
> practice most are not. The Internet SS isn't rounding up MTA authors
> and throwing them in concentration camps. Certainly a UA can be
> constructed which is not MIME-compliant (and indeed there are many of
> those). However, while there is no standards requirement for MIME
> support (and no enforcement even if there were a requirement), those
> UAs which do provide MIME support are used in vastly greater numbers
> than those that do not.  Market forces are infinitely greater than
> standards enforcement as far as the Internet is concerned.

The world changed a lot between the time that 1123 was written and the time
that MIME was written.  At the time 1123 was written the Internet was small
enough, and there was still enough vestage of the ARPAnet mentality, that
imposing a few new requirements on existing protocols was conceivable.  By the
time MIME was written it was obvious to us that trying to insist on
everybody's MUAs or MTAs being upgraded - no matter how trivially - as a
condition for making MIME work was simply untenable, and there was
considerable resistance from some parties to expecting English users to
upgrade their mail tools for the sake of the "few" (at that time) in the net
that didn't speak English.  For that matter, most of the changes in 1123 were
not likely to cause the degree of pain that MIME did.

Anyway, I hope the difference in intent is clear by now.

Date: Mon, 03 Feb 2003 21:00:52 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Keith Moore wrote:

> Actually, no.  There was a difference in intent between the two.  1123 was 
> imposing new requirements on existing protocols, 1341 and 1342 were not
> doing so.  MIME implementation was not mandatory (as a matter of standards 
> compliance) for existing user agents.

Yes, in theory 1123 is mandatory and MIME compliance is not. In theory,
SMTP MTAs had to be compliant with RFC 821 as reaffirmed by 1123.  In
practice most are not. The Internet SS isn't rounding up MTA authors
and throwing them in concentration camps. Certainly a UA can be
constructed which is not MIME-compliant (and indeed there are many of
those). However, while there is no standards requirement for MIME
support (and no enforcement even if there were a requirement), those
UAs which do provide MIME support are used in vastly greater numbers
than those that do not.  Market forces are infinitely greater than
standards enforcement as far as the Internet is concerned.

None of which detracts from the facts:
1. It has been possible to generate internationalized phrases, comments
    and unstructured field contents in any application of 822 text
    message format since RFC 1342 (whether or not a particular
    receiving UA is MIME compliant and properly displays the
    encoded-words is another matter).
2. MIME (RFC 2047/2231) methods are the only means of handling non-ASCII
    content in message header and MIME-part header fields in text
    messages which is compliant with RFC 822 (and by extension to
    applications, such as news, which use the 822 text message format).

Date: Mon, 3 Feb 2003 19:22:10 -0500
From: Keith Moore <[email-masked]>
To: [email-masked], [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

> Just as RFC 1123 section 5.2.14 amended the 822 syntax for date, RFC 1342
> amended the syntax for phrase and comment (and in the latter case with
> far less impact on parsers, since an encoded-word still matches atom (in
> word) in a phrase and ctext in a comment). 

Actually, no.  There was a difference in intent between the two.  1123 was 
imposing new requirements on existing protocols, 1341 and 1342 were not
doing so.  MIME implementation was not mandatory (as a matter of standards 
compliance) for existing user agents.

Keith

Date: Mon, 03 Feb 2003 19:11:43 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Henry Spencer <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Henry Spencer wrote:
> On Mon, 3 Feb 2003, Bruce Lilly wrote:
> 
>>>However, RFC 1036 refers specifically and explicitly to RFC 822, *not* to
>>>"RFC 822 or any update thereof".  RFC 2047, like other post-822 updates to
>>>mail standards, does not apply to news unless we make it so. 
>>
>>OK, guys, you heard Henry -- no more of that 4-digit year nonsense
>>in dates.
>>Yeah, right; everybody on Usenet is going to go back to 2-digit
>>years, just like it says in RFC 822.
> 
> 
> We were talking about what the documents say, not about what is actually
> being done on the net -- a gap that is particularly wide where non-ASCII
> character sets are concerned. 
> 
> Your claim, since you seem to have forgotten it, was that since RFC 1036
> references RFC 822, MIME (an extension and amendment to 822) automatically
> applies to news as well.  Whether it *should* apply to news, and whether
> it is being used *as if* it does, are separate (although related) topics. 

Apparently the satire was too subtle.  So here's the long version:

Just as RFC 1123 section 5.2.14 amended the 822 syntax for date, RFC 1342
amended the syntax for phrase and comment (and in the latter case with
far less impact on parsers, since an encoded-word still matches atom (in
word) in a phrase and ctext in a comment).  We didn't have to wait for
2822 to say that date-time in a text message could use 4-digit years; as
soon as RFC 1123 took effect that was the case.  For encoded-words, the
situation was simpler still; according to the *original* 822 syntax rules,
=?us-ascii*en-us?q?foo?= was always valid as a word in a phrase and in a
comment and in an unstructured field -- the 1342/1522/2047 amendments
simply clarified the interpretation of that text as an encoded-word in
those contexts.

And RFC 1036 does not *merely* _reference_ RFC 822, it is quite clear that
the text message format of 822 applies, even to the extent that where 1036
and 822 differ, 822 has priority and 1036 is deemed to be in error.  See
the exact wording in RFC 1036 section 2, first paragraph.

Much of MIME is orthogonal to RFC 822, as explicitly stated in the MIME RFCs.
But it is quite clear that 1342/1522/2047 encoded-words definitely apply to
RFC 822 text message format, and there is no exclusion for news articles,
voice messaging, or any other specific *application* of that text message
format.

Date: Mon, 03 Feb 2003 15:07:52 -0800 (PST)
From: [email-masked]
Subject: Re: Transformation of Non-ASCII headers
To: Bruce Lilly <[email-masked]>
Cc: Henry Spencer <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> Henry Spencer wrote:

> > However, RFC 1036 refers specifically and explicitly to RFC 822, *not* to
> > "RFC 822 or any update thereof".  RFC 2047, like other post-822 updates to
> > mail standards, does not apply to news unless we make it so.

> OK, guys, you heard Henry -- no more of that 4-digit year nonsense
> in dates.

> Yeah, right; everybody on Usenet is going to go back to 2-digit
> years, just like it says in RFC 822.

Er, exactly.

I'm sorry, but it has been long been understood that references can deemed to
track to subsequent revisions of a specification without explicit revision to
point to the newer version. This is the sort of situation where common sense
has to prevail. Trying to find and fix every RFC each time a document it refers
to is itselft revised would be a huge waste of time.

As such, there is a lot less to be read into the specific reference to RFC 822
in RFC 1036 than people seem to think.

				Ned

Date: Mon, 3 Feb 2003 17:55:16 -0500
From: Keith Moore <[email-masked]>
To: [email-masked], [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

> > The only reason that MIME applies to HTTP is that RFC 2616 explicitly
> > declares it to be so (modulo some small changes). So the MIME standards do
> > not currently apply to Netnews (but our draft will fix that).
> 
> Incorrect; RFC 1036 specifically mentions that news articles use the
> text message format defined in RFC 822, and RFC 2047 explicitly revises
> the RFC 822 ABNF for phrase and for comment to permit encoded-words
> (see RFC 2047 section 5 which clearly and unambiguously spells this out).
> So in the specific case of encoded-words (which is the topic under
> discussion), MIME has applied to news for at least a decade (since
> RFC 1342).

RFC 1036's claims notwithstanding, news and email use different message
formats, with different header fields and different interpretations and syntax
for some of the fields that share a common name.  So it's not clear that
2047's revision of the 822 syntax for the purpose of emailed MIME messages
automatically applies to use of a syntax that was derived from 822 for the use
in netnews.  Second, it's doubtful that changing the format of netnews would
have been consistent with the 822ext working group's charter.   Third,
applicability of MIME to Usenet is not something that has been considered when
advancing MIME on the standards track. If we were to consider the rate of
adoption of MIME within the Usenet community, MIME would probably best be
classified as "experimental" for that purpose.  Finally it would set a bad
precedent if every revision of 822 for email purposes were deemed to change
every use of an 822-like syntax (for instance, in HTTP or SIP).  It's good
that we borrow proven designs for use in new protocols, but each protocol has
its own needs and its own user community and different protocols need to be
allowed to evolve separately.   

That's not to say that I think using MIME (and 2047) with Usenet is a bad idea
- there is at least some commonality between the design constraints of MIME
and those of Usenet messages, there are many user agents that deal with
both, and there are several gateways between the two worlds.  So it is useful if
there's commonality between the message formats.  It just seems a stretch to
declare by fiat that MIME automatically applies to usenet.

Keith

Date: Mon, 3 Feb 2003 17:47:38 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

On Mon, 3 Feb 2003, Bruce Lilly wrote:
> > However, RFC 1036 refers specifically and explicitly to RFC 822, *not* to
> > "RFC 822 or any update thereof".  RFC 2047, like other post-822 updates to
> > mail standards, does not apply to news unless we make it so. 
> 
> OK, guys, you heard Henry -- no more of that 4-digit year nonsense
> in dates.
> Yeah, right; everybody on Usenet is going to go back to 2-digit
> years, just like it says in RFC 822.

We were talking about what the documents say, not about what is actually
being done on the net -- a gap that is particularly wide where non-ASCII
character sets are concerned. 

Your claim, since you seem to have forgotten it, was that since RFC 1036
references RFC 822, MIME (an extension and amendment to 822) automatically
applies to news as well.  Whether it *should* apply to news, and whether
it is being used *as if* it does, are separate (although related) topics. 

Since you have changed the subject, I assume you have conceded your error. 

                                                          Henry Spencer
                                                       [email-masked]

Date: Mon, 03 Feb 2003 17:16:50 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Henry Spencer <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Henry Spencer wrote:

> However, RFC 1036 refers specifically and explicitly to RFC 822, *not* to
> "RFC 822 or any update thereof".  RFC 2047, like other post-822 updates to
> mail standards, does not apply to news unless we make it so. 

OK, guys, you heard Henry -- no more of that 4-digit year nonsense
in dates.

Yeah, right; everybody on Usenet is going to go back to 2-digit
years, just like it says in RFC 822.

Date: Mon, 3 Feb 2003 16:45:09 -0500 (EST)
From: Henry Spencer <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

On Mon, 3 Feb 2003, Bruce Lilly wrote:
> > ...So the MIME standards do
> > not currently apply to Netnews (but our draft will fix that).
> 
> Incorrect; RFC 1036 specifically mentions that news articles use the
> text message format defined in RFC 822, and RFC 2047 explicitly revises
> the RFC 822 ABNF...

However, RFC 1036 refers specifically and explicitly to RFC 822, *not* to
"RFC 822 or any update thereof".  RFC 2047, like other post-822 updates to
mail standards, does not apply to news unless we make it so. 

                                                          Henry Spencer
                                                       [email-masked]

Date: Mon, 03 Feb 2003 16:29:09 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Transformation of Non-ASCII headers
Sender: [email-masked]

Charles Lindsey wrote:
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> 
>>Charles Lindsey wrote:
> 
> 
>>>In <[email-masked]> Bruce Lilly <[email-masked]> writes:
>>> 
>>>
>>>
>>>>1. The draft is incompatible with the MIME standards ...
>>>
>>>The existing MIME standards relate only to Email.
>>>
>>
>>Untrue; MIME has many uses, including HTTP and voice messaging.
> 
> 
> The only reason that MIME applies to HTTP is that RFC 2616 explicitly
> declares it to be so (modulo some small changes). So the MIME standards do
> not currently apply to Netnews (but our draft will fix that).

Incorrect; RFC 1036 specifically mentions that news articles use the
text message format defined in RFC 822, and RFC 2047 explicitly revises
the RFC 822 ABNF for phrase and for comment to permit encoded-words
(see RFC 2047 section 5 which clearly and unambiguously spells this out).
So in the specific case of encoded-words (which is the topic under
discussion), MIME has applied to news for at least a decade (since
RFC 1342).

>>Neither utf-8 nor any other charset provides language tagging.
>>Unicode 3.1 and 3.2 documents provide for language tagging specifically
>>for ACAP; news articls != ACAP.  No known news or mail user agents
>>generate or recognize the Unicode 3.1/3.2 language tagging.
> 
> 
> What is ACAP? I see no mention of it in the Unicode documents. Anyway,
> there seems to be some confusion in this area, so I am writing to Martin
> Duerst for clarification.

This was all thrashed out many months ago. Look in the archive.
Or go to the Unicode consortium web site and search for "ACAP"
(which yields precisely one match).  Look up ACAP in the rfc-index
(FYI, it's related to MIBs and SNMP).

>>No sane, rational person who has given a moment's thought would propose
>>such nonsense. The fields in a message header may contain multiple 
>>languages.
>>Indeed, a single field may contain multipple  languages, e.g.:
>> Keywords: =?us-ascii*en?q?boat?=, =?iso-8859-1*de?boot?=
> 
> 
> Ah! You mean I might write a message with
>     Subject: I left my boots im boot
> which would surely confuse any agent with a text-to-audio facility.
> 
> So how would you write that header (in current email)? And do you suppose
> any user will ever go to the trouble of writing language tags in headers
> even if we give him the facility to do so?

With the appropriate language tags, of course, that's how (it
depends on where the language changes, which is not entirely clear
from your example).  Users may do so where they recognize the
potential for confusion and where their user agent of choice
provides a convenient facility for doing so, or where the user
goes to the trouble to provide tags for the benefits of his
readers in the absence of UA support.  At least it is relatively
simple for a user to do the latter; he has only to use us-ascii
characters -- without consulting a table of Unicode tags and/or
utf-8 encoding tables, how would you present the raw
utf-8/Unicode 3.1 equivalent of the Keywords example?

>>>>3. There is no backwards compatibility ...
>>>
>>What you describe is not backwards compatibility, it is forwards 
>>compatibility
>>(existing content vs. some postulated future software). That is 
>>sophistry.  You
>>know what backwards compatibility really means; it has been explained to you
>>multiple times by many people (including those in the IESG who will have to
>>review and vote on the draft if it ever is submitted).  Presenting 
>>software which
>>has ben designed in good faith in compliance with RFC 1036, MIME, etc. with
>>data which is illegal (8-bit content in message or MIME-part header 
>>fields) is
>>not backwards compatible.
> 
> 
> You will not win any arguments by changing the meaning of the English
> language to prove your point.
> 
> If there is a feature in some existing protocol which becomes illegal in
> the new version, then that is a breech of "backwards compatibility".

Sorry, no -- you have that wrong.

> If there is a feature in some new protocol that will not work with
> implementations written for the old protocol, then that is a breech of
> "forwards compatibility".

Nope, wrong again.

> It seems you want to define these terms the other way around. Does anyone
> else here agree with your view? The weight of evidence is surely against
> you.

Don't take my word for it. Dave Crocker has clearly and unambiguously
explained to you what backwards compatibility does and does not
mean to the IETF, which is what counts.  And inflicting illegal
(per existing standard) content on existing implementations (e.g.
of gateways), as the Usefor draft would do, is most definitely
contrary to maintaining backwards compatibility.

A moment's thought should make it clear that "particular attention
to backwards compatibility" in the Usefor WG charter refers to the
same thing as the IETF usage -- one would hardly require paying
particular attention to the reverse, i.e. old articles being fed
into new implementations; after about 1 day such old articles
would be rejected as stale, and would hardly warrant "particular
attention".

> Just to check, I grepped through my small collection of online RFCs, and
> found usage of "backwards compatibility" in my sense in RFCs 1945, 2060 and
> 2156. I found only one (RFC 2646) that seemed to use it your way, so the
> majority is against you. You are. however, welcome to produce further
> examples.

[Apologies to everybody other than Charles for repeating the content
below rather than simply providing references to the online archives,
but in the past when presented with references Charles has whined about
having to fire up a browser, etc.]

Reread what Dave wrote in <[email-masked]>:
"
I have included segments from your posting that pertain to the question of
impact on installed base.  This issue is fundamental.

To add to Ned's correction, news-> email gatewaying has been around for at
least 20 years, as far as I can recall. Organizations have often wanted to
plug mailing lists into newsgroups. To permit the newsgroup readers to
participate in the mailing list, a news->email gateway is required. In other
environments, the 2-way gatewaying is simply part of a model that lets users
decide how they want to receive and process their group discussion messages.

With respect to the appearance of new capabilities (such as new data formats
and encodings) in the installed base, your use of the term "gobbledegook"
needs to be made more precise. If it means that that a legacy system
receives the data and the data are legal but meaningless -- MIME base64 is
example of this approach -- then it is fine. It permits incremental adoption
without breaking existing systems. The downside for existing systems is that
they do not get the benefit of the new feature, but everything else
continues to work fine.

If it means that the legacy system receives data that are illegal, with
respect to the existing service, then this is not at all fine.
Interoperability requires conforming to a standard, rather than conforming
to it whenever it is convenient. Injecting illegal data is not conforming.

The difference between an upgrade that protects the installed base, versus
an upgrade that requires creating a new installed base is absolutely
fundamental.  Each can be appropriate but explicitly choosing between them
is essential.

The importance of preserving the installed base -- by way of doing an
incremental upgrade rather than a parallel replacement -- is frequently
missed. However the market power of an installed base is massive. Engineers
and vendors ignore it at their peril.

The very strong IETF philosophy has been to try to preserve the installed
base, through optional, incremental upgrade.  It is part of the reason that the
architectural history has been to keep the core infrastructure as simple as
possible.

Infrastructure changes rarely permit incremental upgrade. Hence, a change to
the infrastructure often must be done to the *entire* infrastructure before
there can be benefit to any users. Think about the effort -- and delay --
this requires for a large infrastructure, like the IP Internet, or Usenet. A
delay that is effectively infinite becomes probable when the infrastructure
is very large.

It is certainly true that this approach creates solutions that are ugly. The
encoding aspect of MIME is the epitome of ugliness, in my opinion. And, yes,
incremental ugliness can create an aggregate messiness that eventually
requires complete replacement.  And it is important to look for that
critical mass of cumbersomeness.

But the reason for choosing the MIME ugliness was to permit incremental
upgrade while preserving the installed base -- and to make the upgrade be
strictly in the leaves, with no changes at all to the email infrastructure.

There were previous attempts to introduce multi-media email to the Internet,
through operation of an independent, parallel service that had a multi-media
core.  The installed base ignored those efforts.
"

and in <[email-masked]>
"
The concern is not whether there is an existence proof for the capability,
or even whether there is already a strong support base. The concern is what
happens to the platforms that do not yet support it.
"

and in  <[email-masked]>
"
you seem to miss the point that the IETF does not make new standards that
break old standards.  parallel effort is fine, but you do not lay down a new
thing that violates an old one.

your use of "unlikely" basically ends this discussion.

standards work uses statistics to make guesses about needs, but it does not
use it to decide whether a technology is valid.  either it breaks things or
it doesn't.  "unlikely" is not in the vocabulary.  (it breaks the
discussion.)
"

and in <[email-masked]>
"
Except in any of the places in the installed base that are not expecting
UTF-8.

Oh.  That's right.  That means every place using current standards.

oops.
"

and in <[email-masked]>
"
1.  netascii has a 30-year installed base.

2.  compatibility with netascii is an explicit goal when working with email.

3.  the logic behind that goal is applicable to any other application having
a similar installed base history.

4.  paying attention to the installed base requires worrying about
compatibility with what has been established practise, not what "might" work
or what is "frequently" available.

5.  encodings are different from representations.  representation is the
real, "native" data.  encoding is a way of transporting that data over a
constrained environment.

6.  the fact that one 7-bit encoding scheme was not successful does not
condemn all 7-bit encoding schemes.

7.  breaking standards is not measured by whether someone's code core-dumps.
it is measured by whether it violates the specification.  sending 8-bit data
in a 7-bit environment breaks the specification.  sending valid strings of 7-bit
data that might need further interpretation (to obtain the semantics) does
not.
"

and in <[email-masked]>
"
The issue is not whether there is an existence proof for the practise you
want to standardize.  The issue is, instead, the impact of a new standard on
the installed base that uses the old standard.
"

Others have also given similar advice, and I could have quoted Ned Freed
or others as well.  After all of that, do you really fail to comprehend
what backward compatibility means in terms of protecting the installed
base, or are we just witnessing more of your sophistry?

>>>>4. Compatibility (as exists under RFC 1036) would be broken with 
>>>>interoperating
>>>>  protocols including  IMAP and SMTP.
> 
> 
> There is no requirement to transport Netnews articles by SMTP. In the case
> of moderation, they are required first to be converted to Email messages
> (and our draft tells you how). The Email messages may then be transported
> by SMTP.

Problem is, the draft method isn't part of the existing installed
base of software.  Under the current standard, news articles
*are* email messages (with some minor differences), so a news
article *can* be (and is) mailed to a moderator, almost always
via SMTP (and possibly via IMAP as well).

>>No, there is no probnlm with IMAP, which works quite well for maill and 
>>news,
>>and the IMAP RFCs are fully compliant with RFC 1036 and other relevant
>>RFCs. ...
> 
> 
> IMAP works with the existing news protocols. If, as a result of our draft,
> the news protocols change, then a matching change in the IMAP protocol
> would be in order.

That would again break backwards compatibility, and so would almost
certainly never be approved as such. Now, if you want to propose such
an incompatible format as part of a separate, parallel implementation,
that's another matter.  That may well suffer the same fate as the
incompatible multimedia mail implementations mentioned by Dave Crocker.
Perhaps not. But existing protocols, software installations, etc. could
either continue to support RFC 1036 as is or adopt the hypothetical
new format or both, where practical.

>>>>5. Provision of two alternative mechanisms ... is a dubious practice.
>>>
>>>But necessary in order to pave the way for future progress. The
>>>alternative is for all the internet protocols to remain stagnant for the
>>>next 1000 years.
>>>
>>
>>The "1000 years" rmark is the height of absurdity.
> 
> 
> If you insist that forwards incompatibility can NEVER be allowed, then you
> should be aware that "1000 years" is considerably less than "NEVER". I do
> not see the absurdity of that.

Your failure to see the absurdity does not make it vanish. We
are talking about backwards compatibility, and there are mechanisms,
discussed at length in the past, which permit gradual migration
without breaking backwards compatibility.  If you cannot see the
absurdity of making a 1000 year prediction based on something
like a 30-year history (thats's for TCP/IP and all of the protocols
built on top of it) in which changes in technology have been
accelerating (compare the number of RFCs issued per decade), then
you have a problem that I cannot help you to solve.

>>>>6. The draft requires modification of non-trace header fields, which is 
>>>>incompatible
>>>>  with message signing. ...
> 
> 
>>>I am not aware of any circumstance where non-trace field modification can
>>>break S/MIME.
>>>
>>
>>Modification of body content (e.g. MIME-part header fields) is one such 
>>case.
> 
> 
> No, this situation cannot arise. See RFC 3156 which states:
>     "For this reason all data signed according to this protocol MUST be
>      constrained to 7 bits (8- bit data MUST be encoded using either
>      Quoted-Printable or Base64)."
> So you could not have a UTF-8 header such as
>     Content-Disposition: attachment; filename = "���"
> within the signed portion (so RFC 2231 encoding would be necessary in that
> case).

That is precisely why signed content must originate in a
compatible format at the UA and be maintained so.  It is
pointless to try to sign something that is required to be
modified (e.g. by a gateway to a moderator) in transit.

>>>>7.  The use of raw utf-8 is incompatible with administrative tools such 
>>>>as grep ...
>>>
>>>Modern versions of grep can handle UTF-8 without problem. ...
>>>
>>
>>Examples have ben provided by others.
> 
> 
> And refuted by others.

And the refutations have been refuted. OK, so that discussion
is going nowhere. No matter, it's only one point of many, and
a relatively minor one.

>>>>8. The draft provides no means of moderation ...
>>>
>>>The draft describes in great detail how to communicate with moderators
> 
> ...
> 
>>No, it does not.
> 
> 
> Then please point out where it fails to do so. We discussed this very
> extensively some months ago.

Indeed we did. And what I actually said is "The draft provides no
means of moderation of extended newsgroups which is both acceptable
to moderators and compatible with the existing installed base of
RFC 1036-compliant UAs and gateways.", and that remains the case.
Encapsulation is unacceptable to moderators, and either passing
raw utf-8 or attempting to second-guess the UA in a gateway is
incompatible with the installed base (for that matter, encapsulation
is also incompatible with the existing installed base)..

>>>Moderators who wish to forward articles
>>>making use of the newly introduced features will, of course,
>>>
>>
>>There are, of course, no new features.
> 
> 
> If there are "no new features" in the draft, then what the Hell are you
> complaining about?

A "feature" is something like internationalized display names,
comments, and text in unstructured fields.  And that feature has
been around for over a decade in a backwards-compatible manner via
MIME.  Raw utf-8 is not a feature; it is a means of implementation.
In this case it also happens to be incompatible with the installed
based, inconsistent with Best Current Practice, etc.

>>One problem is the originators UA.  The draft currently permits the UA to
>>generate illegal content, ...
> 
> 
> May I suggest you take the trouble to READ the latest draft. The UA
> problem was fixed months ago. It would have been helpful if you had taken
> the trouble to comment upon my fix when I presented it.

I have read it; the problem persists. Existing gateways cannot handle
the raw utf-8 content (content which is illegal under the current
standard) that the draft permits UAs to generate.

Date: Mon, 03 Feb 2003 14:37:04 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked], :, [email-masked], [email-masked], "Martin J. Duerst" <[email-masked]>
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: "Martin J. Duerst" <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Language Tagging within Unicode
Sender: [email-masked]

Charles Lindsey wrote:
> This message is addressed primarily to Martin Duerst, but is Cced to
> the Usefor and ietf-822 lists. Will people replying please ensure that
> Martin is copied, since I am not aware that he regularly reads either of
> those lists.
> 
> 
> There currently appears to be some confusion between the Unicode community and
> the Email community as to the correct usage of Language Tagging.
> 
> Within Unicode (since version 3.1) there have existed the Tag Characters
> U+E0000-U+E007F which can, in principle, change the language over and over
> again within a single text.
> 
> However, there are discouraging remarks about the use of this facility:
> 
>     ... However, the use of these characters is strongly discouraged. The
>     characters in this block are reserved for use with special protocols.
>     They are not to be used in the absence of such protocols, or with any
>     protocols that provide alternate means for language tagging, such as
>     HTML or XML.  The requirement for language information embedded in
>     plain text data is often overstated. ...
> 
> (However, there is no defined list of those "special protocols".)

There is; one of the Unicode documents related to the introduction
specifically mentions ACAP as the intended use, as has been pointed
out several months ago on the Usefor list when this topic first came
up, and repeated recently. A search for "ACAP" on the Unicode web
site takes one right to the document and highlights the relevant text.
The full document is: http://www.unicode.org/unicode/reports/tr20/index.html
Martin is listed as one of the co-authors. The relevant section is

"
3.10 Language Tag Characters, U+E0000 .. U+E007F

Short description: A series of characters for expressing language tags, based on existing standards for language tags using the rules in [Unicode31].

Reason for inclusion: These characters allow in-band language tagging in situations where full markup is not available, while allowing easy filtering by applications that do not support them. They were solely included for the benefit of those Internet protocols, such as ACAP, which require a standard mechanism for marking language in UTF-8 strings, and at the same time to avoid the use of other tagging schemes that relied on specific details of the encoding form used.

Problems when used in markup: These characters duplicate information that can be expressed in markup.

Problems with other uses: Their special code range allows them to be easily filtered, but applications that do not expect them will treat them as garbage characters.

Replacement markup: Replace with equivalent language markup. XML and XHTML have the xml:lang attribute. HTML has the lang attribute. These attributes follow different scoping rules than the tag characters, therefore this replacement will generally not be a simple 1:1 substitution.

What to do if detected: Browsers may ignore these characters. When received in an editing context, editors may remove or replace them by equivalent markup.
"

[copied and pasted directly from the web site -- Charles, you may
direct complaints about line length to the Unicode consortium.]

>     Because of the extra implementation burden, language tags should be
>     avoided in plain text unless language information is required and it is
>     known that the receivers of the text will properly recognize and maintain
>     the tags. ...

In this case proper recognition requires that all recievers use an
implementation of Unicode 3.1 or 3.2; earlier versions definitely
won't work, and later versions might not work.  Maintaining the
tags not only requires a Unicode editor, it requires an editor which
properly handles the 3.1/3.2 language tags when modifying text -- I
am not aware of any such editor.

> OTOH, within the Email community, I find the following within RFC 2231,
> when introducing a language-tag extension to RFC 2047:
> 
>     In the future it is likely that some character sets will provide
>     facilities for inline language labeling. Such facilities are
>     inherently more flexible than those defined here as they allow for
>     language switching in the middle of a string.

In the sense of character sets as used in RFC 2231, there is no language
lebeling provided by any character set; Unicode 3.1 is not a character
set in that sense -- utf-7 is, but utf-7 (nor any other charset) does
not provide language labeling.

Flexibility is not attained without a price; in the case of paired tags,
one price is high complexity required to maintain tags when editing. Even
higher complexity is required to support nested paired tags, as used in
Unicode 3.1 and 3.2.  That is the "implementation burden" associated
with use of those tags, a burden which does not exist with RFC 2047/2231
tagging as the 2047/2231 methods do not use paired *or* nested tags.

Note also that because of the display rules associated with encoded-words,
it *is* possible to switch languages within a contiguously-displayed
strin using RFC 2047/2231 encoding in human-readable text in comments,
phrases, and in unstructured fields.  It is not possible to do so within
a single *parameter*, but that's not a big deal since most parameters
are intended as protocol elements, not as human-readable text strings.

>     If and when such facilities are developed they SHOULD be used in
>     preference to the language labeling facilities specified here. Note
>     that all the mechanisms defined here allow for the omission of
>     language labels so as to be able to accommodate this possible future
>     usage.

As noted, it hasn't happened yet.

> And also within RFC 3066 I find:
> 
>     The issue of deciding upon the rendering of a character set based on
>     the language tag is not addressed in this memo; however, it is
>     thought impossible to make such a decision correctly for all cases
>     unless means of switching language in the middle of a text are
>     defined (for example, a rendering engine that decides font based on
>     Japanese or Chinese language may produce suboptimal output when a
>     mixed Japanese-Chinese text is encountered).
> 
> Which at least suggests that the ability to change languages in mid-text
> is a useful feature to have.

RFC 2047/2231 provides that capability for human-readable text in message
header fields, MIME-part fields, and MDN and DSN fields, and some rich
text formats (e.g. text/html) provide such capability for message body
text.  RFC 2231 provides the ability to specify language on a per-parameter
basis. Yes, it is a (sometimes) useful feature, and with the exception of
within a single parameter, it is already supported by 2047/2231 for
mesaage header and MIME-part fields, and via MIME-supported rich text
media formats for body content.

> Now whilst that may or may not represent the current view of the Email
> community, it is certainly on the record as such, and so the two
> communities would seem to be pulling in opposite directions.

There is no conflict between 2047/2231 and 3066, nor with all of those
and RFC 2277.  There is no conflict between those RFCs and the Unicode
documents; both sets of documents clearly indicate use of existing
MIME mechanisms in text messages (until such time as a *charset* provides
language labeling -- and even then 2047/2231 will be required indefinitely
to support reading of legacy content).

I don't know off the top of my head whether or not HTML supports *nested*
language tags, but that's not a big issue (for body text), since (unlike
Unicode language-tag-aware editors) there *are* HTML editors widely
available, and I expect that they do the right thing if they support
HTML language tagging.  Moreover, editing body text is practical with an
external program; not so for editing message and MIME-part header fields.

> My immediate concern is with the possibility that raw UTF-8 will become
> the charset for headers within Netnews (and maybe even for Email one day).

Maybe pigs will fly one day; it's not yet time to invest in
pig-manure-resistant umbrellas or reinforce flagpoles to carry the
weight of roosting pigs.

> It is in the nature of headers that the texts within them are short - they
> are not intended for long essays (that is what message bodies are for). So
> it could be argued that language tagging within them is hardly necessary;

See RFC 2277 and consider Subject, Comments, and Keywords headers, not
to mention parenthesized comments in structured header fields and
proper names and nicknames in phrases associated with email mailboxes
(e.g. the proper name "Jesus" is pronounced differently depending on
language).  And there are some individuals with impaired vision who rely
on screen reader technology.

> in which case the question arises as to whether leaving it to Unicode
> tagging for people who really want to do it would be enough.

Unicode 3.1/3.2 are not universally deployed. No header fields may
yet contain unencoded non-ascii octets, either in the message header
or in MIME-part headers.  So the obvious answer is no, Unicode language
tagging is not (yet) practical.  The MIME mechanisms, on the other
hand, have been in place for some time, are widely deployed in mail
and news user agents, and are fully compliant with Best Current
Practice (e.g. RFC 2277).

> Effectively, one would say "It will usually be unnecessary to use language
> tagging within headers but, if it is considered necessary, then the
> language tagging defined for Unicode MAY be used" (note that the
> contexts where this would be applicable are all for human consumption).

That is not applicable in message or MIME-part header fields and
would not comply with the requirements of RFC 2277 for those
fields.  Because the Unicode documents expressly forbid use of the
language tags with MIME, they would be inappropriate for body text
in a MIME message, and there is no way to get body content other
than plain text in US-ASCII with a 7bit transfer encoding except
via MIME.

> So is that an allowable usage according to Unicode 3.2? Observe that
> headers using raw UTF-8 will not be using any MIME protocol.

By definition, no header fields may use raw UTF-8.  Any message
interpreted as a MIME message necessarily uses MIME protocol, and
that includes MIME-part header fields as well as all header fields
in any message that contains a MIME-Version header field. Even if
you package such (non-)"headers" in an application/news-transmission
wrapper, that's still (obviously!) MIME.

> It this is not allowable, then the only way to tag these short texts would
> be through the use of RFC 2047 or RFC 2231 in place of raw UTF-8

No, not "in place of raw UTF-8"; just RFC 2047 and/or RFC 2231 full stop.

 > which is
> OK so far as it goes, but might not be so if those usages are phased out
> in the future (we are talking many many years down the line here, but it
> could happen one day).

So how many pig-manure-resistant umbrellas have you purchased,
Charles?  Or better still, how many would you like to purchase
(such a deal I have for you...)?

As mentioned above, 2047/2231 support will be required indefinitely,
if for no other reason than for reading of legacy content.

Date: Mon, 03 Feb 2003 12:50:21 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: RFC 2231 and CFWS
Sender: [email-masked]

Charles Lindsey wrote:

> Thanks for the pointer to RFC 2978. I see that '%' is also allowed, but
> not '*'. So writing a foolproof parser for extended-initial-value would be
> tricky (you have to scan from R to L until you have identified the two
> rightmost "'"s), but I doubt implementors will bother to go so far.

That is not the only way to skin that particular cat; a grammar-based
parser can easily handle extended parameters including a hypothetical
charset name containing a single quote and/or percent sign without
resorting to reverse scanning -- it's handled by the normal parser
state machine stack.  It's not even particularly "tricky" -- one
simply writes the appropriate grammar rules and it just works.