Subject: Re: IDN (was Did anyone tell Microsoft yet?)
From: "Eric A. Hall" <[email-masked]>
To: Marc Mutz <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Date: Tue, 30 Apr 2002 17:28:42 -0500
List-Id: <ietf-822.imc.org>

Marc Mutz wrote:

> The main problem is therefore the domain part. I must admit that I
> haven't yet read the IDN/punycode draft(s?), but what is the problem
> with using that for the domain?

The biggest problem is that 2047 would not produce labels that conform to
the historically significant subset of letters-digits-hyphen (LDH) which
constitute the hostname syntax used by most applications. There are
several other problems, however, including the use of charset tags for
each label, character substitution (" " vs "_") and other ambiguities.
Although most of the secondary problems could be resolved with a stricter
syntax, there would still be the LDH problem.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Tue, 30 Apr 2002 14:55:51 -0400
Sender: [email-masked]

> I think that the local-part is the smallest of the problems. local-parts are
> uninterpreted and can thus carry any encoding that would be a valid
> local-part in rfc2822, e.g. rfc2047. 

even if the recipient MTA could handle rfc2047 encodings:

- this would still break other message handling software that depends on 
  being able to compare addresses for equality (since 2047 encoding
  doesn't have a canonical form)

- existing MTAs and gateways that translate 2047-encoded character 
  strings would cause addresses to be rewritten to a form in which 
  they would not be valid.

2047 would be a poor choice for the encoding of local-parts.
far better would be to use an encoding similar to that used by IDNA,
since local-parts and DNS names have similar requirements for matching.

Keith

Date: Tue, 30 Apr 2002 20:32:16 +0200
From: Marc Mutz <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: Charles Lindsey <[email-masked]>, [email-masked]
Organization: KDE
Content-description: clearsigned data
Sender: [email-masked]

On Thursday 25 April 2002 13:33, Charles Lindsey wrote:
<snip>
> So we can expect to see headers like
> To: Jürgen Schmidt <[email-masked]>
<snip>

It might be interesting to know how existing mailers handle this. Typing the 
above into the TO: fields of various mailers gives:

Mozilla:
To: =?ISO-8859-1?Q?J=FCrgen?= Schmidt 
	=?ISO-8859-1?Q?=3Cj=FCrgen=40tu=2Dm=FCn?=chen.de>

Netscape 4.78:
To: =?iso-8859-1?Q?J=FCrgen?= Schmidt
        =?iso-8859-1?Q?=3Cj=FCrgen=40tu=2Dm=FCnchen=2Ede=3E?=

KMail 1.4.5:
To: =?iso-8859-1?q?J=FCrgen=20Schmidt?=
  <=?iso-8859-1?q?j=FCrgen=40tu-m=FCnchen=2Ede?=>

Evolution 0.13: I haven't been able to cut'n'paste an ü into the to: field
           there, not even from the Gnome charmap utility...

Sylpheed:
To: Jürgen Schmidt <[email-masked]>
(note: Sylpheed seems to save stuff unencoded in the Outbox)

- From this, one might conclude that simply extending rfc2047 might be the 
easiest upgrade path ;-)

I think that the local-part is the smallest of the problems. local-parts are 
uninterpreted and can thus carry any encoding that would be a valid 
local-part in rfc2822, e.g. rfc2047. The burden to make sense out of this is 
on the receiving end: If someone allows mailboxes with non-US-ASCII 
characters on her machine, then she needs to make sure that she installs 
tools that can handle the new encoding. For everyone else, the local-part 
just looks funny.

The main problem is therefore the domain part. I must admit that I haven't yet 
read the IDN/punycode draft(s?), but what is the problem with using that for 
the domain?

Marc

- -- 
Marc Mutz <[email-masked]>

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Tue, 30 Apr 2002 13:31:17 -0400
Sender: [email-masked]

> >> So we need to know now where Email is going on the IDNA front, and if
> >> there are going to be changes to accomodate IDNA, then one might as well
> >> look into the UTF-8 change at the same time. One upheaval is always better
> >> than two.
> 
> >so when you want to go up a hill, you'd rather climb one 10 foot wall
> >than 10 1-foot steps?
> 
> Yes, in some situations. Particularly when the cost of making a move
> includes a large constant term independent of the height of the step.

if that were the case, I'd agree with you.  but it doesn't apply here,
because there's no need for a massive upgrade.
 
> For example, the cost (in terms of general irritation, hassle, etc) of
> persuading the whole email community to implement some change worldwide is
> largely independent of the size of that change. 

yes, but the cost of persuading the whole email community to implement
a change to UTF-8 in any timeframe significantly shorter than 20 years
is so large that it seems totally infeasible.

who is going to pay for such a change?  who is going to want to deal
with the disruption that it would cause?  certainly not the large number
of users for which ASCII-only addresses and 2047 encodings work 
reasonably well.

the reason many of us favor a solution that doesn't require a massive
upgrade is that we realize that a massive upgrade of the existing mail
system, which would not benefit a significant fraction of current users, 
is a non-starter.  OTOH, if we recommend a solution that is compatible 
with existing mail formats and message handling software, users can
upgrade whenever they see a benefit in doing so.  most of the software
in use when the standard was adopted would get upgraded within a few 
years anyway.

and there's so little benefit in "pure UTF-8" that it's conceivable 
that it would never be worth the trouble to transition to it.  for 
example, "pure UTF-8" wouldn't rid the user agent of the need to 
parse message header fields and to treat different parts of a 
structured field in different ways.  

Keith

To: [email-masked]
Xref: clerew local.mime:1311
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: encoded-words, parameter continuation and visually ordered charsets
Date: Tue, 30 Apr 2002 08:54:12 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>It should in any case be possible to decode all encoded-words, in
>the order in they appear, into a single string of Unicode symbols and 
>then present that string.  If the charset in any of those encoded
>words is implicitly visually ordered then it will be necessary
>for the corresponding sequence of Unicode symbols to be marked as 
>visually ordered also.

Yes, I think that is a desirable property.

Consider the case where the material in question was originally a string
of Unicode symbols, including the proper codings to change the direction
from/to RTL at various points.

It is now passed to some RFC 2047 encoding mechanism (possibly in a
different application from the one that generated the string) in order
that it may become a valid RFC 2822 message. This encoding mechanism
chooses to split the string into encoded words at arbitrary points not
related to the direction changes (because it is a totally naive encoder
knowing nothing of the subtleties of Unicode - just trying to keep the
encoded words within that length limit).

At the far end, it has to be turned back into Unicode, and clearly we want
the overall encoding/decoding to be a nullop.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1313
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Tue, 30 Apr 2002 09:00:06 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> So we need to know now where Email is going on the IDNA front, and if
>> there are going to be changes to accomodate IDNA, then one might as well
>> look into the UTF-8 change at the same time. One upheaval is always better
>> than two.

>so when you want to go up a hill, you'd rather climb one 10 foot wall
>than 10 1-foot steps?

Yes, in some situations. Particularly when the cost of making a move
includes a large constant term independent of the height of the step.

For example, the cost (in terms of general irritation, hassle, etc) of
persuading the whole email community to implement some change worldwide is
largely independent of the size of that change. My point is that we are
shortly have to do that in any case in order to bring IDNA in in an
orderly fashion.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1312
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Tue, 30 Apr 2002 08:55:38 GMT
Lines: 20
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>My understanding is that sendmail uses characters in the ISO 8859
>unassigned code space (the ones that correspond to control characters when
>the eighth bit is stripped).  So you won't notice that effect if you're
>using European characters, but you may notice with other character sets.

Urrrrghhhh! That is a mean trick. Does anyone know if they plan to change
that in some foreseeable future?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 30 Apr 2002 15:10:51 +0200
From: Marc Mutz <[email-masked]>
Subject: Re: encoded-words, parameter continuation and visually ordered charsets
To: Keith Moore <[email-masked]>
Cc: [email-masked]
Organization: KDE
Content-description: clearsigned data
Sender: [email-masked]

On Monday 29 April 2002 14:29, Keith Moore wrote:
<snip>
> It should in any case be possible to decode all encoded-words, in
> the order in they appear, into a single string of Unicode symbols and
> then present that string.  If the charset in any of those encoded
> words is implicitly visually ordered then it will be necessary
> for the corresponding sequence of Unicode symbols to be marked as
> visually ordered also.
<snip>

AFIU, this corresponds to the "unicode level" version in my original mail, 
right?

Marc

- -- 
Marc Mutz <[email-masked]>

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Tue, 30 Apr 2002 09:41:14 -0400
Sender: [email-masked]

> >also, even if transports are "munge free", that doesn't mean that they
> >can parse utf-8 headers correctly.  and various kinds of transports
> >do parse headers.
> 
> OK, so which headers do transports need to parse?

In practice, things that handle mail (like mailing list software,
gateways, virus filters,  spam filters, etc. - certainly not just 
MTAs) parse most of the structured fields for one reason or another.  
Several of them also try to rewrite such fields or to correct 
apparent errors in such fields - for instance, by taking strings 
of illegal characters and quoting them according to RFC 2047
(regardless of where those characters appear in the header).
I've also seen illegal characters replaced by \XXX where XXX is
the octal representation of the codepoint.

It's silly to assume that hundreds of implementations of mail 
software, some of them more than 20 years old, are going to do 
reasonable things with significant amounts of input that is 
clearly illegal according to the specifications for which those
programs were written.  

Keith

To: [email-masked]
Xref: clerew local.mime:1310
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Tue, 30 Apr 2002 08:46:51 GMT
Lines: 35
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>also, even if transports are "munge free", that doesn't mean that they  
>can parse utf-8 headers correctly.  and various kinds of transports
>do parse headers.

OK, so which headers do transports need to parse?

The address headers presumably (either to construct the envelope as in
'sendmail -t', or in transports like uucp which use the To and Cc headers
directly).

The Received header (well, generate rather than parse).

Maybe the Date (for logging)

But I doubt they need to parse Message-ID, References, and the like,
except so far as is needed to extract a unique string of octets for
logging. Anyway, I would recommend insisting on strict ASCII within
msg-ids.

So let us identify the cases and see what troubles might ensue.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: Russ Allbery <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Mon, 29 Apr 2002 20:31:12 -0400
Sender: [email-masked]

also, even if transports are "munge free", that doesn't mean that they  
can parse utf-8 headers correctly.  and various kinds of transports
do parse headers.

Keith

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Mon, 29 Apr 2002 20:26:56 -0400
Sender: [email-masked]

> So we need to know now where Email is going on the IDNA front, and if
> there are going to be changes to accomodate IDNA, then one might as well
> look into the UTF-8 change at the same time. One upheaval is always better
> than two.

so when you want to go up a hill, you'd rather climb one 10 foot wall
than 10 1-foot steps?

To: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Mon, 29 Apr 2002 14:00:37 -0700
Lines: 15
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> Now, AIUI, most transports are munge free. All 8 bits get through. The
> only exception I am aware of is sendmail, which is said to use the 8th
> bit of some headers for internal flags. Does anyone know more detail of
> this (it certainly seems to be able to accomodate 8 bits in the Subject
> line AFAICS)?

My understanding is that sendmail uses characters in the ISO 8859
unassigned code space (the ones that correspond to control characters when
the eighth bit is stripped).  So you won't notice that effect if you're
using European characters, but you may notice with other character sets.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

To: [email-masked]
Xref: clerew local.mime:1306
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Mon, 29 Apr 2002 16:56:58 GMT
Lines: 71
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> Has anybody looked into the transitional issues that would arise if Email
>> was to be moved into allowing UTF-8 headers?

>Yes. There are a number of sticking points, but on the whole it is doable.
>The real question is on what scale the benefits outweigh the costs.

AIUI there are two sets of problems.
	Transport Agents
	User Agents

The problem for transport agents (which inlcudes POP servers, and the
like) is to ensure that the UTF-8 characters are correctly delivered. Such
agents are not usually concerned with the content of the headers except
for addresses and maybe dates. They are going to have to be fixed in any
case so that addresses use IDNA/whatever. Dates should be left strictly in
ASCII. So the real question is munging, tuncation of 8th bit, etc.

Now, AIUI, most transports are munge free. All 8 bits get through. The
only exception I am aware of is sendmail, which is said to use the 8th bit
of some headers for internal flags. Does anyone know more detail of this
(it certainly seems to be able to accomodate 8 bits in the Subject line
AFAICS)?

However, there already exist downgrading mechanisms for use if needed. RFC
2047 for unstructureds and comments. RFC 2231 for parameters.
IDNA/whatever for addresses. What else is there? Clearly, you define that
mesg-ids, tokens, header-names remain in ASCII (that is what Usefor has
done). What problems remain?

OTOH, expecting transports to downgrade may be a forlorn hope. Transports
which advertise 8BITMIME are supposed to downgrade when forwarding to one
that doesn't, but few actually do it. This does not appear to have caused
the sky to fall in, however.

User agents have different problems, since they need to display headers,
and also to understand them for various purposes. Clearly, existing user
agents are going to be around for much longer than existing transport
agents. OTOH, one can take the view (as Usefor does) that, if you expect
to receive communications in Chinese (whatever), you make sure you upgrade
to a User Agent with the necessary capability. If you don't expect to
receive such communications, then you don't bother. The worst that then
happens is that you see some gibberish from time to time. Well, actually,
I see such gibberish all the time from Korean spam, but I don't regard
that as a bug in my User Agent, but rather as a bug in the Internet in
Korea :-( .

As to when the time would be ripe to make such a change, I think we need
to realize that we are going to HAVE to make some changes to accomodate
IDNA pretty soon now. For sure, as soon as IDNA becomes an RFC, people
will expect these domain names to suddenly start working in email (and
they will, or course be diasppointed, but that won't stop all sorts of ad
hoc fixes from appearing).

So we need to know now where Email is going on the IDNA front, and if
there are going to be changes to accomodate IDNA, then one might as well
look into the UTF-8 change at the same time. One upheaval is always better
than two.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1305
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Mon, 29 Apr 2002 16:16:03 GMT
Lines: 73
Sender: [email-masked]

In <[email-masked]> "Charles Lindsey" <[email-masked]> writes:

>In <[email-masked]> Keith Moore <[email-masked]> writes:

>>If case-folding is done before encoding as in nameprep, then the comparison
>>will succeed.  If the case-folding is not done, then the comparison will 
>>fail unless the two local-parts being compared are spelled with the same
>>casing - essentially forcing case-insensitivity.

>Case-folding is easily done in ASCII, and even in ISO-8859-1, but there is
>no way a mailer can do case folding in some strange part of Unicode that
>it has never heard of (or which had not even been defined at the time that
>mailer was compiled).

>We had this problem with newsgroup-names in Usefor, and the best we could
>do was to say that they MUST be in lower-case (to be enforced by hierarchy
>administrators who issue newgroup messages). Anyone who types a name with
>an upper -case letter MIGHT be lucky and have it arrive in the correct
>newsgroup, but more likely it would result in posting to s non-existent
>group.

>BTW, how does NAMEPREP cope with this problem? Looks like another draft I
>need to look at :-( .

Right, I have looked at NAMEPREP now, and am somewhat horrified.

Firstly, any implementation of it MUST contain a huge set of tables of
characters to be translated and characters to be disallowed.

Secondly, the tables are based on Unicode 3.1. There is no provision for
the tables to be updated for future versions of Unicode. It will need a
new RFC. This seems contrary to the philosophy of Unicode according to
which they have agreed that future versions of Unicode will always be
upwards compatible. Essentially, this means that all that will ever happen
is that the table of Unassigned Codepoints will be reduced, and some of
the formerly Unassigned Codes may then appear in the other tables. But
NAMEPREP seems not to envisage this possibility, so it is a disater
waiting to happen.

So my view is that it is too much to expect these tables to be correctly
incorporated in the User's mailing agent in every PC worldwide. However,
there are far fewer Mail Transport Agents in the world than there are PCs,
so there may be some hope that their tables will be up to date, and kept
so.

So I think that makes a strong case for doing any IDNA conversions at the
front end of MTAs, rather than in User Agents, which is contrary to what
Keith was suggesting a couple of days ago.

Note that the approach we took in Usefor for newsgroup-names was somewhat
more draconian. New newsgroups MUST NOT be created except in the proper
normalized form (so those creating new groups need to understand their
Unicode). Then, so far as User Agents are concerned, they MUST use the
canonical form (which is lowercase, NFKC in fact). If they get it wrong
their articles may not appear where expected, which is Tough. Posting
agents MAY use something like Nameprep, but there is no obligation to do
so.

But I can quite well see that you cannot be as draconian as that in DNS
applications or in Email.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
cc: [email-masked]
Subject: Re: encoded-words, parameter continuation and visually ordered charsets 
Date: Mon, 29 Apr 2002 08:29:00 -0400
Sender: [email-masked]

I do not think in terms of "sorting" encoded-words before presentation,
and I doubt this is the best way to describe the process.

IMHO the encoded-words should be decoded left-to-right, that is, in the
order in which they appear in the message. However, this should not
constrain the ordering of presentation of RTL text.   If multiple encoded-
words are needed to encode a single segment of RTL text, the characters
within that segment should ideally be encoded in logical order, so that the 
first encoded-word contains the characters that are read first, the
second encoded-word contains the characters following that, and so on.  
Thus it may be necessary for a receiving MUA (at least, one that generates
the display output in strict LTR order) to decode multiple encoded-words 
before it can display any of them - up to a single line of output.

(However even an MUA that cannot display RTL text may need to buffer
multiple encoded-words in order to generate a visually-pleasing 
presentation of the header field, so I don't see this as an additional
burden for supporting RTL text)

It should in any case be possible to decode all encoded-words, in
the order in they appear, into a single string of Unicode symbols and 
then present that string.  If the charset in any of those encoded
words is implicitly visually ordered then it will be necessary
for the corresponding sequence of Unicode symbols to be marked as 
visually ordered also.

Date: Sun, 28 Apr 2002 21:59:28 +0200
From: Marc Mutz <[email-masked]>
Subject: encoded-words, parameter continuation and visually ordered charsets
To: [email-masked]
Organization: KDE
Content-description: clearsigned data
Sender: [email-masked]

Hi!

Neither RFC2047 nor RFC2231 seem to define how to handle sequences of 
encoded-words and parameter continuation with visually-ordered charsets like 
iso-8859-8.

There are two alternatives, which differ in when the splitting into 
encoded-words occurs (assuming user input is processed in Unicode 
internally):

1. encoded-words are sorted logically, even when the charset used by them is
   visually ordered:
   That's kind of natural, since the different encoded-words can use
   different charsets each, not all of which may be visually ordered.
   An implementation would thus split the to-be-encoded text at the unicode 
   level, encode each part into a fitting charset, then encode the resulting 
   octet-sequences as encoded-words and insert the encoded words in the order 
   in which they correspond to the source unicode character sequence:

   stringlist = split( unicode_string );
   foreach string in stringlist {
     eightBitText = applyCharsetTransformation( string, charset );
     encodedWord = encodeRFC2047( eightBitText, charset, language );
     phrase.append( encodedWord );
   }
   header.insert( phrase );

2. encoded-words are sorted visually if the the charset used is so.
   An implementation would thus encode the to-be-encoded text using a fitting 
   charset, then split the resulting octet-sequence (with knowledge of 
   character boundaries), then encodes each octet-sequence into encoded-words:

   eigtBitString = applyCharsetTransformation( unicode_string, charset );
   eightBitStringList = split( eightBitString );
   foreach chunk in eightBitStringList {
     encodedWord = encodeRFC2047( eigthBitText, charset, language );
     if ( charset.isRTL ) {
       phrase.prepend( encodedWord );
     } else {
       phrase.append( encodedWord );
     }
   }
   header.insert( phrase );

RFC2231 filename*0*=iso-8859-8''foobarbaz; filename*1=Foo has the same 
ambiguity.

We currently use (1) and had no users complaining about that. But what do 
other implementations do?

Marc

- -- 
Marc Mutz <[email-masked]>

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Fri, 26 Apr 2002 22:52:31 -0400
Sender: [email-masked]

> >If case-folding is done before encoding as in nameprep, then the comparison
> >will succeed.  If the case-folding is not done, then the comparison will
> >fail unless the two local-parts being compared are spelled with the same
> >casing - essentially forcing case-insensitivity.
> 
> Case-folding is easily done in ASCII, and even in ISO-8859-1, but there is
> no way a mailer can do case folding in some strange part of Unicode that
> it has never heard of (or which had not even been defined at the time that
> mailer was compiled).

correct.  and IDNA doesn't do case-folding except for ASCII letters, for
consistency with the handling of ASCII-only DNS names.

Keith

To: [email-masked]
Xref: clerew local.mime:1296
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 26 Apr 2002 11:51:50 GMT
Lines: 32
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>If case-folding is done before encoding as in nameprep, then the comparison
>will succeed.  If the case-folding is not done, then the comparison will 
>fail unless the two local-parts being compared are spelled with the same
>casing - essentially forcing case-insensitivity.

Case-folding is easily done in ASCII, and even in ISO-8859-1, but there is
no way a mailer can do case folding in some strange part of Unicode that
it has never heard of (or which had not even been defined at the time that
mailer was compiled).

We had this problem with newsgroup-names in Usefor, and the best we could
do was to say that they MUST be in lower-case (to be enforced by hierarchy
administrators who issue newgroup messages). Anyone who types a name with
an upper -case letter MIGHT be lucky and have it arrive in the correct
newsgroup, but more likely it would result in posting to s non-existent
group.

BTW, how does NAMEPREP cope with this problem? Looks like another draft I
need to look at :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
cc: Keith Moore <[email-masked]>, "Charles Lindsey" <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Fri, 26 Apr 2002 21:11:00 -0400
Sender: [email-masked]

> Keith Moore <[email-masked]> writes:
> 
> >> Has anybody looked into the transitional issues that would arise if Email
> >> was to be moved into allowing UTF-8 headers?
> >
> > yes, though perhaps not in sufficient detail.  
> 
> Simply stating that it wouldn't work completely with current systems,
> but could work in future modified systems, seems compelling enough to
> go for it in the long run to me.  (I doubt anyone is challenging that
> position?)

actually I haven't seen a single compelling argument to move to utf-8.
however if I had the luxury of designing a new mail system, I would
certainly use utf-8 for human-readable text.  I just don't think that
we'll have that luxury for a long time.

> > frankly I think it would be easier to transition to a completely different
> > mail format.  that is, I think that having two slightly different 
> > mail formats (one ascii, another utf-8) is more confusing and more
> > likely to lead to interoperability problems than having two obviously
> > distinct formats which require explicit translation between the two.
> 
> I can't see why this would be easier, could you elaborate?  

because with a completely incompatible (though translatable) format 
nobody would pretend that it's acceptable to shove a new-format
message into a mail system that only understands 822/2822/MIME.
it will be absolutely necessary to translate at the boundaries
between the two systems, and it will be absolutely clear where
such translation is necessary.   if the translation fails, 
it will also be fairly clear *where* the translation failed.

on the other hand, if you just shove utf-8 into protocol elements 
that currently expect ASCII, there is no clear boundary between the 
systems - and thus no clear (to the user) deliniation of who is 
responsible for the translation.  

> Having a
> new format that is perfectly backwards compatible if 7bit encodings
> are used (as a utf-8 wire format would be) seems much better than
> inventing something completely different.

if you think about it, "compatible if 7bit encodings are used"
isn't terribly important - since the entire justification for
this change is to *not* be restricted to 7bit encodings.

> >> Or are you resigned to the fact that we shall still be encoding headers
> >> using RFC 2047 in 40 years time even though all transports and operating
> >> systems will be 8bit (and even bibary) clean by that time?
> >  
> > please don't cite 40 years of extrapolation from current practice as
> > 'the fact'.  
> 
> It will be 40 years unless someone is thinking about the future
> upgrade path now.  

first of all, people are thinking about the future upgrade path
now - it's just that we can't usefully specify it at this point.
second, the 40 that's pure speculation on your part, and totally
unwaranted.  Internet mail hasn't even been around 30 years yet,
and it's still evolving - why in the world do you think you can
expect it to be static for the next 40 years?  

> IMHO it should have been specified that software
> must move to 8bit clean after 2000-01-01, or something, back when the
> MIME specs were written 

hey, if you think it's reasonable to impose your idea of appropriate
operational constraints on people who were there at the time, dream 
on.  we could only mandate those things for which we could get consensus.
there was wide consensus that a flag day wouldn't work, there was 
significant objection at the time to *ever* expecting systems
to be 8-bit clean, 8-bit cleanliness would not have rid us of the need
for encodings in MIME (including the need for something like 2047 -
we would still have needed a per-element charset label because utf-8
was not available).  besides which 8-bit cleanliness is the *least* 
of the problems associated with using utf-8.  

> 
> > no, I don't think that 8bit transparency of mail transports alone is 
> > sufficient to warrant the pain of that transition - because zero
> > additional functionality is gained beyond a small bandwidth savings, 
> > and the pain of having to upgrade millions of installations of hundreds
> > of different mail programs (MTAs, UAs, list servers, mail filters, 
> > POP servers, IMAP servers, etc.) is too substantial for that minimal
> > gain - which is mostly wanted for the sake of purity.  it doesn't 
> > lessen complexity, it increases it - because mail handling programs
> > would still have to recognize the old format as well as the new one
> > for many years.
> >
> > what would be worth the pain of transition?  a hypothetical redesign 
> > of the internet email system that is both reliable (in that you can 
> > expect messages to be delivered with integrity) and safe to use (in 
> > that it won't expose your systems to security threats).
> 
> The benefit would be reduced software complexity.  

that argument simply doesn't hold up under analysis.  

> Right now it seems as if the internet email system just gets more and 
> more complex to implement.  IMHO that's not the best path to take.

that's the way systems tend to evolve - they get more and more complex 
over time (as they become more efficient or adapt to new circumstances)
until changes in external conditions prompt a revolutionary change.
then they start getting more and more complex again.  

Keith

Date: Fri, 26 Apr 2002 16:29:43 -0700 (PDT)
From: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> Keith Moore <[email-masked]> writes:

> >don't worry about UTF-8 or usenet compatibility anytime soon.
> >it's far more important to minimize disruption in email service
> >than it is to use utf-8 on the wire, or to try to keep email and
> >usenet in sync.

> Has anybody looked into the transitional issues that would arise if Email
> was to be moved into allowing UTF-8 headers?

Yes. There are a number of sticking points, but on the whole it is doable.
The real question is on what scale the benefits outweigh the costs.

> Or are you resigned to the fact that we shall still be encoding headers
> using RFC 2047 in 40 years time even though all transports and operating
> systems will be 8bit (and even bibary) clean by that time?

I'm resigned to the fact that people don't seem to understand that creating a
backwards-compatible scheme is a necessary first step. Once this is done all
sorts of cute things, including but not limited to UTF-8 message headers, are
possible. But this can only happen after the initial work is done.

				Ned

To: Keith Moore <[email-masked]>
Cc: "Charles Lindsey" <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
From: Simon Josefsson <[email-masked]>
Date: Sat, 27 Apr 2002 00:47:09 +0200
Lines: 60
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> Has anybody looked into the transitional issues that would arise if Email
>> was to be moved into allowing UTF-8 headers?
>
> yes, though perhaps not in sufficient detail.  

Simply stating that it wouldn't work completely with current systems,
but could work in future modified systems, seems compelling enough to
go for it in the long run to me.  (I doubt anyone is challenging that
position?)

> frankly I think it would be easier to transition to a completely different
> mail format.  that is, I think that having two slightly different 
> mail formats (one ascii, another utf-8) is more confusing and more
> likely to lead to interoperability problems than having two obviously
> distinct formats which require explicit translation between the two.

I can't see why this would be easier, could you elaborate?  Having a
new format that is perfectly backwards compatible if 7bit encodings
are used (as a utf-8 wire format would be) seems much better than
inventing something completely different.

>> Or are you resigned to the fact that we shall still be encoding headers
>> using RFC 2047 in 40 years time even though all transports and operating
>> systems will be 8bit (and even bibary) clean by that time?
>  
> please don't cite 40 years of extrapolation from current practice as
> 'the fact'.  

It will be 40 years unless someone is thinking about the future
upgrade path now.  IMHO it should have been specified that software
must move to 8bit clean after 2000-01-01, or something, back when the
MIME specs were written (much like some PKIX standards specify that
after 2003-01-01, or something, certificates with certain properties
should not be issued by CAs).  Now we're stuck with quoted-printable
and other hacks for maybe 40 years.  The only software still not
adopting MIME is because MIME is too complex for the purpose (e.g.,
/bin/mail), and they will never upgrade now.

> no, I don't think that 8bit transparency of mail transports alone is 
> sufficient to warrant the pain of that transition - because zero
> additional functionality is gained beyond a small bandwidth savings, 
> and the pain of having to upgrade millions of installations of hundreds
> of different mail programs (MTAs, UAs, list servers, mail filters, 
> POP servers, IMAP servers, etc.) is too substantial for that minimal
> gain - which is mostly wanted for the sake of purity.  it doesn't 
> lessen complexity, it increases it - because mail handling programs
> would still have to recognize the old format as well as the new one
> for many years.
>
> what would be worth the pain of transition?  a hypothetical redesign 
> of the internet email system that is both reliable (in that you can 
> expect messages to be delivered with integrity) and safe to use (in 
> that it won't expose your systems to security threats).

The benefit would be reduced software complexity.  No need to worry
about quoted-printable and similar 7bit hacks.  Right now it seems as
if the internet email system just gets more and more complex to
implement.  IMHO that's not the best path to take.

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet? 
Date: Fri, 26 Apr 2002 14:06:05 -0400
Sender: [email-masked]

> >> However, I would like to know why 2047 is thought to be a "poor mechanism"
> >> for this purpose (i.e. why was it not standardized that way - it seems to
> >> work in a wide variety of situations).
> 
> 
> >partially because 2047 imposes strict limitations on the lengths of
> >encoded-words, doesn't allow them to appear in quoted strings, and
> >contains characters which aren't legal in MIME parameters.
> 
> But WHY doesn't it allow them to appear in quoted strings? What harm
> ensures?

it was an explicit decision of the working group.  I think the idea
was that the whole intent of a quoting mechanism is to convey things
intact from end-to-end, so quoted strings should not be subject
to additional interpretation.  2047 is in effect an additional quoting
mechanism, to be used instead of "..." rather than in conjunction with it.

and of course 2047 simply was not designed for use with parameters  -
so the WG didn't consider that use case.  

> I just tried a test message to
>         To: "Jürgen D. Schmidt" <...>
> and my mailer (dtmail) sent it as
>         To: "=?ISO-8859-1?Q?J=FCrgen_D._Schmidt?=" <...>
> which is, of course, wrong even though most mailers would have done the
> same. The "correct" way to do it is, apparently
>         To: =?ISO-8859-1?Q?J=FCrgen?= "D. Schmidt" <...>
> 
> And the limitation on length doesn't really matter because there is
> provision for sticking split pieces together again.

right but it makes assumptions about how MTAs and other mail
handling software wraps message header fields (based on existing 
practice in the early 1990s), which doesn't hold for headers within 
message bodies.

> >partially because there are a few things out there that unilaterally
> >translate all 2047 encodings in a message from one charset to another,
> >or that decode all 2047 encodings - and 2047 was explicitly designed to
> >permit the former.
> 
> And you are saying that RFC2231 does not allow that, and that was a good
> enough reason why RFC2231 encoded differently than RFC2047? 

this is part of why we decided to encode parameters differently - we
decided that 2047 did not work well enough to be used as a general-purpose
parameter-encoding mechanism.  (we weren't looking at just the problem
of encoding filenames, we were looking at the problem of encoding
parameters)

Keith

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Fri, 26 Apr 2002 13:59:24 -0400
Sender: [email-masked]

> Has anybody looked into the transitional issues that would arise if Email
> was to be moved into allowing UTF-8 headers?

yes, though perhaps not in sufficient detail.  

frankly I think it would be easier to transition to a completely different
mail format.  that is, I think that having two slightly different 
mail formats (one ascii, another utf-8) is more confusing and more
likely to lead to interoperability problems than having two obviously
distinct formats which require explicit translation between the two.
 
> Or are you resigned to the fact that we shall still be encoding headers
> using RFC 2047 in 40 years time even though all transports and operating
> systems will be 8bit (and even bibary) clean by that time?
 
please don't cite 40 years of extrapolation from current practice as
'the fact'.  

yes, I expect that there will eventually be a transition to a new message 
format.

no, I don't think that 8bit transparency of mail transports alone is 
sufficient to warrant the pain of that transition - because zero
additional functionality is gained beyond a small bandwidth savings, 
and the pain of having to upgrade millions of installations of hundreds
of different mail programs (MTAs, UAs, list servers, mail filters, 
POP servers, IMAP servers, etc.) is too substantial for that minimal
gain - which is mostly wanted for the sake of purity.  it doesn't 
lessen complexity, it increases it - because mail handling programs
would still have to recognize the old format as well as the new one
for many years.

what would be worth the pain of transition?  a hypothetical redesign 
of the internet email system that is both reliable (in that you can 
expect messages to be delivered with integrity) and safe to use (in 
that it won't expose your systems to security threats).

my guess is that this is at least 10-15 years away though.  there's
just too much investment in the current infrastructure.

Keith

To: [email-masked]
Xref: clerew local.mime:1293
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet?
Date: Fri, 26 Apr 2002 11:17:32 GMT
Lines: 46
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> However, I would like to know why 2047 is thought to be a "poor mechanism"
>> for this purpose (i.e. why was it not standardized that way - it seems to
>> work in a wide variety of situations).

>partially because 2047 imposes strict limitations on the lengths of 
>encoded-words, doesn't allow them to appear in quoted strings, and 
>contains characters which aren't legal in MIME parameters. 

But WHY doesn't it allow them to appear in quoted strings? What harm
ensures?

I just tried a test message to
	To: "Jürgen D. Schmidt" <...>
and my mailer (dtmail) sent it as
	To: "=?ISO-8859-1?Q?J=FCrgen_D._Schmidt?=" <...>
which is, of course, wrong even though most mailers would have done the
same. The "correct" way to do it is, apparently
	To: =?ISO-8859-1?Q?J=FCrgen?= "D. Schmidt" <...>

And the limitation on length doesn't really matter because there is
provision for sticking split pieces together again.

>partially because there are a few things out there that unilaterally
>translate all 2047 encodings in a message from one charset to another,
>or that decode all 2047 encodings - and 2047 was explicitly designed to
>permit the former.

And you are saying that RFC2231 does not allow that, and that was a good
enough reason why RFC2231 encoded differently than RFC2047? Remember that
the case we are looking as is where a name parameter in a Content-Type (or
more properly a Content-Distribution) was written as
	name "=?koi8-r?B?Q8/XxdQuZG9j?="

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1295
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 26 Apr 2002 11:37:49 GMT
Lines: 45
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> > a. encode domain names in IDNA
>> > b. encode local-parts in an encoding that is to be defined, but which
>> >    hopefully is very similar, perhaps identical, to IDNA
>> 
>> The problem here is that nameprep is stricter than mailbox naming has
>> historically allowed for. Specifically, the use of mandatory lowercase
>> characters in nameprep will prohibit mixed-case local-parts.

>Good point.  OTOH, if we defined a special version of nameprep for 
>local-parts to allow mixed-case, then local-parts of internationalized
>email addresses would be subject to case-sensitive comparison.
>Of course, strictly speaking, interpretation of the local-part is up to 
>the domain, but in practice, case-insensitivity is often taken for
>granted by both users and software.

I have not studied the Punycode specification in detail, but I have
experimented using the code provided in the appendix.

If I encode CHLchl, itc comes out as chlchl- (which then decoded to
chlchl).

However, if I construct CHLchl- and then decode _that_, it comes out as
CHLchl. So Punycode appears not to be reversible, which is a Bad Thing.

But there is a further problem. Presumably, two '-'s in succession are
illegal in a domain name, so if the DNS sees zz--tu-mnchen-t9a, it knows
that decoding is needed. But zz--jrgen-kva is already a valid local-part as
it stands. Perhaps the mail really was intended for that entity.

So I think some limitation of the RFC 2822 syntax is going to be required
whatever.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:1294
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Date: Fri, 26 Apr 2002 11:21:17 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>don't worry about UTF-8 or usenet compatibility anytime soon.
>it's far more important to minimize disruption in email service
>than it is to use utf-8 on the wire, or to try to keep email and 
>usenet in sync.

Has anybody looked into the transitional issues that would arise if Email
was to be moved into allowing UTF-8 headers?

Or are you resigned to the fact that we shall still be encoding headers
using RFC 2047 in 40 years time even though all transports and operating
systems will be 8bit (and even bibary) clean by that time?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 25 Apr 2002 15:38:06 -0400
Sender: [email-masked]

> > Good point.  OTOH, if we defined a special version of nameprep for
> > local-parts to allow mixed-case, then local-parts of internationalized
> > email addresses would be subject to case-sensitive comparison.
> 
> Are you referring to the encoded or unencoded (original) representation?

I'm assuming that the comparison will be done using the encoded version,
because many tools (including but not limited to SMTP servers) will not 
be ACE-aware and will treat the addresses as if they were ordinary ASCII.

> ACE encoded representations of variable-case local-parts will always fail
> since the encoding sequence will differ for each input. 

If case-folding is done before encoding as in nameprep, then the comparison
will succeed.  If the case-folding is not done, then the comparison will 
fail unless the two local-parts being compared are spelled with the same
casing - essentially forcing case-insensitivity.

> OTOH, if the mail
> system extracts the original unencoded sequence as part of delivery then
> it can do whatever comparisons it wants.

Which essentially means that if the nameprep for local parts does not
do case-folding, then the burden for doing the case-sensitive comparison 
moves to the SMTP server  and other tools that must handle the mail - not 
a desirable result IMHO.   
 
> The upside with mail is that comparison is purely a local issue (in
> theory) as opposed to DNS where comparison occurs at every cache in
> between the end-points.

It's more a local issue in theory than in practice.  Lots of things
depend on the ability to compare email addresses - mailing lists 
that check against a list of subscribers being one common example.

Keith

Date: Thu, 25 Apr 2002 14:27:38 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

Keith Moore wrote:

> Good point.  OTOH, if we defined a special version of nameprep for
> local-parts to allow mixed-case, then local-parts of internationalized
> email addresses would be subject to case-sensitive comparison.

Are you referring to the encoded or unencoded (original) representation?

ACE encoded representations of variable-case local-parts will always fail
since the encoding sequence will differ for each input. OTOH, if the mail
system extracts the original unencoded sequence as part of delivery then
it can do whatever comparisons it wants.

The upside with mail is that comparison is purely a local issue (in
theory) as opposed to DNS where comparison occurs at every cache in
between the end-points.

Another option is for the i18n mail people (whoever that turns out to be)
to just mandate that i18n local-parts are always lowercase normalized.

> Of course, strictly speaking, interpretation of the local-part is up to
> the domain, but in practice, case-insensitivity is often taken for
> granted by both users and software.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 25 Apr 2002 15:16:19 -0400
Sender: [email-masked]

> > a. encode domain names in IDNA
> > b. encode local-parts in an encoding that is to be defined, but which
> >    hopefully is very similar, perhaps identical, to IDNA
> 
> The problem here is that nameprep is stricter than mailbox naming has
> historically allowed for. Specifically, the use of mandatory lowercase
> characters in nameprep will prohibit mixed-case local-parts.

Good point.  OTOH, if we defined a special version of nameprep for 
local-parts to allow mixed-case, then local-parts of internationalized
email addresses would be subject to case-sensitive comparison.
Of course, strictly speaking, interpretation of the local-part is up to 
the domain, but in practice, case-insensitivity is often taken for
granted by both users and software.

Keith

Date: Thu, 25 Apr 2002 14:10:00 -0500
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

Keith Moore wrote:

> a. encode domain names in IDNA
> b. encode local-parts in an encoding that is to be defined, but which
>    hopefully is very similar, perhaps identical, to IDNA

The problem here is that nameprep is stricter than mailbox naming has
historically allowed for. Specifically, the use of mandatory lowercase
characters in nameprep will prohibit mixed-case local-parts.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: IDN (was Did anyone tell Microsoft yet?) 
Date: Thu, 25 Apr 2002 13:26:22 -0400
Sender: [email-masked]

> From the mail POV, there are two separate questions:
> 
> 1. What to do about that header, when passing it as a header to SMTP?

a. encode domain names in IDNA
b. encode local-parts in an encoding that is to be defined, but which
   hopefully is very similar, perhaps identical, to IDNA
c. encode human-readable text in RFC 2047

> 2. What to do about the addr-spec part of it when constructing the
> envelope (i.e. RCPT)?

a. encode domain names in IDNA
b. encode local-parts in an encoding that is to be defined, but which
   hopefully is very similar, perhaps identical, to IDNA

header and envelope encoding of addresses should be the same.

don't worry about UTF-8 or usenet compatibility anytime soon.
it's far more important to minimize disruption in email service
than it is to use utf-8 on the wire, or to try to keep email and 
usenet in sync.

Keith

Date: Thu, 25 Apr 2002 12:55:24 -0400
From: Lawrence Greenfield <[email-masked]>
To: [email-masked], Charles Lindsey <[email-masked]>
Subject: Re: IDN (was Did anyone tell Microsoft yet?)
Sender: [email-masked]

   From: "Charles Lindsey" <[email-masked]>
   Date: Thu, 25 Apr 2002 11:33:46 GMT
[...]
   BUT, whatever we do, please let us do the SAME THING for both 'jürgen' and
   'tu-münchen'.

This is very important to me---the local part and the domain should be
encoded in the same way whenever possible.  I think we'll have
huge implementation problems if we end up with two different
stringprep algorithms and two different encodings for a single e-mail
address.

(I also suspect that the SMTP envelope needs to be encoded the same
way as the RFC 822 header fields but I can believe that this is less
critical.)

Larry

To: [email-masked]
Xref: clerew local.mime:1283
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: IDN (was Did anyone tell Microsoft yet?)
Date: Thu, 25 Apr 2002 11:33:46 GMT
Lines: 104
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>Agreed.  OTOH, such considerations should go into the original 
>design of protocols.  Complex solutions have complex errors 
>leading to interop failures.  If the solution is simple, it is 
>more likely that only simple errors occur.  I'm afraid we'll get 
>our share of complex errors when IDN start to make its way into 
>mail and other infrastructure... The horrors, the horrors.

Maybe, maybe, but it is surely going to happen, so this looks like a good
excuse to ask what the current thinking is.

Currently, the IDNA draft (draft-ietf-idn-idna-07.txt) tells you how to
encode a non-ASCII domain name so that it looks like an ASCII domain name,
which you can then offer to gethostbyname and friends. For sure, people
will want to start using non-ASCII in domain names, and they will want to
use them in local-parts too.

So we can expect to see headers like

To: Jürgen Schmidt <[email-masked]>

(I have stuck to characters displayable in ISO-8859-1; the real customers,
of course, will be the Chinese and the Koreans).

The IDNA recommendation is that applications should display everything in
character sets understandable by the users, and users should not have to
type anything other than those character sets. So the user will see (or
type) exactly the header I showed above. IDNA also recommends keeping
stuff in local character sets as long as possible, only downgrading to
Punycode when about to call gethostbyname.

>From the mail POV, there are two separate questions:

1. What to do about that header, when passing it as a header to SMTP?

2. What to do about the addr-spec part of it when constructing the
envelope (i.e. RCPT)?

The answer to the two questions may or may not be the same.

1. What to do about that header, when passing it as a header to SMTP?

Current wisdom is that 'Jürgen Schmidt' should be encoded using RFC 2047
(but don't try putting it in a quoted-string as '"Jürgen Schmidt"').
'jürgen' and 'tu-münchen' are currently illegal, of course. Possible
future extensions might allow:

A. Extending RFC 2047 for use in those cases.
B. Encoding as per IDNA, giving
	<[email-masked]>
   where 'zz--' is the "ACE prefix" (they have not decided yet exactly
   what it will be).
C. Invent YAEFUTA. I think we want that like we want a hole in the head.
   We already have two such encodings (RFC 2047 and RFC 2231, and some
   people think that is already two too many :-( ).
D. Translate to UTF-8, and allow that in the transport protocol. That is
   the ideal solution, but it is not clear exactly how we get to that
   state from the state we are in. Transition could be painful, but maybe
   the pain is worth it in view of the benefit.

BUT, whatever we do, please let us do the SAME THING for both 'jürgen' and
'tu-münchen'.

2. What to do about the addr-spec part of it when constructing the
envelope (i.e. RCPT)?

Again, the same four solutions are possible, but solution B is making
better sense that in case 1, because the envlope is not (normally) for
human reading, and at least the translation to 'zz--tu-mnchen-t9a' is
going to be needed before the transport can determine the IP address to
send it to; also, it should work immediately with existing transports, so
long as an upgraded gethostbyname is available. (But I would still prefer
a UTF-8 solution).

So, which way do people think we are likely to jump?

Note that my particular concern is with the local-part, because we have a
problem in Usefor when sending mail to moderators. Suppose there were to
be a newsgroup
	dk.test.utf8-æøå.moderated
(the unmoderated group already exists, for experimental purposes). Then
you would submit articles to the moderator at
	[email-masked]

But, of course, you can't do that at the moment. So the present Usefor
draft contains some weasel words to the effect that there is a problem
here, and that its proper solution will have to await the invention of a
suitable extension to the Email protocols. But if somebody can give some
hint as to what that extension might eventually be, I might be able to
write some less-weazelish words.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 25 Apr 2002 09:53:45 -0400
From: Rens Troost <[email-masked]>
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet?
To: Keith Moore <[email-masked]>, Simon Josefsson <[email-masked]>
Cc: Marc Mutz <[email-masked]>, [email-masked]
Sender: [email-masked]

At 08:20 AM 4/25/2002 -0400, Keith Moore wrote:
>at the time.   we could, of course, could have defined
>a clean way to pass arbitrary data in parameters and
>made it part of the original protocol design.  I don't
>think anybody ever raised the idea at the time.

I seem to recall the idea being raised repeatedly
and foundering in a turbulent sea of encoding politics.
I'd go back and look at the archives, but I don't feel
like re-living it :)

-Rens

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet? 
Date: Thu, 25 Apr 2002 09:40:38 -0400
Sender: [email-masked]

> >similarly, I recommend against implementing automatic rfc 2047
> >decoding for parameters in general -  partly because there's a small
> >chance that you'll misinterpret a parameter that way, partly because
> >it conflicts with standard usage, and partly because 2047 is a poor
> >mechanism for this purpose (which is why the standard doesn't use it).
> 
> I think the small chance that he really meant 'file it in a file with the
> name "=?koi8-r?B?Q8/XxdQuZG9j?="' is a resonable risk to take.
> 
> However, I would like to know why 2047 is thought to be a "poor mechanism"
> for this purpose (i.e. why was it not standardized that way - it seems to
> work in a wide variety of situations).

partially because 2047 only tries to solve the problem for human-readable
text - and specifically not for text that needs to be compared with 
anything other text by a computer.  so it doesn't specify anything about
normalization or canonicalization or any of those other issues that
(for instance) the IDN folks have had to worry about.  also, had this been
intended as a general encoding mechanism for paramters, there would have
been a strong incentive to specify a particular character encoding scheme.
but since 2047 was for human-readable text only, it seemed acceptable
to allow any charset to be used.

partially because 2047 imposes strict limitations on the lengths of 
encoded-words, doesn't allow them to appear in quoted strings, and 
contains characters which aren't legal in MIME parameters.  

partially because there are a few things out there that unilaterally
translate all 2047 encodings in a message from one charset to another,
or that decode all 2047 encodings - and 2047 was explicitly designed to
permit the former.  but while it might be reasonable to translate
human-readable text if you know the recipient can read the translated
version, it's not reasonable to translate parameters that aren't 
intended for humans.  

Keith

From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
cc: Keith Moore <[email-masked]>, Marc Mutz <[email-masked]>, [email-masked]
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet? 
Date: Thu, 25 Apr 2002 08:20:59 -0400
Sender: [email-masked]

> OTOH, such considerations should go into the original
> design of protocols.  

of course.  though it's somewhat difficult to see how
we could have dealt with I18N better in MIME, given the
technologies and deployment constraints that existed 
at the time.   we could, of course, could have defined
a clean way to pass arbitrary data in parameters and
made it part of the original protocol design.  I don't 
think anybody ever raised the idea at the time.

Keith

To: [email-masked]
Xref: clerew local.mime:1282
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet?
Date: Thu, 25 Apr 2002 10:18:08 GMT
Lines: 25
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>similarly, I recommend against implementing automatic rfc 2047 
>decoding for parameters in general -  partly because there's a small
>chance that you'll misinterpret a parameter that way, partly because
>it conflicts with standard usage, and partly because 2047 is a poor 
>mechanism for this purpose (which is why the standard doesn't use it).

I think the small chance that he really meant 'file it in a file with the
name "=?koi8-r?B?Q8/XxdQuZG9j?="' is a resonable risk to take.

However, I would like to know why 2047 is thought to be a "poor mechanism"
for this purpose (i.e. why was it not standardized that way - it seems to
work in a wide variety of situations).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: Keith Moore <[email-masked]>
Cc: Marc Mutz <[email-masked]>, [email-masked]
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet?
From: Simon Josefsson <[email-masked]>
Date: Thu, 25 Apr 2002 01:29:40 +0200
Lines: 18
Sender: [email-masked]

Keith Moore <[email-masked]> writes:

>> If you are referring to the QP in MIME headers; yes, add QP 
>> decoding of filename like strings in MIME headers.  It 
>> increases interop alot even though it violates the standard.
>
> the problem with this theory is that increasing local 
> interoperablity may decrease global interoperability.
>
> in short, if implementors accept too much brain damage then we 
> sometimes find that we get stuck with it.

Agreed.  OTOH, such considerations should go into the original 
design of protocols.  Complex solutions have complex errors 
leading to interop failures.  If the solution is simple, it is 
more likely that only simple errors occur.  I'm afraid we'll get 
our share of complex errors when IDN start to make its way into 
mail and other infrastructure... The horrors, the horrors.

From: Keith Moore <[email-masked]>
To: Simon Josefsson <[email-masked]>
cc: Marc Mutz <[email-masked]>, [email-masked]
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet? 
Date: Wed, 24 Apr 2002 18:19:46 -0400
Sender: [email-masked]

> If you are referring to the QP in MIME headers; yes, add QP decoding
> of filename like strings in MIME headers.  It increases interop alot
> even though it violates the standard.  

the problem with this theory is that increasing local interoperablity
may decrease global interoperability.  

in short, if implementors accept too much brain damage then we 
sometimes find that we get stuck with it.

Keith

From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
cc: [email-masked]
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet? 
Date: Wed, 24 Apr 2002 14:02:12 -0400
Sender: [email-masked]

> - From a bug report against _KMail_:
> 
> > Compare two attachment header with smae name and look on difference!
> > Outlook Express:
> > Content-Type: application/msword;
> >       name="=?koi8-r?B?Q8/XxdQuZG9j?="
>  
> Should one work around this?

since there's no name parameter defined for application/msword, 
strictly speaking, you should probably ignore it.

OTOH if you implement a heuristic that says that in the absence of
a content-disposition filename parameter, you'll look for a name 
paramter in the content-type field, then you might also consider
implementing a heuristic that says that if the name paramter is 
rfc 2047 encoded then you will attempt to decode it.  

in any case, you should only treat the filename as a suggestion - 
a default value that the user needs to explicitly confirm.  you
should never use it without asking the user first - and it's only
"safe" to use these heuristics if you follow this rule.

similarly, I recommend against implementing automatic rfc 2047 
decoding for parameters in general -  partly because there's a small
chance that you'll misinterpret a parameter that way, partly because
it conflicts with standard usage, and partly because 2047 is a poor 
mechanism for this purpose (which is why the standard doesn't use it).

Keith

To: Marc Mutz <[email-masked]>
Cc: [email-masked]
Subject: Re: [OT] (or not?) Did anyone tell Microsoft yet?
From: Simon Josefsson <[email-masked]>
Date: Wed, 24 Apr 2002 19:56:26 +0200
Lines: 20
Sender: [email-masked]

Marc Mutz <[email-masked]> writes:

> Hi!
>
> From a bug report against _KMail_:
>
>> Compare two attachment header with smae name and look on difference!
>> Outlook Express:
>> Content-Type: application/msword;
>> 	name="=?koi8-r?B?Q8/XxdQuZG9j?="
>
> Should one work around this?

If you are referring to the QP in MIME headers; yes, add QP decoding
of filename like strings in MIME headers.  It increases interop alot
even though it violates the standard.  You should not generate such
garbage though.

Making KMail send a auto complaint to the originator asking her to ask
their Microsoft support to fix their software is optional though. :)

Date: Wed, 24 Apr 2002 17:40:37 +0200
From: Marc Mutz <[email-masked]>
Subject: [OT] (or not?) Did anyone tell Microsoft yet?
To: [email-masked]
Organization: KDE
Sender: [email-masked]

Hi!

- From a bug report against _KMail_:

> Compare two attachment header with smae name and look on difference!
> Outlook Express:
> Content-Type: application/msword;
> 	name="=?koi8-r?B?Q8/XxdQuZG9j?="

Should one work around this?

Marc

- -- 
Marc Mutz <[email-masked]>

From: Keith Moore <[email-masked]>
To: [email-masked] (Kai Henningsen)
cc: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Fri, 12 Apr 2002 10:22:15 -0400
Sender: [email-masked]

> Personally, I call this automatic decompression "broken".

some browser seem to understand that when you download a file
(as opposed to viewing a file) that you probably want an exact
copy of the original file.  others, unfortunately, don't.

Date: 11 Apr 2002 21:45:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Russ Allbery)  wrote on 09.04.02 in <[email-masked]>:

> Yes, that sounds extremely annoying, and that certainly doesn't happen to
> me.  Apparently it's not as ubiquitous as it should be, but when I visit
> gzipped content on the web, it's decompressed on the fly by the browser
> and then handled as if it weren't compressed.

Something that *I* find bloody annoying. When I download a file named  
bla.tar.gz, and then apply tar tvzf to it, I *don't* want to see "not in  
gzipped format"! And when I then download bla.tar.gz.sign, I'd like to  
verify that the archive really matches the cryptographic checksum, but if  
it's already uncompressed I can't.

Personally, I call this automatic decompression "broken".

MfG Kai

Date: Thu, 11 Apr 2002 18:23:34 +1200 (NZST)
From: [email-masked] (Peter Gutmann)
To: [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

Sam Roberts <[email-masked]> writes:

>S/MIME only uses the signed and enveloped types from CMS, but RFC2630
>specifies authenticated as well. Its not well described there, but it is a
>way of "signing" something using a key agreement algorithm, such that the
>signature can ONLY be verified by using the sender or the receiver's
>private key.

Hmm, are you sure you're not interpreting a bit too much into this
data type?  The intent is to allow MAC'ing of data, and to do that you
need to share a key with the other side, which is why you need a key
exchange to precede it.  One of those happens to be X9.42.  There are
two issues with this:

1. The only implementation which is known to support X9.42 in any
   useful manner is the SFL.
2. There don't seem to be any implementations which support
   authenticatedData, or at least that was the case when I asked on 
   the S/MIME list a while back for someone to do some interop testing 
   with.

(Oh, and the obvious third point: If you're doing a PKC operation for
 the key exchange anyway, you may as well just sign the thing directly.
 At best, you can claim that this gives you a signature capability
 using an encryption-only key).

I've always seen the only real use for this data type as password-
based integrity protection for data you're carrying around, using in
conjunction with passwordRecipientInfo.

>What this gives you is repudiation. If I authenticate something to
>you, you know I authenticated it. But if you forward it somebody
>else, they can't verify the signature, they have no way of knowing
>for sure that I authenticated it, and I can deny it. Even if the
>recipients private key is compromised, they still can't prove
>that *I* authenticated it, since either one of us can create the
>"signature", all they know is one of us made the authenticated data.

Well, only if you use some of the obscure X9.42 options which don't
identify sender and receiver.  The default is to list both, although
there are a pile of optional parameters and with only one known
implementation who knows what will happen in real life.

Peter.

From: Volker Augustin <[email-masked]>
To: [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Date: Thu, 11 Apr 2002 06:58:04 +0200
Cc: [email-masked], [email-masked]
Sender: [email-masked]

Am Thursday 11 April 2002 03:34:03 schrieb Keith Moore 
> > But - for me - replying should only go back to the original sender and
> > not to someone else.
>
> I don't think you'll find much widespread support for that view.
> There are too many real world cases where you want to change the
> set of recipients to whom a reply is sent.

Of course not. The point was only that the name of an action describes
a "general" action contract, i.e. in case of the "Reply To" you expect
that you can reply to a message. You would be badly shaken if you 
could only reply to the sender.
The same should be true for forwarding. The general action contract is
to forward something you received to someone else. It should be up to
the person who is forwarding to decide whether to forward the whole
message or only parts of it.

Volker Augustin

Date: Wed, 10 Apr 2002 22:59:01 -0400
From: Sam Roberts <[email-masked]>
To: [email-masked]
Cc: Paul Shields <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Mail-Followup-To: [email-masked], Paul Shields <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

There actually is a cryptographically sound technique that helps
here, I was talking about it with a cryptographer at work last
week while reading RFC 2630, and not getting the difference
between authenticated and signed data.

S/MIME only uses the signed and enveloped types from CMS, but RFC2630
specifies authenticated as well. Its not well described there, but it is a
way of "signing" something using a key agreement algorithm, such that the
signature can ONLY be verified by using the sender or the receiver's
private key.

Now, since the receiver knows he didn't authenticate it, he knows the
sender must have.

What this gives you is repudiation. If I authenticate something to
you, you know I authenticated it. But if you forward it somebody
else, they can't verify the signature, they have no way of knowing
for sure that I authenticated it, and I can deny it. Even if the
recipients private key is compromised, they still can't prove
that *I* authenticated it, since either one of us can create the
"signature", all they know is one of us made the authenticated data.

I hope my explanation was clear, its easier on a white board!

This CMS authenticated data is only possible with key agreement
algorithms, i.e., you can't do it with RSA. I could be wrong
about it not being a defined payload for S/MIME, but I don't
think so. Too bad. It also has a very odd "feature" or "bug" when you
authenticate the same content to multiple receivers in the CMS message.
I'll explain if anybody is interested.

Sam

Quoteing [email-masked], on Wed, Apr 10, 2002 at 06:06:51PM +0200:
> On 2002-04-10 10:46:26 -0400, Paul Shields wrote:
> 
> >Should we have a selectable option on sign-encrypt that specifies 
> >that the signature must be removed from the plaintext after 
> >verifying it?
> 
> What, precisely, prevents the recipient's implementation from 
> ignoring that flag?
> 
> In the suggestions circulated in this thread, some folks are making 
> the same basic design error all over the place: You want to place 
> trust in software which is under the complete control of an 
> individual you don't trust.  Don't do it.  It's impossible.
> 
> The features which are being suggested _may_ help against 
> unintentional errors the recipient may make (like, storing, by 
> accident, the wrong love letters on the wrong hard drive, which [I 
> seem to recall] was the original reasoning behind the "for your eyes 
> only" [or whatever it was called] function of pgp 2).
> 
> They don't, however, help against malicious behavior on the 
> recipient's side.
> 
> -- 
> Thomas Roessler                        <[email-masked]>

-- 
Sam Roberts <[email-masked]> (Vivez sans temps mort!)

From: Keith Moore <[email-masked]>
To: Volker Augustin <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless 
Date: Wed, 10 Apr 2002 21:34:28 -0400
Sender: [email-masked]

> But - for me - replying should only go back to the original sender and not
> to someone else. 

I don't think you'll find much widespread support for that view.
There are too many real world cases where you want to change the
set of recipients to whom a reply is sent.

From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], [email-masked], Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 10 Apr 2002 21:31:15 -0400
Sender: [email-masked]

> Yes. Google says http://www.rsasecurity.com/news/pr/951226.html and the
> the novelty of the patent appears to be that if you do compression and
> encryption together, you don't need intermediate storage, thereby gaining
> performance.

which is of course BS.  you *always* have intermediate storage, the only
question is the size of the buffer.

Keith

Date: Wed, 10 Apr 2002 20:39:54 +0200
From: Marcel Waldvogel <[email-masked]>
To: Marc Mutz <[email-masked]>
CC: [email-masked], Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Sender: [email-masked]

Marc Mutz wrote:

>Both camps here agree that you can't stop the receiver from sending the 
>formerly encrypted message wherever she wants - in the clear.
>
>Why I am sceptic about allowing forwarding formerly encrypted mails 
>unencryptedly or after re-encryption is that - for me - forwarding shouldn't 
>change the original message. If you want to change the message, reply to it 
>and edit the recipients. If you forward, you actually want to annotate the 
>original message with a few lines of notes, then send the stuff on the the 
>recipient, much like sticking these yellow post-it strips to a folder and 
>write "you do that!" on them before telling the secretary to carry it to the 
>next room.
>

I like bringing up the secretary/post-it metaphor. But assume the letter 
was encrypted, or, in "real-world terms", written in Suaheli, a language 
which nobody else in your office speaks. Putting a "you do that!" 
post-it on the letter won't be useful to your secretary, you first need 
to decrypt (aka "translate") the letter first [*]. This is what I 
believe should be done (after the appropriate warning to the user about 
the consequences).

-Marcel

[*] And this brings us back to the very subject of this e-mail 
conversation :-)

From: Volker Augustin <[email-masked]>
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Date: Wed, 10 Apr 2002 21:56:52 +0200
To: [email-masked]
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> Why I am sceptic about allowing forwarding formerly encrypted mails
> unencryptedly or after re-encryption is that - for me - forwarding
> shouldn't change the original message.

As we already saw different people associate different actions with
 "forward".

> If you want to change the message, reply to it and edit the recipients.

But - for me - replying should only go back to the original sender and not
to someone else. So, from my point of view you should not even be able
to change the recipient field after selecting the reply action. Different
people, different views again.

> If you forward, you actually want to annotate the original message with
> a few lines of notes, then send the stuff on the the recipient, much like
> sticking these yellow post-it strips to a folder and write "you do that!"
> on them before telling the secretary to carry it to the next room.

Actually, in your interpretation you would stick the folder in a box with a
padlock for which you know that the recipient does not have a key, then
tell the secretary to carry it to the next room.

IMO there need to be two different forward actions. Actually we already have
them: Forward and Forward as Attachment. Their interpretation should be
applied only to the content and not the envelope. The encryption is the en-
velope here, but the signature belongs to the content.
Of course you could name them "Forward annotated" and "Forward" instead.

Volker

To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Wed, 10 Apr 2002 12:23:50 -0700
Lines: 69
Sender: [email-masked]

ned <[email-masked]> writes:

> I suggest that these be a gzip-8bit (the difference in overhead between
> this and binary is insignificant, and the benefits of having line
> oriented data are huge)

I assume that by "8bit" here you mean something very similar to yEnc,
namely an encoding that only escapes those characters known to be unsafe
in a nearly binary-clean environment (generally NUL, CR, LF, and an escape
character) and adds CRLF pairs every so often.  If that's what you meant,
I recommend not using the term "8bit", as the MIME definition of the 8bit
CTE is something rather different.

> and gzip-base85 (the EBCDIC concerns that drove the use of a 64
> character alphabet versus an 85 character alphabet seem to be one of the
> few things that are truly no longer a concern for email).

Personally, I really don't see base85 as worth the work.  The amount of
gain over base64 is relatively small, and I doubt there are many
environments that will both implement a brand new encoding and that can't
use something similar to yEnc.

But I wouldn't argue strongly against it.

>> If something like yEnc is introduced as another CTE, we'd have to
>> introduce it twice, once for uncompressed content and once for
>> compressed content... but I'm not sure that's really a problem.

> Why bother? I believe it is easy to generate "uncompressed gzip" if you
> really want to. And I'd rather solve the problem yEnc seeks to solve as
> part of all this...

In other words, your proposal would be to introduce a new yEnc-like
encoding *only* allowing compressed data, requiring all data encoded with
that CTE to be compressed or at least appear to be compressed?

I don't really like that idea, but I'm not sure I can put my finger on
why.  I guess it just seems unclean to "compress" data that can't actually
be compressed just so as to fulfill the requirements of CTE (such as with
a JPEG image or an MP3 file).

>> I do think that we probably at least need to consider the need for two
>> types of compression.  gzip is a great default choice, sitting in a
>> fairly sweet spot between time and space tradeoffs, but bzip2 produces
>> significantly better compression if you're willing to take ~10x as long
>> to compress and I expect that people will start asking for it fairly
>> quickly.

> Humpf. I see the rationale, but I'm really uncomfortable with more than
> one compression.

Another factor worth considering here is sleeper patents, although gzip is
probably safe from that given how long it's been in use.  But compression
is a patent-rich field, so allowing for the possibility of more than one
algorithm may help in making the protocol a bit more law-proof.

> Well, there's a bit of a mess in the case of gzip-8bit and email and
> downgrading, but yes, gzip-base85 or gzip-base64 can both be managed
> IMO.

Aren't these issues essentially identical to the issues posed by RFC 3030
and able to be handled in the same way?  I presume that the yEnc-like
encoding that would be added could be handled by any server advertising
8BITMIME, and mail systems that want to worry about non-8bit-clean transit
paths could recode just like they would for a CTE of 8bit (although in
this case recoding to base64 is the obvious approach).

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

From: Marc Mutz <[email-masked]>
Organization: KDE
To: Marcel Waldvogel <[email-masked]>
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Date: Wed, 10 Apr 2002 21:12:39 +0200
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

On Wednesday 10 April 2002 20:39, Marcel Waldvogel wrote:
<snip>
> I like bringing up the secretary/post-it metaphor. But assume the letter
> was encrypted, or, in "real-world terms", written in Suaheli, a language
> which nobody else in your office speaks. Putting a "you do that!"
> post-it on the letter won't be useful to your secretary,
<snip>

You don't know my boss. He recently forwarded me a Windows Virus infected mail 
because he couldn't print the attachment (a audio/x-wav labelled .doc.pif 
file). Thank God he works under Linux ;-)

Marc

- -- 
Marc Mutz <[email-masked]>

From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
cc: [email-masked], Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless 
Date: Wed, 10 Apr 2002 13:52:00 -0400
Sender: [email-masked]

> Why I am sceptic about allowing forwarding formerly encrypted mails
> unencryptedly or after re-encryption is that - for me - forwarding shouldn't
> change the original message. 

I agree with this much.  A UA that claims to be forwarding a message shouldn't
misrepresent the message's content.  If the subject message is encrypted, the 
corresponding content of the forwarded message should also be encrypted.
It would be fair for the UA to refuse to forward such a message to someone  
who wasn't a recipient - or at least to warn the forwarder that the recipient
probably won't be able to read it.

OTOH if only part of the message is forwarded, say the body or an attachment,
the original encryption has to be removed anyway.  

It seems like the issue here is really one of how the forwarded message is
represented to the ultimate recipient - does it look just like any other
forwarded message, or can the recipient tell that it's been altered?

Have we ever defined exactly what a forwarded message should look like anyway?

Keith

Date: Wed, 10 Apr 2002 13:37:53 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Bart Schaefer wrote:

> (anybody else remember btoa?).

btoa used base85, and I believe is the same base85 encoding that Ned's 
been referring to in his messages to this thread.

 From the btoa man page:

	FEATURES
		It uses a compact base-85 encoding so that
		4 bytes are encoded into 5 characters.
	BUGS
		It uses an obscure base-85 "squoz code" scheme
		to encode 4 bytes into 5 characters.

	Tony Hansen
	[email-masked]

From: Marc Mutz <[email-masked]>
Organization: KDE
To: [email-masked], Keith Moore <[email-masked]>
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Date: Wed, 10 Apr 2002 19:33:17 +0200
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

On Wednesday 10 April 2002 18:10, [email-masked] wrote:
<snip>
> > Nor is it clear that this is a "problem".    At least, there appear to be
> > more "problems" associated with mechanisms that do purport to control
> > what a recipient can do with a message than mechanisms that merely
> > provide protection against interception of a message in transit from
> > sender to recipient.
>
> Exactly. This entire problem space has a character similar to that of copy
> protection, digital rights management and all that. There are also lessons
> to be learned from the Multics work on environments that try to support
> multiple levels of secrecy and integrity simultaneously. (Some of the
> stories told by the folks who worked on things like text editors for mixed
> level data are particularly amusing.)
<snip>
> You can't win. And given how much this pisses off users, making them even
> less receptive to following whatever rules they're supposed to be
> following, you shouldn't even try.
<snip>

Actually, we already knew this ;-)
Both camps here agree that you can't stop the receiver from sending the 
formerly encrypted message wherever she wants - in the clear.

Why I am sceptic about allowing forwarding formerly encrypted mails 
unencryptedly or after re-encryption is that - for me - forwarding shouldn't 
change the original message. If you want to change the message, reply to it 
and edit the recipients. If you forward, you actually want to annotate the 
original message with a few lines of notes, then send the stuff on the the 
recipient, much like sticking these yellow post-it strips to a folder and 
write "you do that!" on them before telling the secretary to carry it to the 
next room.

Marc

- -- 
Marc Mutz <[email-masked]>

Date: Wed, 10 Apr 2002 13:16:28 -0400
From: Paul Shields <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Sender: [email-masked]

Derek wants this to be enforced or not to have it at all.
Ned points out that users will be angry if you try to enforce it.

All I wanted was friendly handling instructions, a convenience
feature just like 'eyes-only'.  It won't try to prohibit
forwarding, it is simply a request that the receiver remove the
signature after decrypt+verify.

Yes, Derek will be able to get around it unless someone figures out
how to tie the signature to the encryption. Yes, it
won't be supported by some implementations. But does it make matters
worse than they are now?  Is it something that
has value ONLY if it is enforceable?

- --Paul.

Derek Atkins wrote:

  Paul Shields <[email-masked]> writes:

  > Should we have a selectable option on sign-encrypt that specifies
  > that the signature must be
  > removed from the plaintext after verifying it?

  How would you enforce this?  This is just like the
"for-her-eyes-only"
  flag on literal text.  It's a notation to keep the good guy honest,
  but wont protect you from someone who really wants to get around
it.

[email-masked] wrote:

  All of this stuff falls down at some point because you just can't
  compartmentalize people's brains. Prohibit forwarding of content
and people
  will save stuff to files and then mail the files. Prohibit saving
to files and
  people will cut and paste. Prohibit cut and paste and people will
type what one
  window says into another. Prohibit people from having mutiple
windows up at the
  same time for this purpose and they will write stuff down on paper
(which
  introduces its own set of problems). Prohibit writing stuff down
(good luck)
  and people will try and remember it, and then enter it incorrectly
later.

  You can't win. And given how much this pisses off users, making
them even less
  receptive to following whatever rules they're supposed to be
following, you
  shouldn't even try.

                                  Ned

From: "Bart Schaefer" <[email-masked]>
Date: Wed, 10 Apr 2002 17:10:40 +0000
Comments: In reply to [email-masked] "Re: Content-Transfer-Encoding and yEnc" (Apr  9, 11:51pm)
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

On Apr 9, 11:51pm, [email-masked] wrote:
}
} > following CTEs:
} 
} >     binary
} >     7bit
} >     8bit
} >     quoted-printable
} >     base64
} 
} >     gzip (binary)
} >     gzip+base64
} 
} [...] I agree this is the right idea.
} 
} > I do think that we probably at least need to consider the need for two
} > types of compression.
} 
} Humpf. I see the rationale, but I'm really uncomfortable with more than
} one compression.

I seem to recall that this all came up long ago, before MIME was even a
proposed standard, and was rejected precisely because of the combinatoric
problem:  Every time there's a new encoding or new compression scheme, you
end up having to support every combination of compression and encoding.

Sun tried to solve this years ago in Mailtool with the X-Sun-Encoding-Info
header, which was an unordered list of transformations that had been applied
to the body part, e.g., "tar, compress, btoa" (anybody else remember btoa?).
The receiving software had to know which keywords meant transport encoding,
compression, archiving, etc. and undo them in the "right" order no matter
what the order in the header field.  This was an interoperability disaster.

As far as I can tell, none of the reasons why compression was excluded
from MIME in the first place have become any less valid over the years.

-- 
Bart Schaefer                                                   Zanshin
http://www.well.com/user/barts                   http://www.zanshin.com

Date: Wed, 10 Apr 2002 18:48:00 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Cc: [email-masked], Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

[email-masked]
> Good point, although I've heard that there's a patent covering the combining
> of compression and encryption. What's the story?

Yes. Google says http://www.rsasecurity.com/news/pr/951226.html and the
the novelty of the patent appears to be that if you do compression and
encryption together, you don't need intermediate storage, thereby gaining
performance.

Grrr.

--Arnt

Date: Wed, 10 Apr 2002 09:37:27 -0700 (PDT)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], Russ Allbery <[email-masked]>, [email-masked]
Sender: [email-masked]

> [email-masked]
> > Indeed, one can argue that compression should be negotiated as an SMTP
> > (or whatever protocol) extension, and should not enter in to the message body
> > format at all. The argument against that is encrypted messages, where the
> > compression has to be done as part of the encryption operation to be
> > effective. However, now that S/MIME has (or is getting) a compression
> > layer added, maybe this solves the compression of encrypted content
> > problem and leaves us free to address the compression of unencrypted
> > content at the protocol level.

> TLS does compression, so RFC2595 would have the side effect of providing
> compressed SMTP, provided implementations negotiate something better than
> CompressionMethod.null.

Good point, although I've heard that there's a patent covering the combining
of compression and encryption. What's the story?

				Ned

Date: Wed, 10 Apr 2002 18:36:41 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Cc: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

[email-masked]
> Indeed, one can argue that compression should be negotiated as an SMTP
> (or whatever protocol) extension, and should not enter in to the message body
> format at all. The argument against that is encrypted messages, where the
> compression has to be done as part of the encryption operation to be
> effective. However, now that S/MIME has (or is getting) a compression
> layer added, maybe this solves the compression of encrypted content 
> problem and leaves us free to address the compression of unencrypted
> content at the protocol level.

TLS does compression, so RFC2595 would have the side effect of providing
compressed SMTP, provided implementations negotiate something better than
CompressionMethod.null.

--Arnt

Date: Wed, 10 Apr 2002 18:06:51 +0200
From: Thomas Roessler <[email-masked]>
To: Paul Shields <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Mail-Followup-To: Paul Shields <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

On 2002-04-10 10:46:26 -0400, Paul Shields wrote:

>Should we have a selectable option on sign-encrypt that specifies 
>that the signature must be removed from the plaintext after 
>verifying it?

What, precisely, prevents the recipient's implementation from 
ignoring that flag?

In the suggestions circulated in this thread, some folks are making 
the same basic design error all over the place: You want to place 
trust in software which is under the complete control of an 
individual you don't trust.  Don't do it.  It's impossible.

The features which are being suggested _may_ help against 
unintentional errors the recipient may make (like, storing, by 
accident, the wrong love letters on the wrong hard drive, which [I 
seem to recall] was the original reasoning behind the "for your eyes 
only" [or whatever it was called] function of pgp 2).

They don't, however, help against malicious behavior on the 
recipient's side.

-- 
Thomas Roessler                        <[email-masked]>

Date: Wed, 10 Apr 2002 09:10:21 -0700 (PDT)
From: [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
To: Keith Moore <[email-masked]>
Cc: Marc Mutz <[email-masked]>, [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> > What if, in a mail user agent, the user wants to forward an encrypted message?
> > Allow it? Deny it?
> > Re-encrypt or remove encryption?
> > The problem is, of course, that the original sender might not like his
> > encrypted text being sent out in the clear again...

> I don't think this is something we can control.  If you encrypt something with
> a recipient's public key, you're implicitly giving the recipient the ability
> to decrypt that message and redistribute it to his heart's content.  You can
> issue instructions to the recipient about redistribution of the messgae, but
> you can't control whether the recipient follows those instructions.

> Nor is it clear that this is a "problem".    At least, there appear to be
> more "problems" associated with mechanisms that do purport to control what
> a recipient can do with a message than mechanisms that merely provide
> protection against interception of a message in transit from sender to
> recipient.

Exactly. This entire problem space has a character similar to that of copy
protection, digital rights management and all that. There are also lessons to
be learned from the Multics work on environments that try to support multiple
levels of secrecy and integrity simultaneously. (Some of the stories told by
the folks who worked on things like text editors for mixed level data are
particularly amusing.)

All of this stuff falls down at some point because you just can't
compartmentalize people's brains. Prohibit forwarding of content and people
will save stuff to files and then mail the files. Prohibit saving to files and
people will cut and paste. Prohibit cut and paste and people will type what one
window says into another. Prohibit people from having mutiple windows up at the
same time for this purpose and they will write stuff down on paper (which
introduces its own set of problems). Prohibit writing stuff down (good luck)
and people will try and remember it, and then enter it incorrectly later.

You can't win. And given how much this pisses off users, making them even less
receptive to following whatever rules they're supposed to be following, you
shouldn't even try.

				Ned

Date: Tue, 09 Apr 2002 23:51:54 -0700 (PDT)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> > They do? It must be really transparent then, because I myself keep
> > downoading zipped .pdfs and gzipped postscript, and either I save it to
> > a file (on Linux) and deal with it myself, or IE opens the file with
> > WinZIP, and then I double-click it in WinZIP, which opens acrobat, for
> > example. Pretty much exactly what I do with email, and its annoying in
> > both cases.

> Yes, that sounds extremely annoying, and that certainly doesn't happen to
> me.  Apparently it's not as ubiquitous as it should be, but when I visit
> gzipped content on the web, it's decompressed on the fly by the browser
> and then handled as if it weren't compressed.

The HTTP profile of MIME is very different from that of email. In HTTP the
choice was made to do compression of objects as a whole, not on a leaf by leaf
basis. A content-encoding header was defined for this purpose, and it supports
gzip (as well as other stuff).

And since HTTP is single hop it is possible to negotiate encodings
and avoid sending something the receiver doesn't understand.

> A collection of data is a different type of object, one that's not really
> dealt with by any of the discussion so far on this topic.  I think the
> closest MIME currently gets to this is the multipart/apple-double (or
> whatever the correct MIME type is) that handles two-part Mac files.  I
> think that treating an archive file like a tar file or a ZIP file that
> contains multiple files (as opposed to ZIP just used as a compression
> algorithm) as an application in its own right.  It's essentially a portion
> of a file system, not an individual file.

Quite true. An archive possess additional properties. I note in passing that
the trick of using archives for email has been tried in the past: The NeXT used
compressed tar files. It worked very poorly in practice, in part because the
archive's labelling capabilities weren't at all aligned with the needs of
message parts. There was a time when I got complaints about such messages
fairly regularly, but I haven't seen one in quite a while at this point.

> The gzip+base64 situation is a complicated one, and I'm not sure the right
> way of handling it.  Intuitively, it seems to me like we need at least the
> following CTEs:

>     binary
>     7bit
>     8bit
>     quoted-printable
>     base64

>     gzip (binary)
>     gzip+base64

I suggest that these be a gzip-8bit (the difference in overhead between this
and binary is insignificant, and the benefits of having line oriented data are
huge) and gzip-base85 (the EBCDIC concerns that drove the use of a 64 character
alphabet versus an 85 character alphabet seem to be one of the few things that
are truly no longer a concern for email). But I agree this is the right idea.

> I'm not sure that we really need any kind of full layering, though, where
> you can stack arbitrary encodings.  gzip+8bit makes no sense, for example,
> and gzip+quoted-printable isn't going to win much in any situation over
> gzip+base64.  If something like yEnc is introduced as another CTE, we'd
> have to introduce it twice, once for uncompressed content and once for
> compressed content... but I'm not sure that's really a problem.

Why bother? I believe it is easy to generate "uncompressed gzip" if you
really want to. And I'd rather solve the problem yEnc seeks to solve
as part of all this...

> I do think that we probably at least need to consider the need for two
> types of compression.  gzip is a great default choice, sitting in a fairly
> sweet spot between time and space tradeoffs, but bzip2 produces
> significantly better compression if you're willing to take ~10x as long to
> compress and I expect that people will start asking for it fairly quickly.

Humpf. I see the rationale, but I'm really uncomfortable with more than
one compression.

> Well, the compression and encoding part of this still, to me, feels like a
> transport layer that's clearly separable from the type of the content.
> Archives (miniature file systems) are a more complex issue that I think
> MIME will likely have to punt on and just assign content types to, because
> you can't really do anything with them other than bring up a file system
> browser that can launch applications on individual contents.

Absolutely. There's a place for these things. It just isn't in doing
general messaging.

> > I agree CTE seems a great place to put transport-layer
> > compression. But...  you can't do it without apps that don't understand
> > the new CTEs (all of them, right now) not decoding the content, and in
> > addition they will treat it as application/octet-stream.

> > Isn't that what will happen? It just seems to me there is no migration
> > path. Am I missing something?

> There's a migration path (you save the file and then decompress it) which
> actually looks a little bit like what we have now, but yes, that part is
> the only part that to me seems to leave something to be desired.  On the
> other hand, is there any standardized way of handling compressed files
> now?  (Are content types for gzip even registered?  I didn't believe that
> there were.)  So from a standardization perspective, or even from a
> deployed software perspective, what exactly are we transitioning from?
> What do we have to have legacy support for?  Who are the current users and
> what are they doing?

The rules say unknown encodings have to be presented as an series of octets of
unknown types. The rules also say you're supposed to offer to save such stuff
to a file. So you have the same options of dealing with the stuff manually.

> If what people are currently doing is using content types that are treated
> as pretty much equivalent to application/octet-stream now, then the
> transition isn't much of an issue.

Well, there's a bit of a mess in the case of gzip-8bit and email and
downgrading, but yes, gzip-base85 or gzip-base64 can both be managed IMO.

				Ned

Date: Tue, 09 Apr 2002 23:46:59 -0700 (PDT)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Compression is fundamentally a transport issue, not an application issue,
> and as such should be added or removed as part of the transport layer, not
> as part of the application dispatch layer.  The Content-Transfer-Encoding
> vs. Content-Type distinction already encapsulates that, and shouldn't be
> abandoned.

Indeed, one can argue that compression should be negotiated as an SMTP
(or whatever protocol) extension, and should not enter in to the message body
format at all. The argument against that is encrypted messages, where the
compression has to be done as part of the encryption operation to be
effective. However, now that S/MIME has (or is getting) a compression
layer added, maybe this solves the compression of encrypted content 
problem and leaves us free to address the compression of unencrypted
content at the protocol level.

				Ned

To: Paul Shields <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
From: Derek Atkins <[email-masked]>
Date: 10 Apr 2002 11:51:38 -0400
Lines: 25
Sender: [email-masked]

Paul Shields <[email-masked]> writes:

> Should we have a selectable option on sign-encrypt that specifies
> that the signature must be
> removed from the plaintext after verifying it?

How would you enforce this?  This is just like the "for-her-eyes-only"
flag on literal text.  It's a notation to keep the good guy honest,
but wont protect you from someone who really wants to get around it.

For example, many a time I've used 'pgp -fm input.asc >& output.txt'
to get around the for-her-eyes-only "bug".

The only way to really enforce this is to mathematically tie the
signature to the encryption.  This would require a whole new line of
mathematics (assuming you want to continue to hide the sender's
identity to non-recipients).

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       [email-masked]                        PGP key available

Date: Wed, 10 Apr 2002 10:46:26 -0400
From: Paul Shields <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Sender: [email-masked]

Without handling instructions that cause the signature to be removed,
someone forwarding the
message could cause trouble for the original sender.

I raised this as an issue a few years back on the PGP MIME list, but
the folks there were
unconvinced. It seems they wanted to be able to keep the signature
after decryption.

Should we have a selectable option on sign-encrypt that specifies
that the signature must be
removed from the plaintext after verifying it?

- --
Paul Shields

[email-masked] wrote:

  From: John Dlugosz

  IMO,

  If forwarding the decrypted plaintext also removed the signature,
there
  would be less trouble.  The content could be reputiated, since it
can't be
  distinguised from the forwarder just making it up.

  But I think PGP uses "sign, then encrypt" which means software
could
  decrypt but leave the signature intact.  As I recall, this was
thought to
  be a non-problem with respect to re-targeting, because you can put
the
  recipient's name in the message at the application level.  e.g.
"Dear Sue,"
  is part of the message, so it can't be mistaken as a message to
Joan.  But
  that doesn't handle the issue of private information in the
message.  It's
  possible for Sue to reveal the signed message to someone else, who
can
  verify the signature, without needing Sue's key.  I would prefer if
that
  were impossible--without Sue's key, the message can't be
authenticated
  either.

From: Keith Moore <[email-masked]>
To: Sam Roberts <[email-masked]>
cc: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 10 Apr 2002 09:59:21 -0400
Sender: [email-masked]

> I agree CTE seems a great place to put transport-layer compression. But...
> you can't do it without apps that don't understand the new CTEs (all of them,
> right now) not decoding the content, and in addition they will treat it
> as application/octet-stream.
> 
> Isn't that what will happen? It just seems to me there is no migration
> path. Am I missing something?

yes, but at least for now, we're talking about usenet - and usefor is already
significantly revising the format for usenet messages.  usenet UAs will need
to be upgraded to support the new encodings.  usenet users will probably 
continue to explicitly send split-up .zip files, .gz files or whatever until 
UAs widely support those encodings, but the UAs will eventually do so.

for those that don't it should still be possible to save the message and
feed it to an external decoder.

Keith

From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless 
Date: Wed, 10 Apr 2002 09:50:50 -0400
Sender: [email-masked]

> What if, in a mail user agent, the user wants to forward an encrypted message?
> Allow it? Deny it?
> Re-encrypt or remove encryption?
> The problem is, of course, that the original sender might not like his
> encrypted text being sent out in the clear again...

I don't think this is something we can control.  If you encrypt something with
a recipient's public key, you're implicitly giving the recipient the ability
to decrypt that message and redistribute it to his heart's content.  You can
issue instructions to the recipient about redistribution of the messgae, but 
you can't control whether the recipient follows those instructions.

Nor is it clear that this is a "problem".    At least, there appear to be
more "problems" associated with mechanisms that do purport to control what
a recipient can do with a message than mechanisms that merely provide 
protection against interception of a message in transit from sender to
recipient.

Keith

Date: Wed, 10 Apr 2002 10:19:19 +0100
To: Jon Callas <[email-masked]>
From: Graham Klyne <[email-masked]>
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Cc: [email-masked], [email-masked]
Sender: [email-masked]

At 02:27 PM 4/9/02 -0700, Jon Callas wrote:
>A trusted channel to an untrustworthy person is not secure. And to
>paraphrase Laotse, you cannot create trust with cryptography, no matter how
>much cryptography you use.

Excellent!!

This is the first time I've been aware of a cryptosystems person make this 
point.

#g

-------------------
Graham Klyne
<[email-masked]>

Date: Tue, 09 Apr 2002 23:00:00 -0500
From: "Eric A. Hall" <[email-masked]>
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Russ Allbery wrote:

> The gzip+base64 situation is a complicated one, and I'm not sure the
> right way of handling it.

Recursive encoding of an entity. Self-defeating but there you have it.

> Archives (miniature file systems) are a more complex issue that I think
> MIME will likely have to punt on and just assign content types to,
> because you can't really do anything with them other than bring up a
> file system browser that can launch applications on individual contents.

Yeah, and application/tar (or an equivalent) is probably the best way to
do that in the layers here

Agree with everything else Russ said

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 09 Apr 2002 19:55:59 -0700
Lines: 111
Sender: [email-masked]

Sam Roberts <[email-masked]> writes:

> They do? It must be really transparent then, because I myself keep
> downoading zipped .pdfs and gzipped postscript, and either I save it to
> a file (on Linux) and deal with it myself, or IE opens the file with
> WinZIP, and then I double-click it in WinZIP, which opens acrobat, for
> example. Pretty much exactly what I do with email, and its annoying in
> both cases.

Yes, that sounds extremely annoying, and that certainly doesn't happen to
me.  Apparently it's not as ubiquitous as it should be, but when I visit
gzipped content on the web, it's decompressed on the fly by the browser
and then handled as if it weren't compressed.

> I agree it should be transparent. An "encapsulates" parameter to
> content-type would allow this, and downgrade gracefully, TO THE CURRENT
> SITUATION, as far as I can tell.

That is the problem with the Content-Transfer-Encoding, namely that the
downgrade situation isn't as nice.  That means that encapsulation makes
for an easier transition, but using a CTE makes for a much nicer long-term
situation.

> You could look at it like that. Or you could say people take things
> mpegs, and tar them up.

A collection of data is a different type of object, one that's not really
dealt with by any of the discussion so far on this topic.  I think the
closest MIME currently gets to this is the multipart/apple-double (or
whatever the correct MIME type is) that handles two-part Mac files.  I
think that treating an archive file like a tar file or a ZIP file that
contains multiple files (as opposed to ZIP just used as a compression
algorithm) as an application in its own right.  It's essentially a portion
of a file system, not an individual file.

The situation is clearer with gzip than with ZIP, since gzip uses a single
file model and doesn't combine the issues of compression and archiving.

> Then they take the tar file, and they gzip it. Then it gets base64
> encoded by the mail agent.

The gzip+base64 situation is a complicated one, and I'm not sure the right
way of handling it.  Intuitively, it seems to me like we need at least the
following CTEs:

    binary
    7bit
    8bit
    quoted-printable
    base64

    gzip (binary)
    gzip+base64

I'm not sure that we really need any kind of full layering, though, where
you can stack arbitrary encodings.  gzip+8bit makes no sense, for example,
and gzip+quoted-printable isn't going to win much in any situation over
gzip+base64.  If something like yEnc is introduced as another CTE, we'd
have to introduce it twice, once for uncompressed content and once for
compressed content... but I'm not sure that's really a problem.

I do think that we probably at least need to consider the need for two
types of compression.  gzip is a great default choice, sitting in a fairly
sweet spot between time and space tradeoffs, but bzip2 produces
significantly better compression if you're willing to take ~10x as long to
compress and I expect that people will start asking for it fairly quickly.

I see no reason to include ZIP in the list of encodings.  It doesn't seem
to offer, in the area of compression, any interesting benefits that
neither gzip nor bzip2 already serve, and it's not a stream-oriented
format like gzip and bzip2 both are which makes it much harder to handle
internally in the software that undoes CTEs.  (It's also not standardized
in an RFC so far as I know, and in fact bzip2 encoding should probably
wait for standardization of that format as well.  gzip is documented in
RFC 1952, but should probably have a standards-track specification prior
to a standards-track RFC that specifies a gzip CTE.)

> The content-transfer-encoding/content-type pair is just 2 deep. It
> doesn't look like it can represent this.

Well, the compression and encoding part of this still, to me, feels like a
transport layer that's clearly separable from the type of the content.
Archives (miniature file systems) are a more complex issue that I think
MIME will likely have to punt on and just assign content types to, because
you can't really do anything with them other than bring up a file system
browser that can launch applications on individual contents.

> I agree CTE seems a great place to put transport-layer
> compression. But...  you can't do it without apps that don't understand
> the new CTEs (all of them, right now) not decoding the content, and in
> addition they will treat it as application/octet-stream.

> Isn't that what will happen? It just seems to me there is no migration
> path. Am I missing something?

There's a migration path (you save the file and then decompress it) which
actually looks a little bit like what we have now, but yes, that part is
the only part that to me seems to leave something to be desired.  On the
other hand, is there any standardized way of handling compressed files
now?  (Are content types for gzip even registered?  I didn't believe that
there were.)  So from a standardization perspective, or even from a
deployed software perspective, what exactly are we transitioning from?
What do we have to have legacy support for?  Who are the current users and
what are they doing?

If what people are currently doing is using content types that are treated
as pretty much equivalent to application/octet-stream now, then the
transition isn't much of an issue.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 09 Apr 2002 21:20:56 -0500
From: "Eric A. Hall" <[email-masked]>
To: Sam Roberts <[email-masked]>
CC: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Sam Roberts wrote:

> surely you aren't claiming that people don't actually send zipped
> pdfs with a content-type of:
> 
> content-type: application/x-zip-compressed

You need to separate the notion of ~"working with zip files" from the
notion of transferring compressed data. If you want to keep doing things
the old way, send application/zip and use CTE of whatever. Trying to store
zip "files" inside of MIME entities is forcing a collision of operational
metaphors which must be resolved.

Do we need a multipart/container object of some kind, where the first
entity is a table of contents to the rest of the entities (thereby making
it act like a traditional zip file, but using MIME as storage), or should
it be a single entity with an embedded TOC (using the current tar or zip
or whatever mechanism)? Do we need a new disposition method for "expand"
to go along with a single entity?

A couple of things we know: (1) the application/zip model makes the actual
content opaque, and (2) sending 100s of files as individual members of a
multipart entity (as would be the case with large tarball packages mapped
directly to MIME-as-storage) won't work very well.

Personally I am wondering if the real solution here isn't another major
media-type family which doesn't have the "application" prohibitions, and
which allows MIME agents to directly decode the data. Sounds like
multipart, except that old systems will treat multipart/unknown as
multipart/mixed, showing you the hundreds of files from that tarball as
discrete attachments. Hard problem.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 9 Apr 2002 22:18:32 -0400
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Tue, Apr 09, 2002 at 06:01:47PM -0700:
> Tony Hansen <[email-masked]> writes:
> 
> > Keith, I usually agree with you, but not in this case. This suggested
> > content-type DOES describe the content type: it IS a zip-compressed file
> > that's enclosed there. It just happens that you also know what was
> > compressed.
> 
> But no one cares that it's ZIP-compressed except the mail system (in the
> broader sense that incorporates the automatic actions of the user's MUA).
> The user certainly doesn't unless they're an expert.
> 
> Users, and by this I mean average mail users, not sophisticated e-mail
> experts like the people on this mailing list, expect compression to be a
> transparent process (if they're even aware that it exists and is
> possible).  Having to deal with compression confuses them, and web
> browsers just automatically decompress things because of this.  That's

They do? It must be really transparent then, because I myself keep
downoading zipped .pdfs and gzipped postscript, and either I save
it to a file (on Linux) and deal with it myself, or IE opens the
file with WinZIP, and then I double-click it in WinZIP, which opens
acrobat, for example. Pretty much exactly what I do with email, and
its annoying in both cases.

> what they're used to, and that view of things makes sense.  It shouldn't
> *matter* that something is compressed.  We compress files to reduce the
> transit time, not because it's somehow important that a particular file be
> compressed.

I agree it should be transparent. An "encapsulates" parameter to
content-type would allow this, and downgrade gracefully, TO THE CURRENT
SITUATION, as far as I can tell.

> In order to have end-to-end transparency of compression, where you can do
> compression or not do compression as fits the particular situation and
> have the content handled the same regardless, I think Keith and Ned are
> right and you *have* to mark the real content in the Content-Type header.
> Content is dispatched to applications on the basis of the Content-Type.
> With the sort of Content-Type that's been proposed here, the application
> is going to uncompress the message, pull the *correct* content type out of
> the parameter, and then act as if that was the Content-Type anyway for the
> purpose of dispatch, which argues that one is shoehorning things into the
> wrong model.  Any protocol is described with a step that goes something
> like "now pretend that this header field *actually* said..." is, IMO,
> probably misdesigned.

You could look at it like that. Or you could say people take things mpegs,
and tar them up. Then they take the tar file, and they gzip it. Then it gets
base64 encoded by the mail agent. The content-type should reflect the real
nature of the content, and allow it to be generated and manipulated
automatically, by tools that wish to. It looks to me like the kind of
recursion you get with nested multiparts.

The content-transfer-encoding/content-type pair is just 2 deep. It doesn't
look like it can represent this.

Anyhow, a few people seem to think this is the wrong way, but only one
person has suggested a CTE that would do compression, and that idea was
shot down as being hopelessly complicated (the b64+zip-... approach).

> Compression is fundamentally a transport issue, not an application issue,
> and as such should be added or removed as part of the transport layer, not
> as part of the application dispatch layer.  The Content-Transfer-Encoding
> vs. Content-Type distinction already encapsulates that, and shouldn't be
> abandoned.

I agree CTE seems a great place to put transport-layer compression. But...
you can't do it without apps that don't understand the new CTEs (all of them,
right now) not decoding the content, and in addition they will treat it
as application/octet-stream.

Isn't that what will happen? It just seems to me there is no migration
path. Am I missing something?

The one thing may folks seem to agee on is that there is a problem,
so if having the content-type describe how content of one type was
used to wrap content of another isn't right, what would the CTEs look
like? What's the other approach?

Sam

-- 
Sam Roberts <[email-masked]> (Vivez sans temps mort!)

To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Tue, 09 Apr 2002 18:01:47 -0700
Lines: 42
Sender: [email-masked]

Tony Hansen <[email-masked]> writes:

> Keith, I usually agree with you, but not in this case. This suggested
> content-type DOES describe the content type: it IS a zip-compressed file
> that's enclosed there. It just happens that you also know what was
> compressed.

But no one cares that it's ZIP-compressed except the mail system (in the
broader sense that incorporates the automatic actions of the user's MUA).
The user certainly doesn't unless they're an expert.

Users, and by this I mean average mail users, not sophisticated e-mail
experts like the people on this mailing list, expect compression to be a
transparent process (if they're even aware that it exists and is
possible).  Having to deal with compression confuses them, and web
browsers just automatically decompress things because of this.  That's
what they're used to, and that view of things makes sense.  It shouldn't
*matter* that something is compressed.  We compress files to reduce the
transit time, not because it's somehow important that a particular file be
compressed.

In order to have end-to-end transparency of compression, where you can do
compression or not do compression as fits the particular situation and
have the content handled the same regardless, I think Keith and Ned are
right and you *have* to mark the real content in the Content-Type header.
Content is dispatched to applications on the basis of the Content-Type.
With the sort of Content-Type that's been proposed here, the application
is going to uncompress the message, pull the *correct* content type out of
the parameter, and then act as if that was the Content-Type anyway for the
purpose of dispatch, which argues that one is shoehorning things into the
wrong model.  Any protocol is described with a step that goes something
like "now pretend that this header field *actually* said..." is, IMO,
probably misdesigned.

Compression is fundamentally a transport issue, not an application issue,
and as such should be added or removed as part of the transport layer, not
as part of the application dispatch layer.  The Content-Transfer-Encoding
vs. Content-Type distinction already encapsulates that, and shouldn't be
abandoned.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Tue, 09 Apr 2002 15:00:11 -0700 (PDT)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Keith Moore <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> > So, I suggested:
> >
> > content-type: application/x-zip-compressed; encapsulates=application/pdf

> sorry, this is just unacceptable.  there are too many things (e.g. mail
> filters) that expect the content-type to actually describe the content.

And even if you get past those expectations, the multipart/signed experience
shows that any expectation that a implementation-defined handler for a type
isn't likely to be able to turn around and feed additional parts back into
the agent.

We have a well-defined structure. We need to stick to it.

				Ned

Date: Tue, 9 Apr 2002 14:27:04 -0700
To: [email-masked], [email-masked]
From: Jon Callas <[email-masked]>
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Cc: [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

At 3:33 PM -0500 4/9/02, [email-masked] wrote:

>But I think PGP uses "sign, then encrypt" which means software could
>decrypt but leave the signature intact.  As I recall, this was thought to
>be a non-problem with respect to re-targeting, because you can put the
>recipient's name in the message at the application level.  e.g. "Dear Sue,"
>is part of the message, so it can't be mistaken as a message to Joan.  But
>that doesn't handle the issue of private information in the message.  It's
>possible for Sue to reveal the signed message to someone else, who can
>verify the signature, without needing Sue's key.  I would prefer if that
>were impossible--without Sue's key, the message can't be authenticated
>either.

OpenPGP specifies that the signature goes inside the envelope. The reason
for this is that if the signature is outside the envelope, then the
attacker knows both the source and destination of the message. In short, it
provides cryptographically enhanced traffic analysis.

Yes, you're right, someone can decrypt a signed message and then send on
the signed, decrypted message. They can also take a screenshot of the
screen and send the image on to someone else. I know many people who do not
sign messages for this very reason. We added a non-signature integrity
check to OpenPGP messages so you'd have some way of knowing you got the
message intact without requiring the sender to sign it.

There are security risks to signing messages. A good friend of mine is of
the opinion that you should never sign anything that you aren't willing to
forward on to City Hall for inclusion in the public record. He never signed
anyone ele's key for this reason, let alone a message. I'll add that this
guy was an VP at an investment bank, not a cypherpunk with outlaw fantasies.

I'm not quite that extreme, but I don't sign anything that I wouldn't sign
if it were on paper, not on bits.

Mathematics cannot solve real-world security problems. If you're confessing
secrets to someone who gossips, you have security problems. If you send
signed, encrypted messages to someone who will decrypt them and send them
(or send a screenshot of your messagae) on, then you have the same security
problem.

A trusted channel to an untrustworthy person is not secure. And to
paraphrase Laotse, you cannot create trust with cryptography, no matter how
much cryptography you use.

	Jon

Date: Tue, 9 Apr 2002 16:08:03 -0500
From: "Clinton E. Troutman" <[email-masked]>
Reply-To: "Clinton E. Troutman" <[email-masked]>
Organization: CeTro
To: Simon Josefsson <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re[2]: Bug#40394: forwarding an encrypted PGP message is useless
Sender: [email-masked]

- --- Reply created Tuesday, 09 April 2002, 15:59:49 ---

- ------- Original Message --------
On Tuesday, 09 April 2002, 14:32, you wrote:

SJ> Then the original sender should not send the text to someone who
will
SJ> do that.

SJ> I don't see how the standard could prevent the user from doing
SJ> this. If it is prevented, then it is only the applications'
doing, so
SJ> it wouldn't be difficult to override it.  This isn't a technical
SJ> problem. Whenever you introduce encryption between two parties
you
SJ> have this issue.

SJ> Whether to re-encrypt or remove encryption is up to the user
SJ> forwarding the message.  What the defaults should be probably
depend
SJ> too much on the application to give any generic recomendation on.

SJ> Just my two cents.

- ------- End Original Message --------

Absolutely agreed.

Not a technical problem.

It is a problem of data released "to the wild" and what another party
may "do" with that data once released.

The allowable disposition of the data beyond the party of first
release should be specifically stated to prevent "missteps" by that
party.

And, if you are concerned that the data *not* be released beyond that
first party, perhaps you should consider either:
- - another method of release, or
- - your relationship with the party to whom you are releasing the
data,
or
- - don't release that data in the first place.

(Interesting how this seems to parallel the current debates on
copyright...)

- -----
Clinton E. Troutman
mailto:[email-masked]
- -----
"You can tell the Mafia is running the cockfight if the duck wins." -
Anon.
ÀFÿ\„$%

Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
From: [email-masked]
Date: Tue, 9 Apr 2002 15:33:34 -0500
Sender: [email-masked]

From: John Dlugosz

IMO,

If forwarding the decrypted plaintext also removed the signature, there
would be less trouble.  The content could be reputiated, since it can't be
distinguised from the forwarder just making it up.

But I think PGP uses "sign, then encrypt" which means software could
decrypt but leave the signature intact.  As I recall, this was thought to
be a non-problem with respect to re-targeting, because you can put the
recipient's name in the message at the application level.  e.g. "Dear Sue,"
is part of the message, so it can't be mistaken as a message to Joan.  But
that doesn't handle the issue of private information in the message.  It's
possible for Sue to reveal the signed message to someone else, who can
verify the signature, without needing Sue's key.  I would prefer if that
were impossible--without Sue's key, the message can't be authenticated
either.

Marc Mutz <[email-masked]>@mail.imc.org on 04-09-2002 12:47:01 PM

Sent by:  [email-masked]

To:   [email-masked]
cc:   [email-masked], [email-masked]
Subject:  Re: Bug#40394: forwarding an encrypted PGP message is useless

[ OK, I put this where it belongs: this touches OpenPGP and Internet
messages,
 so I'm forwarding this to the ietf lists ]

We have a problem and we think we are not alone with it (see the mozilla
link
below):

What if, in a mail user agent, the user wants to forward an encrypted
message?
Allow it? Deny it?
Re-encrypt or remove encryption?

The problem is, of course, that the original sender might not like his
encrypted text being sent out in the clear again...

Was this discussed here already? What was the consensus reached, if any?

Marc

On Tuesday 09 April 2002 13:01, Volker Augustin wrote:
> > Peronally, I think the best way is to ask the user
> > what to do when forwarding (inlined) an encrypted message. We cannot
> > prevent that he/she will forward the plaintext using a workaround. So
we
> > better delegate the resposibility for this action to the user itself.
> >
> > There is still the question, what should be the default way. Another
way
> > would to set the default to the last recently used way. Hm, I'm sure
> > you'll find a good solution.
>
> That might be a good solution. When trying for forward bring up a dialog
> asking whether to allow forwarding just this time or also for the future.
>
> I just did an intensive search via google. Here are some of my findings:
>
> 1. Some mail programs have an option to prevent forwarding of encrypted
>     messages. I could not find a single one explicitely preventing it.
> Some commercial application however give the system administrator the
> possibility to enforce this option system-wide.
>
> 2. RFC 1421 about PEM states (http://www.freesoft.org/CIE/RFC/1421/55.htm
)
>
>    "Some level of support for generating and processing nested and
> annotated PEM messages (for forwarding purposes) is to be provided, and
an
> implementation should be able to reduce ENCRYPTED messages to MIC-ONLY or
> MIC-CLEAR for forwarding."
>
> 3. There is an IETF draft about "Intended Recipients Attribute for the
>    Cryptographic Message Syntax (CMS)"
>    (http://www.ietf.org/internet-drafts/draft-ietf-smime-ira-00.txt)
saying
>
>     "The problem of intent that as expressed in [MALFWD] is beyond the
>     control of S/MIME protocol or its implementers.  The use of the
digital
>     signatures and encryption is correctly in the hands of the user.
> However, the intended recipients attribute offers a mechanism to reduce
the
> likelihood of undetected malicious forwarding."
>
> 4. The mozilla team has the same problem:
>     http://www.mozilla.org/mailnews/compose_send_plugin_arch_notes.html
>
>     "open issues:
>     1) forwarding an encrypted message?
>     2) standards for content type S/MIME. PGP. GPG. OpenPGP.
>     3) Eudora and Outlook, etc. what do we need to do to support reading
> encrypted message from other clients? "
>
> Regards,
> Volker
> _______________________________________________
> KMail Developers mailing list
> [email-masked]
> http://mail.kde.org/mailman/listinfo/kmail

- --
Marc Mutz <[email-masked]>

Date: Tue, 09 Apr 2002 16:30:30 -0400
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Keith, I usually agree with you, but not in this case. This suggested 
content-type DOES describe the content type: it IS a zip-compressed file 
that's enclosed there. It just happens that you also know what was 
compressed.

	Tony Hansen
	[email-masked]

Keith Moore wrote:

>>So, I suggested:
>>content-type: application/x-zip-compressed; encapsulates=application/pdf
> sorry, this is just unacceptable.  there are too many things (e.g. mail 
> filters) that expect the content-type to actually describe the content.  

To: Marc Mutz <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
From: Simon Josefsson <[email-masked]>
Date: Tue, 09 Apr 2002 21:32:00 +0200
Lines: 23
Sender: [email-masked]

Marc Mutz <[email-masked]> writes:

> What if, in a mail user agent, the user wants to forward an encrypted message?
> Allow it? Deny it?
> Re-encrypt or remove encryption?
>
> The problem is, of course, that the original sender might not like his 
> encrypted text being sent out in the clear again...

Then the original sender should not send the text to someone who will
do that.

I don't see how the standard could prevent the user from doing
this. If it is prevented, then it is only the applications' doing, so
it wouldn't be difficult to override it.  This isn't a technical
problem. Whenever you introduce encryption between two parties you
have this issue.

Whether to re-encrypt or remove encryption is up to the user
forwarding the message.  What the defaults should be probably depend
too much on the application to give any generic recomendation on.

Just my two cents.

From: "Leon Kuunders" <[email-masked]>
To: "Marc Mutz" <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>
Subject: RE: Bug#40394: forwarding an encrypted PGP message is useless
Date: Tue, 9 Apr 2002 21:08:28 +0200
Importance: Normal
Sender: [email-masked]

I don't know if this was discussed, but for me this is not a question of
encryption, but of trust. If you don't like someone else to forward your
messages, send them on paper. With a company logo, a watermark and a unique
number.

It is the risc of living :)

-Leon.

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Marc Mutz
> Sent: 09 April 2002 19:47
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
>
>
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> [ OK, I put this where it belongs: this touches OpenPGP and
> Internet messages,
>  so I'm forwarding this to the ietf lists ]
>
> We have a problem and we think we are not alone with it (see the
> mozilla link
> below):
>
> What if, in a mail user agent, the user wants to forward an
> encrypted message?
> Allow it? Deny it?
> Re-encrypt or remove encryption?
>
> The problem is, of course, that the original sender might not like his
> encrypted text being sent out in the clear again...
>
> Was this discussed here already? What was the consensus reached, if any?
>
> Marc
>
> On Tuesday 09 April 2002 13:01, Volker Augustin wrote:
> > > Peronally, I think the best way is to ask the user
> > > what to do when forwarding (inlined) an encrypted message. We cannot
> > > prevent that he/she will forward the plaintext using a
> workaround. So we
> > > better delegate the resposibility for this action to the user itself.
> > >
> > > There is still the question, what should be the default way.
> Another way
> > > would to set the default to the last recently used way. Hm, I'm sure
> > > you'll find a good solution.
> >
> > That might be a good solution. When trying for forward bring up a dialog
> > asking whether to allow forwarding just this time or also for
> the future.
> >
> > I just did an intensive search via google. Here are some of my findings:
> >
> > 1. Some mail programs have an option to prevent forwarding of encrypted
> >     messages. I could not find a single one explicitely preventing it.
> > Some commercial application however give the system administrator the
> > possibility to enforce this option system-wide.
> >
> > 2. RFC 1421 about PEM states
(http://www.freesoft.org/CIE/RFC/1421/55.htm)
>
>    "Some level of support for generating and processing nested and
> annotated PEM messages (for forwarding purposes) is to be provided, and an
> implementation should be able to reduce ENCRYPTED messages to MIC-ONLY or
> MIC-CLEAR for forwarding."
>
> 3. There is an IETF draft about "Intended Recipients Attribute for the
>    Cryptographic Message Syntax (CMS)"
>    (http://www.ietf.org/internet-drafts/draft-ietf-smime-ira-00.txt)
saying
>
>     "The problem of intent that as expressed in [MALFWD] is beyond the
>     control of S/MIME protocol or its implementers.  The use of the
digital
>     signatures and encryption is correctly in the hands of the user.
> However, the intended recipients attribute offers a mechanism to reduce
the
> likelihood of undetected malicious forwarding."
>
> 4. The mozilla team has the same problem:
>     http://www.mozilla.org/mailnews/compose_send_plugin_arch_notes.html
>
>     "open issues:
>     1) forwarding an encrypted message?
>     2) standards for content type S/MIME. PGP. GPG. OpenPGP.
>     3) Eudora and Outlook, etc. what do we need to do to support reading
> encrypted message from other clients? "
>
> Regards,
> Volker
> _______________________________________________
> KMail Developers mailing list
> [email-masked]
> http://mail.kde.org/mailman/listinfo/kmail

- --
Marc Mutz <[email-masked]>

Date: Tue, 9 Apr 2002 20:39:55 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Marc Mutz <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
Sender: [email-masked]

Marc Mutz <[email-masked]>
> The problem is, of course, that the original sender might not like his 
> encrypted text being sent out in the clear again...

The original sender might not his text being sent out at all. He sent it
to you; you're showing it to some third party.

Personally, I suggest using as much encryption as reasonably possible. In
general, and naturally also in this particular case.

--Arnt

Date: Tue, 9 Apr 2002 20:20:36 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: Marc Mutz <[email-masked]>, [email-masked], [email-masked]
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
Sender: [email-masked]

Keith Moore <[email-masked]>
> why bother?  just generate a subset of HTML.  more MUAs will be able to
> read it that way.

And more people filter away the message. Rightly or wrongly.

--Arnt

Date: Tue, 09 Apr 2002 19:47:01 +0200
From: Marc Mutz <[email-masked]>
Subject: Re: Bug#40394: forwarding an encrypted PGP message is useless
To: [email-masked]
Cc: [email-masked], [email-masked]
Organization: KDE
Sender: [email-masked]

[ OK, I put this where it belongs: this touches OpenPGP and Internet messages,
 so I'm forwarding this to the ietf lists ]

We have a problem and we think we are not alone with it (see the mozilla link 
below):

What if, in a mail user agent, the user wants to forward an encrypted message?
Allow it? Deny it?
Re-encrypt or remove encryption?

The problem is, of course, that the original sender might not like his 
encrypted text being sent out in the clear again...

Was this discussed here already? What was the consensus reached, if any?

Marc

On Tuesday 09 April 2002 13:01, Volker Augustin wrote:
> > Peronally, I think the best way is to ask the user
> > what to do when forwarding (inlined) an encrypted message. We cannot
> > prevent that he/she will forward the plaintext using a workaround. So we
> > better delegate the resposibility for this action to the user itself.
> >
> > There is still the question, what should be the default way. Another way
> > would to set the default to the last recently used way. Hm, I'm sure
> > you'll find a good solution.
>
> That might be a good solution. When trying for forward bring up a dialog
> asking whether to allow forwarding just this time or also for the future.
>
> I just did an intensive search via google. Here are some of my findings:
>
> 1. Some mail programs have an option to prevent forwarding of encrypted
>     messages. I could not find a single one explicitely preventing it. 
> Some commercial application however give the system administrator the
> possibility to enforce this option system-wide.
>
> 2. RFC 1421 about PEM states (http://www.freesoft.org/CIE/RFC/1421/55.htm)
>
>    "Some level of support for generating and processing nested and
> annotated PEM messages (for forwarding purposes) is to be provided, and an
> implementation should be able to reduce ENCRYPTED messages to MIC-ONLY or
> MIC-CLEAR for forwarding."
>
> 3. There is an IETF draft about "Intended Recipients Attribute for the
>    Cryptographic Message Syntax (CMS)"
>    (http://www.ietf.org/internet-drafts/draft-ietf-smime-ira-00.txt) saying
>
>     "The problem of intent that as expressed in [MALFWD] is beyond the
>     control of S/MIME protocol or its implementers.  The use of the digital
>     signatures and encryption is correctly in the hands of the user.
> However, the intended recipients attribute offers a mechanism to reduce the
> likelihood of undetected malicious forwarding."
>
> 4. The mozilla team has the same problem:
>     http://www.mozilla.org/mailnews/compose_send_plugin_arch_notes.html
>
>     "open issues:
>     1) forwarding an encrypted message?
>     2) standards for content type S/MIME. PGP. GPG. OpenPGP.
>     3) Eudora and Outlook, etc. what do we need to do to support reading
> encrypted message from other clients? "
>
> Regards,
> Volker
> _______________________________________________
> KMail Developers mailing list
> [email-masked]
> http://mail.kde.org/mailman/listinfo/kmail

- -- 
Marc Mutz <[email-masked]>

From: Keith Moore <[email-masked]>
To: Marc Mutz <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: BOF: Format-Flowed and Non-Western Charsets 
Date: Tue, 09 Apr 2002 14:02:59 -0400
Sender: [email-masked]

> BTW: Is it worthwhile to implement text/enriched generation in a user agent
> where HTML composing will not be available for the forseeable future?

why bother?  just generate a subset of HTML.  more MUAs will be able to
read it that way.

(I wish all MUAs generated only a tiny subset of HTML.)

Keith

From: Keith Moore <[email-masked]>
To: Sam Roberts <[email-masked]>
cc: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Tue, 09 Apr 2002 14:01:32 -0400
Sender: [email-masked]

> So, I suggested:
> 
> content-type: application/x-zip-compressed; encapsulates=application/pdf

sorry, this is just unacceptable.  there are too many things (e.g. mail 
filters) that expect the content-type to actually describe the content.  

Keith

Date: Tue, 09 Apr 2002 09:59:27 +0200
From: Marc Mutz <[email-masked]>
Subject: Re: BOF: Format-Flowed and Non-Western Charsets
To: [email-masked], [email-masked]
Organization: KDE
Sender: [email-masked]

On Wednesday 20 March 2002 22:45, [email-masked] wrote:
<snip>
> text/richtext. text/enriched. format=flowed. Time to stop.
<snip>

BTW: Is it worthwhile to implement text/enriched generation in a user agent 
where HTML composing will not be available for the forseeable future?

Or is there not enough deployed support for reading this?
I _do_ know that Pegasus seems to be able to compose it, since I sometimes get 
text/enriched mails from a Pegasus user.

Marc

- -- 
Marc Mutz <[email-masked]>

Date: Sun, 7 Apr 2002 16:33:02 -0400
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Sun, Apr 07, 2002 at 08:44:17AM -0700:
> > Quoteing [email-masked], on Sat, Apr 06, 2002 at 06:13:17PM -0800:

I'm pretty confused. These comments don't seem to be on
what I proposed, but I just noticed I mistyped "content-type"
as "application-type" in my example, even though I called it
content type. Maybe that set you off?

Because surely you aren't claiming that people don't actually send zipped
pdfs with a content-type of:

content-type: application/x-zip-compressed

?

This could be considered a violation of 2.2.1, and even a bad idea,
and you could say it doesn't work well, and I would agree with you.

So, you propose that there could be a new CTE, which would be treated as
"binary" by agents which don't know it. It appears to me that an
agent that doesn't understand that CTE will be required to treat it
as "binary". Its not clear to me that this will result in a MUA being
able to do anything with it. Certainly treating it as the advertised
content type won't work, since it would be compressed. I'm also curious
about whether you would suggest two new CTEs, one base64 encoded (for
non 8bit safe transfer, like some SMTP) and one not (for 8bit safe
NNTP tranfer, for example). It just doesn't seem orthogonal to me. But
you like the idea, so I'd like to hear why you don't think these are
problems, if you have time (and you seem to have some :-).

So, I suggested:

content-type: application/x-zip-compressed; encapsulates=application/pdf

It is valid, according to the famous 2.2.1, it will be handled correctly
when passed to whatever is registered as a viewer for that type. It will
also be handled even more correctly if the MUA supports the encapsulates
parameter, and realizes it can be unwrapped and treated as application/pdf.

> > Sure, I could live with a compressed CTE. I could even live with your
> > favorite algorithm (hopefully, so can everybody else, or do we need
> > one CTE per algorithm? yuck.).
> 
> Then why are you arguing that it must be done some other way? It is exactly
> this sort of thing that leads to nothing being done every time this comes
> up -- arguments like this one exhaust everyone so no documents get produced.

I'm not arguing that it MUST be done any particular way, there seem to be two
ways that are syntactically valid, addition of a parameter to content-type,
and definition of a new CTE. One appears to have the advantage of having a
fallback mode to being the same as the current situation, one seems to have a
fallback mode of total non-interoperability. I'm still curious why you
favor a new (or 2 new?) CTEs. If it was so clearly right, it would already
have been done, no?

Good day,
Sam

-- 
Sam Roberts <[email-masked]> (Vivez sans temps mort!)

Date: Sun, 07 Apr 2002 08:44:17 -0700 (PDT)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Sam Roberts <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Quoteing [email-masked], on Sat, Apr 06, 2002 at 06:13:17PM -0800:
> >
> > > > > Has this been proposed? It seems a parameter to the content-transfer-encoding
> > > > > would be handy. I'm sure I'm not the only one who has wondered why we
> > > > > have to call things application/pick-your-compression-format instead
> > > > > of application/tar, as it should be.
> > > >
> > > > It was proposed over a decade ago. It was rejected then because it was felt
> > > > to be too complex.
> >
> > > Really? There seems to be much more complex parts of MIME. Oh well.
> >
> > I have to disagree. The complexity implied by a parameterized CTE is
> > in practice pretty large; there is nothing in the core MIME capabilities
> > that is comparable.

> Well, I have to disagree. Given that you must pass content through a content
> decoder (base64,qp,etc.), passing content through a decompression library as
> well is pretty trivial.

> Correctly transmitting non-ascii text in headers is harder, IMO.

> What was supposed to be hard?

> > > Not that I think this will go anywhere, but what about adding a parameter
> > > to Content-Type. It accepts parameters already, and I hope that most
> > > implementations would just ignore any parameters it didn't support.
> >
> > But that's precisely the problem: You don't want something that completely
> > alters the way the content is interpreted to be ignored. The possible
> > operational results are nothing short of grotesque.

> Describe them. These "grotesque" operational results are current practice.

No they are not. There is currently no practice where a content-type
is sent saying something is of some type or other but a parameter happens
to add that oh by the way the content is compressed. The results of doing
this would likely include crashing applications all over the place.

Now, you may claim that applications should be robust against this sort
of thing. I agree. But they aren't. And this isn't likely to change.

> Ignoring this information is the same as not sending it, and right now it
> isn't sent. Instead, compressed data is sent as application/compression-algo,
> it is decompressed on the other side, and people then try and guess what the
> stuff is.

That's an entirely different proposition. Sending something that happens
to be compressed with an unregistered content type nobody understands
is not the same as tacking on a compression parameter. And you yourself
have just described the problem with this approach.

> > > application-type: application/x-zip-compressed; encapsulates=application/pdf

> > And now you've violated the MIME concept that the content type and subtype
> > actually describe the content in some useful way. That's not acceptable
> > either; in fact it is quite specifically prohibited; see RFC 2048 section
> > 2.2.1.

> Huh? I described existing practice.

People violate the standards all the time. Just because they do so doesn't
mean it works well, is a good idea, or is commonplace. In this case it is
none of the three.

> The content IS x-zip-compressed, as required by 2.2.1.

> > Look, this stuff was all thought out in detail more than a decade back. For
> > various reasons that we need not get into defining a compressed CTE wasn't a

> Perhaps getting into them might describe why the ball was dropped.

Sorry. You'll have to find someone else to try and summarize many megabytes
of mail sent over a decade ago. I have better things to do with my time.

> > viable option at the time. But we all knew it needed to be something that MIME
> > could accomodate. But a multiplicity of ways of doing it was felt to be a bad
> > idea. So MIME was designed so that there is one, and only one, place this fits:
> > One or more CTEs.

> Well, it looks like two, and only two, to me.

> > Now, we could retroactivaly impose some structure on CTE names that fit within
> > the current syntax, but it seems somewhat pointless to do so: Combinations like
> > gzip-quoted-printable, or worse gzip-8bit, aren't useful. All this does is
> > raise the bar for implementations. Let's just define a couple of compressed
> > CTEs and be done with it.

> Sure, I could live with a compressed CTE. I could even live with your
> favorite algorithm (hopefully, so can everybody else, or do we need
> one CTE per algorithm? yuck.).

Then why are you arguing that it must be done some other way? It is exactly
this sort of thing that leads to nothing being done every time this comes
up -- arguments like this one exhaust everyone so no documents get produced.

> Btw, that compressed data, how is it encoded? base64, likely. It seems we'll
> be wrapping two encodings into a single token. Weird, but if it gets
> the job done, fine.

Well, the one obvious thing to do is define an efficient 8bit encoding. This
has been discussed in another thread. But yes, I agree a 7bit one is probably
also useful. However, we could make some progress in that space by moving to
base85. I believe the EBCDIC transport issues that drove the base64 choice are
well and truly gone by now.

> Of course, this means that a CTE which is base64 gzipped data will get
> treated as binary by MUAs that don't understand this CTE. So it'll
> either get passed to the application that handles that content-type,
> which will choke, or it will get saved as a b64 blob, and they won't
> be able to do anything with it. Seems grotesque to me.

That's not what the standards say must be done. Anything with an unrecognized
CTE has to be treated as data of an unknown type. Any implementation that fails
to do so is seriously incompliant with the MIME specification.

> I'm more convinced that an "encapsualates" parameter is nice, it degrades
> to the current situation, as opposed to degrading to useless.

I've already explained why this is a nonstarter. I won't bother to repeat it.
And I'm going to heed my own often-given advice now and stop discussing this
over-discussed issue now.

				Ned

Date: Sun, 7 Apr 2002 11:02:17 -0400
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Sat, Apr 06, 2002 at 06:13:17PM -0800:
> 
> > > > Has this been proposed? It seems a parameter to the content-transfer-encoding
> > > > would be handy. I'm sure I'm not the only one who has wondered why we
> > > > have to call things application/pick-your-compression-format instead
> > > > of application/tar, as it should be.
> > >
> > > It was proposed over a decade ago. It was rejected then because it was felt
> > > to be too complex.
> 
> > Really? There seems to be much more complex parts of MIME. Oh well.
>  
> I have to disagree. The complexity implied by a parameterized CTE is
> in practice pretty large; there is nothing in the core MIME capabilities
> that is comparable.

Well, I have to disagree. Given that you must pass content through a content
decoder (base64,qp,etc.), passing content through a decompression library as
well is pretty trivial.

Correctly transmitting non-ascii text in headers is harder, IMO.

What was supposed to be hard?

> > Not that I think this will go anywhere, but what about adding a parameter
> > to Content-Type. It accepts parameters already, and I hope that most
> > implementations would just ignore any parameters it didn't support.
> 
> But that's precisely the problem: You don't want something that completely
> alters the way the content is interpreted to be ignored. The possible
> operational results are nothing short of grotesque.

Describe them. These "grotesque" operational results are current practice.
Ignoring this information is the same as not sending it, and right now it
isn't sent. Instead, compressed data is sent as application/compression-algo,
it is decompressed on the other side, and people then try and guess what the
stuff is.

> This is the same reason you cannot just define a content-compression field,
> or content-disposition parameter, or anything else, to do this.
> 
> > Doing:
> 
> > content-type: application/tar; compression=gzip
> 
> > would be fairly accurate, but if a MUA didn't support the compression
> > parameter it would think that the part was application/tar, but tar
> > would choke on it.
> 
> Exactly.
> 
> > On the other hand, you could do:
> 
> > content-type: application/x-gunzip; encapsulates=application/tar
> 
> > or:
> 
> > application-type: application/x-zip-compressed; encapsulates=application/pdf
> 
> And now you've violated the MIME concept that the content type and subtype
> actually describe the content in some useful way. That's not acceptable
> either; in fact it is quite specifically prohibited; see RFC 2048 section
> 2.2.1.

Huh? I described existing practice.

The content IS x-zip-compressed, as required by 2.2.1.

> Look, this stuff was all thought out in detail more than a decade back. For
> various reasons that we need not get into defining a compressed CTE wasn't a

Perhaps getting into them might describe why the ball was dropped.

> viable option at the time. But we all knew it needed to be something that MIME
> could accomodate. But a multiplicity of ways of doing it was felt to be a bad
> idea. So MIME was designed so that there is one, and only one, place this fits:
> One or more CTEs.

Well, it looks like two, and only two, to me.

> Now, we could retroactivaly impose some structure on CTE names that fit within
> the current syntax, but it seems somewhat pointless to do so: Combinations like
> gzip-quoted-printable, or worse gzip-8bit, aren't useful. All this does is
> raise the bar for implementations. Let's just define a couple of compressed
> CTEs and be done with it.

Sure, I could live with a compressed CTE. I could even live with your
favorite algorithm (hopefully, so can everybody else, or do we need
one CTE per algorithm? yuck.).

Btw, that compressed data, how is it encoded? base64, likely. It seems we'll
be wrapping two encodings into a single token. Weird, but if it gets
the job done, fine.

Of course, this means that a CTE which is base64 gzipped data will get
treated as binary by MUAs that don't understand this CTE. So it'll
either get passed to the application that handles that content-type,
which will choke, or it will get saved as a b64 blob, and they won't
be able to do anything with it. Seems grotesque to me.

I'm more convinced that an "encapsualates" parameter is nice, it degrades
to the current situation, as opposed to degrading to useless.

> > Hopefully, a MUA could handle it as a zip file, and be right, or
> > it could lookup how to unwrap it in mailcap, then handle it as PDF,
> > which would be more right.
> 
> Dream on.

Well, your kind words aside, MUAs already handle zip and pdf, I fail to
see why handling them sequentially would be such a problem for us.

I'm beginning to wonder if this wasn't dealt with in MIME because
nobody could agree where to put it.

Sam

-- 
Sam Roberts <[email-masked]> (Vivez sans temps mort!)

Date: Sat, 06 Apr 2002 18:13:17 -0800 (PST)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Sam Roberts <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> > > Has this been proposed? It seems a parameter to the content-transfer-encoding
> > > would be handy. I'm sure I'm not the only one who has wondered why we
> > > have to call things application/pick-your-compression-format instead
> > > of application/tar, as it should be.
> >
> > It was proposed over a decade ago. It was rejected then because it was felt
> > to be too complex.

> Really? There seems to be much more complex parts of MIME. Oh well.
 
I have to disagree. The complexity implied by a parameterized CTE is
in practice pretty large; there is nothing in the core MIME capabilities
that is comparable.

> > But it would also change the syntax of the header, as CTE (the header)
> > currently doesn't allow parameters.
> > What about a list of cte's (then better dubbed
> > "content-transfer-transformations") to be applied in the format
> > innermostCTE+...+outermostCTE
> > (or vice versa), so you could say
> > C.-T.-E: gzip+base64
> > meaning "first remove base64 encoding, then gunzip". The + here is chosen for
> > being not tspecial (the cte identifier is a "token") and suggestive of "and"
> > and of the "+xml" convention (rfc3023).
> > A legacy agent would parse the cte name to be the full "gzip+base64" and get
> > an unknown cte, much like when plain "gzip" would be used.

> Right, so most agents in existence would just choke, since they "know"
> all the possible CTEs. So, this is syntactically backwards compatible,
> but won't degrade gracefully.

The MIME specifications specifically says how to deal with an unknown CTE. They
degrade as gracefully as anything like this could. A syntactically incompatible
change, on the other hand, is by definition outside the scope of the standard.
Implementations could potentially handle this is all sorts of ways, some of
them pretty horrendous operationally.

> Not that I think this will go anywhere, but what about adding a parameter
> to Content-Type. It accepts parameters already, and I hope that most
> implementations would just ignore any parameters it didn't support.

But that's precisely the problem: You don't want something that completely
alters the way the content is interpreted to be ignored. The possible
operational results are nothing short of grotesque.

This is the same reason you cannot just define a content-compression field,
or content-disposition parameter, or anything else, to do this.

> Doing:

> content-type: application/tar; compression=gzip

> would be fairly accurate, but if a MUA didn't support the compression
> parameter it would think that the part was application/tar, but tar
> would choke on it.

Exactly.

> On the other hand, you could do:

> content-type: application/x-gunzip; encapsulates=application/tar

> or:

> application-type: application/x-zip-compressed; encapsulates=application/pdf

And now you've violated the MIME concept that the content type and subtype
actually describe the content in some useful way. That's not acceptable
either; in fact it is quite specifically prohibited; see RFC 2048 section
2.2.1.

Look, this stuff was all thought out in detail more than a decade back. For
various reasons that we need not get into defining a compressed CTE wasn't a
viable option at the time. But we all knew it needed to be something that MIME
could accomodate. But a multiplicity of ways of doing it was felt to be a bad
idea. So MIME was designed so that there is one, and only one, place this fits:
One or more CTEs.

Now, we could retroactivaly impose some structure on CTE names that fit within
the current syntax, but it seems somewhat pointless to do so: Combinations like
gzip-quoted-printable, or worse gzip-8bit, aren't useful. All this does is
raise the bar for implementations. Let's just define a couple of compressed
CTEs and be done with it.

> Hopefully, a MUA could handle it as a zip file, and be right, or
> it could lookup how to unwrap it in mailcap, then handle it as PDF,
> which would be more right.

Dream on.

				Ned

Date: Sat, 6 Apr 2002 20:02:54 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Quoteing [email-masked], on Sat, Apr 06, 2002 at 10:50:33AM -0800:
> 
> 
> > >
> > > In <[email-masked]> Russ Allbery <[email-masked]> writes:
> > > >Dan Kohn <[email-masked]> writes:
> > > >It's not really been considered for Usenet because essentially all of the
> > > >binaries posted to Usenet are already compressed.  (GIFs, JPEGs, MP3s, ZIP
> > > >files, compressed movies, etc.)  A compressed CTE would be a lot more
> > > >useful in mail than on Usenet.
> > >
> > > Indeed so.
> 
> > snip...
> 
> > > But there is another problem with a gzip CTE, which is that you may have
> > > gzip on top of Base64, or gzip on top of yEnc, or qzip on top of whatever
> > > else. Maybe, in an ideal world, one would define an encoding (yEnc say)
> > > and then define a gzip option inside that.
> > >
> > > Or maybe you could define a compression=gzip parameter that could be used
> > > with any CTE. That would seem to be more flexible.
> 
> > Has this been proposed? It seems a parameter to the content-transfer-encoding
> > would be handy. I'm sure I'm not the only one who has wondered why we
> > have to call things application/pick-your-compression-format instead
> > of application/tar, as it should be.
> 
> It was proposed over a decade ago. It was rejected then because it was felt
> to be too complex.

Really? There seems to be much more complex parts of MIME. Oh well.
 
> I disagreed with the decision then and I disagree with it now. However,
> the issue is moot, because a major change in the syntax of the CTE parameter
> is no longer possible. Think for a moment about the size of the installed
> base and it should be obvious why this is so.

Fair enough.

Quoteing [email-masked], on Sat, Apr 06, 2002 at 08:54:11PM +0200:
> 
> On Friday 05 April 2002 19:03, Charles Lindsey wrote:
> <snip>
> > Or maybe you could define a compression=gzip parameter that could be used
> > with any CTE. That would seem to be more flexible.
> <snip>
> 
> More flexible, yes.
> 
> But it would also change the syntax of the header, as CTE (the header) 
> currently doesn't allow parameters.
> What about a list of cte's (then better dubbed 
> "content-transfer-transformations") to be applied in the format
> innermostCTE+...+outermostCTE
> (or vice versa), so you could say
> C.-T.-E: gzip+base64
> meaning "first remove base64 encoding, then gunzip". The + here is chosen for 
> being not tspecial (the cte identifier is a "token") and suggestive of "and" 
> and of the "+xml" convention (rfc3023).
> A legacy agent would parse the cte name to be the full "gzip+base64" and get 
> an unknown cte, much like when plain "gzip" would be used.

Right, so most agents in existence would just choke, since they "know"
all the possible CTEs. So, this is syntactically backwards compatible,
but won't degrade gracefully.

Not that I think this will go anywhere, but what about adding a parameter
to Content-Type. It accepts parameters already, and I hope that most
implementations would just ignore any parameters it didn't support.

Doing:

content-type: application/tar; compression=gzip

would be fairly accurate, but if a MUA didn't support the compression
parameter it would think that the part was application/tar, but tar
would choke on it.

On the other hand, you could do:

content-type: application/x-gunzip; encapsulates=application/tar

or:

application-type: application/x-zip-compressed; encapsulates=application/pdf

Hopefully, a MUA could handle it as a zip file, and be right, or
it could lookup how to unwrap it in mailcap, then handle it as PDF,
which would be more right.

Sam

-- 
Sam Roberts <[email-masked]> (Vivez sans temps mort!)

From: Marc Mutz <[email-masked]>
Organization: KDE
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Date: Sat, 6 Apr 2002 20:54:11 +0200
Sender: [email-masked]

On Friday 05 April 2002 19:03, Charles Lindsey wrote:
<snip>
> But there is another problem with a gzip CTE, which is that you may have
> gzip on top of Base64, or gzip on top of yEnc, or qzip on top of whatever
> else. Maybe, in an ideal world, one would define an encoding (yEnc say)
> and then define a gzip option inside that.
>
> Or maybe you could define a compression=gzip parameter that could be used
> with any CTE. That would seem to be more flexible.
<snip>

More flexible, yes.

But it would also change the syntax of the header, as CTE (the header) 
currently doesn't allow parameters.
What about a list of cte's (then better dubbed 
"content-transfer-transformations") to be applied in the format
innermostCTE+...+outermostCTE
(or vice versa), so you could say
C.-T.-E: gzip+base64
meaning "first remove base64 encoding, then gunzip". The + here is chosen for 
being not tspecial (the cte identifier is a "token") and suggestive of "and" 
and of the "+xml" convention (rfc3023).
A legacy agent would parse the cte name to be the full "gzip+base64" and get 
an unknown cte, much like when plain "gzip" would be used.

I know it's uglier than introducing a "compression" parameter, but it doesn't 
change the syntax of the cte header field and gets rid of the need of 
backreferencing a classical CTE to be used with any newly defined 
(compressing) one.

The yEnc problem of where to put the CRC would also be solvable:
C.-T.-E: yenc-crc+yenc
meaning compute the crc, append (or prepend) it, then do the actual encoding 
that is yenc.
Not that I like it this way...

Marc

- -- 
Marc Mutz <[email-masked]>

Date: Sat, 06 Apr 2002 10:50:33 -0800 (PST)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Sam Roberts <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> >
> > In <[email-masked]> Russ Allbery <[email-masked]> writes:
> > >Dan Kohn <[email-masked]> writes:
> > >It's not really been considered for Usenet because essentially all of the
> > >binaries posted to Usenet are already compressed.  (GIFs, JPEGs, MP3s, ZIP
> > >files, compressed movies, etc.)  A compressed CTE would be a lot more
> > >useful in mail than on Usenet.
> >
> > Indeed so.

> snip...

> > But there is another problem with a gzip CTE, which is that you may have
> > gzip on top of Base64, or gzip on top of yEnc, or qzip on top of whatever
> > else. Maybe, in an ideal world, one would define an encoding (yEnc say)
> > and then define a gzip option inside that.
> >
> > Or maybe you could define a compression=gzip parameter that could be used
> > with any CTE. That would seem to be more flexible.

> Has this been proposed? It seems a parameter to the content-transfer-encoding
> would be handy. I'm sure I'm not the only one who has wondered why we
> have to call things application/pick-your-compression-format instead
> of application/tar, as it should be.

It was proposed over a decade ago. It was rejected then because it was felt
to be too complex.

I disagreed with the decision then and I disagree with it now. However,
the issue is moot, because a major change in the syntax of the CTE parameter
is no longer possible. Think for a moment about the size of the installed
base and it should be obvious why this is so.

				Ned

Date: Fri, 5 Apr 2002 15:34:11 -0500
From: Sam Roberts <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Mail-Followup-To: [email-masked]
Sender: [email-masked]

> 
> In <[email-masked]> Russ Allbery <[email-masked]> writes:
> >Dan Kohn <[email-masked]> writes:
> >It's not really been considered for Usenet because essentially all of the
> >binaries posted to Usenet are already compressed.  (GIFs, JPEGs, MP3s, ZIP
> >files, compressed movies, etc.)  A compressed CTE would be a lot more
> >useful in mail than on Usenet.
> 
> Indeed so.

snip...

> But there is another problem with a gzip CTE, which is that you may have
> gzip on top of Base64, or gzip on top of yEnc, or qzip on top of whatever
> else. Maybe, in an ideal world, one would define an encoding (yEnc say)
> and then define a gzip option inside that.
> 
> Or maybe you could define a compression=gzip parameter that could be used
> with any CTE. That would seem to be more flexible.

Has this been proposed? It seems a parameter to the content-transfer-encoding
would be handy. I'm sure I'm not the only one who has wondered why we
have to call things application/pick-your-compression-format instead
of application/tar, as it should be.

Sam

-- 
Sam Roberts <[email-masked]> (Vivez sans temps mort!)

To: [email-masked]
Xref: clerew local.mime:1222
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Content-Transfer-Encoding and yEnc
Date: Fri, 5 Apr 2002 17:03:50 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Dan Kohn <[email-masked]> writes:

>> I don't mean to open a rat hole here, but has the use of gzip as part of
>> the yEnc encoding been seriously considered?

>It's not really been considered for Usenet because essentially all of the
>binaries posted to Usenet are already compressed.  (GIFs, JPEGs, MP3s, ZIP
>files, compressed movies, etc.)  A compressed CTE would be a lot more
>useful in mail than on Usenet.

Indeed so.

BTW, and changing the topic, does anyone know when the patent behind GIFs
expires?

>(Why some compressed formats have large sections of NULs, I have no idea.
>That makes me think that either some of those compression formats are less
>than ideal or the advantages of byte-shifting have been overstated.)

Yes, I didn't quite understand that (I was just passing on the reason for
byte shifting as I had seen it explained).

But there is another problem with a gzip CTE, which is that you may have
gzip on top of Base64, or gzip on top of yEnc, or qzip on top of whatever
else. Maybe, in an ideal world, one would define an encoding (yEnc say)
and then define a gzip option inside that.

Or maybe you could define a compression=gzip parameter that could be used
with any CTE. That would seem to be more flexible.

As for the yEnc CTE, it has been explained to the people concerned that
favourable noises have been made here, and that they ought to come here
and pursue it themselves, but no response yet. My only role in this was to
try and break the impasses/standoff that seemed to have been reached.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Thu, 04 Apr 2002 11:02:50 -0800 (PST)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> In <[email-masked]> [email-masked] writes:

> >Well, yEnc as presently formulated has a failure mode where some messages
> >will grow to 2X their original size. I also don't like the fact that
> >it shifts the range of almost every character. It is possible to fix both
> >of these problems and have an encoding with an upper bound guarantee that
> >leaves the original data mostly untouched.

> >Just my personal take -- neither of these are necessarily showstoppers.

> I think every encoding scheme has a pathological case which produces a "2X
> their original size" situation.

On the contrary, the vast majority don't have any such case. base64, uuencode,
and base85, for example, consistently increase size by 33% or 20% no matter
what the input.

The yEnc class of encodings readilty admits a block-by-block approach that
_guarantees_ at most 2-3% overhead.

> I think such an accurrence is wildly improbable in the yEnc case.

Unlikely, perhaps. But wildly improbably is a stretch. And why take the
chance?

> AIUI, the reason for the range shifting is that the NUL character tends to
> occur with more than its random probability in typical binaries (even to
> the extent of huge sequences of NUL bytes). Therefore, it is desirable
> that NUL itself can be transported without stuffing (or course, some other
> random character then gets saddled with that honour).

Sure, but there are ways to accomplish this that don't require blanket
range shifting.

				Ned

To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
From: Russ Allbery <[email-masked]>
Organization: The Eyrie
Date: Thu, 04 Apr 2002 10:55:34 -0800
Lines: 16
Sender: [email-masked]

Dan Kohn <[email-masked]> writes:

> I don't mean to open a rat hole here, but has the use of gzip as part of
> the yEnc encoding been seriously considered?

It's not really been considered for Usenet because essentially all of the
binaries posted to Usenet are already compressed.  (GIFs, JPEGs, MP3s, ZIP
files, compressed movies, etc.)  A compressed CTE would be a lot more
useful in mail than on Usenet.

(Why some compressed formats have large sections of NULs, I have no idea.
That makes me think that either some of those compression formats are less
than ideal or the advantages of byte-shifting have been overstated.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 04 Apr 2002 08:17:44 -0800 (PST)
From: [email-masked]
Subject: RE: Content-Transfer-Encoding and yEnc
To: Dan Kohn <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Sender: [email-masked]

> I don't mean to open a rat hole here, but has the use of gzip as part of
> the yEnc encoding been seriously considered?

> I understand that a compressed CTE was considered as part of the
> original MIME architecture but that nothing could be agreed on at the
> time.  If something non-backward compatible, specific to the NNTP
> medium, requiring new code be deployed, and designed for reducing the
> size of files, it seems sort of odd not to compress the binary before
> CRLF stuffing and shifting the NULLs.

I definitely agree that a gzipped CTE would be a good idea.

				Ned

To: [email-masked]
Xref: clerew local.mime:1216
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Content-Transfer-Encoding and yEnc
Date: Thu, 4 Apr 2002 10:16:16 GMT
Lines: 71
Sender: [email-masked]

In <[email-masked]> Ian Bell <[email-masked]> writes:

>The definition of a new CTE for netnews is only the first part of the
>problem. For yEnc to be generally usable, there will need to be a
>mechanism for splitting messages and probably checksumming the split
>parts.

yEnc includes a CRC check for each part (that is good enough for checking
integrity, though MD5, which is more costly, may be needed for some other
purposes).

I believe yEnc also makes provision for stitching parts together. That
might not be such a good thing (is an encoding really the right place to
do that?) but I see that other, more appropriate, mechanisms are under
discussion in this thread.

>However, the use of message/partial for splitting yEnc messages in the
>first place hits RFC2046 Section 5.2.2.

>    "If [non-7bit] messages were encountered at a gateway into a 7bit
>    transport environment, ...  For this reason, it is specified
>    that entities of type "message/partial" must always have a
>    content-transfer-encoding of 7bit (the default)...

>The answer to Charles'  question "[would] a proposal for a CTE for
>encoding binaries within an 8bit domain stand a reasonable chance of
>being accepted by the IESG" has been quite positive. What chance is
>there of a relaxation of the above message/partial restriction being
>acceptable for use only within netnews, again assuming a technically
>sound and properly specified proposal?

Yes, in an earlier draft of Usefor, we did relax this restriction for use
withint Netnews. But then we had an attack of "MIME compliance fever" and
took it out again :-( . Of course, if Ned will let us put it back again
........

But seriously, message/partial is flawed in many ways, as I think others
on this thread have agreed. The Usefor draft is pretty lukewarm about it,
though it does suggest using the References-header so that reading agents
that do threading can at least offer the parts in the right order (so you
will see at once whether any are missing).

But, ideally, we need a mechanism with the following properties:

1. It should be possible to decode any part into its intended form without
reference to the other parts. This means each part must contain the
Content-Type, charset etc. So even if the part turns out to be a message
written in Chinese characters, you can at least read that bit and obtain
some clue as to whether you are interested enough to go looking for the
other bits. In the case of a pure binary, it may or may not be useful to
see just a bit in the middle, depending on the format (but even a clip out
of the middle of a long video could be useful if the format allows it,
which I imagine mpeg does).

2. It should be possible, given a few of the parts, to determine what
parts are missing, and to identify those parts as part of the whole if and
when they eventually turn up.

3. The mechanism should facilitate automatic reassembly of the parts.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

content-class: urn:content-classes:message
Subject: RE: Content-Transfer-Encoding and yEnc
Date: Thu, 4 Apr 2002 08:07:49 -0800
From: "Dan Kohn" <[email-masked]>
To: "Charles Lindsey" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

I don't mean to open a rat hole here, but has the use of gzip as part of
the yEnc encoding been seriously considered?

I understand that a compressed CTE was considered as part of the
original MIME architecture but that nothing could be agreed on at the
time.  If something non-backward compatible, specific to the NNTP
medium, requiring new code be deployed, and designed for reducing the
size of files, it seems sort of odd not to compress the binary before
CRLF stuffing and shifting the NULLs.

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>
Essays announced on <mailto:[email-masked]>

-----Original Message-----
From: Charles Lindsey [mailto:[email-masked]] 
Sent: Thursday, April 04, 2002 01:40
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc

In <[email-masked]> [email-masked]
writes:

>Well, yEnc as presently formulated has a failure mode where some
messages
>will grow to 2X their original size. I also don't like the fact that
>it shifts the range of almost every character. It is possible to fix
both
>of these problems and have an encoding with an upper bound guarantee
that
>leaves the original data mostly untouched.

>Just my personal take -- neither of these are necessarily showstoppers.

I think every encoding scheme has a pathological case which produces a
"2X
their original size" situation. I think such an accurrence is wildly
improbable in the yEnc case.

AIUI, the reason for the range shifting is that the NUL character tends
to
occur with more than its random probability in typical binaries (even to
the extent of huge sequences of NUL bytes). Therefore, it is desirable
that NUL itself can be transported without stuffing (or course, some
other
random character then gets saddled with that honour).

-- 
Charles H. Lindsey ---------At Home, doing my own
thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web:
http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8
3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14
A4 AB A5

From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
cc: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Thu, 04 Apr 2002 08:08:09 -0500
Sender: [email-masked]

> I think every encoding scheme has a pathological case which produces a "2X
> their original size" situation.

nah.  some of them check for blowup on a chunk-by-chunk basis, and if that 
happens, they encode that chunk in another method that is more effecient.
all compression mechanisms do have cases where they "compress" to larger
than original size, but not all can produce output that is 2x the original 
size, or even close.

Keith

To: [email-masked]
Xref: clerew local.mime:1215
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Content-Transfer-Encoding and yEnc
Date: Thu, 4 Apr 2002 09:40:06 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>Well, yEnc as presently formulated has a failure mode where some messages
>will grow to 2X their original size. I also don't like the fact that
>it shifts the range of almost every character. It is possible to fix both
>of these problems and have an encoding with an upper bound guarantee that
>leaves the original data mostly untouched.

>Just my personal take -- neither of these are necessarily showstoppers.

I think every encoding scheme has a pathological case which produces a "2X
their original size" situation. I think such an accurrence is wildly
improbable in the yEnc case.

AIUI, the reason for the range shifting is that the NUL character tends to
occur with more than its random probability in typical binaries (even to
the extent of huge sequences of NUL bytes). Therefore, it is desirable
that NUL itself can be transported without stuffing (or course, some other
random character then gets saddled with that honour).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 03 Apr 2002 14:46:50 -0500
Sender: [email-masked]

> an pointer to an Intrenet-Draft would be most appreciated.

It's not at that point yet. Feel free to help him develop one.

Date: Wed, 03 Apr 2002 13:39:23 -0600
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Keith Moore wrote:
> 
> > Yeah, that's all in there. I've posted an excerpt.
> 
> an pointer to an Intrenet-Draft would be most appreciated.

It's not at that point yet. Feel free to help him develop one.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 03 Apr 2002 14:36:28 -0500
Sender: [email-masked]

> Yeah, that's all in there. I've posted an excerpt.

an pointer to an Intrenet-Draft would be most appreciated.

Date: Wed, 03 Apr 2002 13:23:37 -0600
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Keith Moore wrote:
> 
> > MIC?
> 
> message integrity check - like MD5 or SHA-1.
> 
> (and you have to include the MIC for the whole message along with
> every fragment.  however you can also use the MIC as the message
> identifier, which might be kind of handy)

Yeah, that's all in there. I've posted an excerpt.

I'm not completely sold on some of the architectural decisions. I would
almost rather use a set of meta-headers in the first part (ala
message/partial) but this isn't for email... News has just as much risk of
losing the first part as any other part so that is somewhat risky. Also,
there is some difficulty in finding the parts in the NNTP model. You don't
want to force people to download every message in the newsgroup to locate
necessary information.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 03 Apr 2002 13:19:18 -0600
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

"Eric A. Hall" wrote:

> http://www.exit109.com/~jeremy/news/binaries/ has an essay

The relevant part of that chain is below. Comments for consideration
should probably be sent directly to the author ([email-masked]) or
posted to news.software.nntp. I'm quite sure he would appreciate any help
from MIME wizards. Rants can of course go anywhere.

-**-

application/file-fragment

The MIME type application/file-fragment specifies that a MIME body
contains a fragment of a larger file. The segment must be contiguous. A
collection of application/file-fragment entities can be created by
splitting the original, raw datastream at whatever intervals are
appropriate, and encoding the result with some transfer-encoding in order
to include it in a MIME entity. The divisions must occur at least on byte
boundaries. It may be desirable to only create divisions on boundaries of
multiples of four bytes. (The datastream should be considered an
octet-stream, not a bit-stream.) The segments in a series need not all be
the same size.

For the purposes of this specification, the term series refers to a
sequence of posts, done all at once, containing fragments of the same
file. A complete series contains all of the data from the original file,
and an incomplete series contains only some of the data (as in the case of
a partial repost). A complete series should contain the data in sequence;
that is, the data in part number two should be that immediately after the
data in part number one.

Parameters

application/file-fragment defines the following parameters.

name 
      Specifies the name of the file. This must have the same value in all
messages in a series, and is required. (However, implementations should be
careful about blindly using the specified filename when saving data to
disk.)

offset 
      Specifies the starting point of this fragment, in bytes, within the
original (unencoded) data of which this fragment is part. Offsets are
zero-based (the first byte in the file is at offset zero). This parameter
is required. 

length 
      The length, in bytes, of the unencoded data represented by this
fragment. This parameter is required. 

number 
      Where this fragment falls in sequence, in this series of posts. This
parameter is required if the fragment is being posted as part of a series,
whether or not the series is complete with regard to the original file. If
a single fragment of a file is being posted, it need not be used. (Thus,
if not present, its value should be assumed to be 1.) 

total 
      The total number of fragments in this series of posts. As with
number, this parameter is required unless this is the only fragment of the
file being posted. (Thus, if not present, its value should be assumed to
be 1.) 

id 
      Semantically identical to a message-id, this is a unique identifier
for the series of posts. It must be the same in all posts comprising a
series. This parameter is required in any series, and optional if only a
single fragment is being posted. It can be used by a client to locate
other posts in a particular series, and can be used to distinguish between
multiple posts of the same file. It should not be the same as any
message-id of any post in a series. (The reason this parameter is required
is to allow automated searching for other posts in a series by clients.) 

md5 
      Contains the MD5 checksum (as in Content-MD5) of the full,
reassembled, decoded, original file of which this fragment is a part. This
can be used by a client to verify the integrity of the final, decoded
file. It can also be used to locate different posts of the same file
(reposts) in order to fill in "gaps" in in the file, even if the reposts
were done with a different filename, and can be used to differentiate
between different files posted under the same filename. It must be the
same in all posts in a series, and is required (the reason it is required
is to allow automated searching for fragments of a desired file by
clients). 

type 
      Specifies the MIME type of the full file. This parameter is
optional, and should only be included where appropriate (that is, if the
value will not provide any useful information to a client, as in the case
of application/octet-stream, it should not be specified). If it is not
present, the file should be treated as type application/octet-stream. 

As an example of what such a header might look like, with all parameters
given (broken onto multiple lines for readability):

Content-Type: application/file-fragment; name=foo.pdf; offset=0;
length=500000; number=1; total=15; id="<[email-masked]>";
md5=7c595c4ea2befbd6f9d0658506207438; type=application/pdf

In addition to the Content-Type header, each application/file-fragment
should have a Content-MD5 header which contains an MD5 checksum of the
(decoded) data contained within that fragment. This can be used to detect
corrupt fragments, so that a client can discard them and look for another
post containing the needed data. Content-MD5 is described in RFC 1864.
Since the checksum is taken from the original data, it should survive
recoding to a different transfer-encoding.

A description of the file can be included in the Content-Description
header (defined in RFC 2045). If it is used in a series (whether complete
or incomplete with regard to the original file), it must be present in the
first post in that series, and is optional in the other segments (and
would be redundant in them). This way, a client looking for a description
of the file need only check the headers of the first post in the series.

A Content-Disposition header, if present, should not contain the filename
of the original file; the header is meant to describe only the entity
within that one message, so it does not apply for that purpose.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 03 Apr 2002 14:17:47 -0500
Sender: [email-masked]

> MIC?

message integrity check - like MD5 or SHA-1.

(and you have to include the MIC for the whole message along with
every fragment.  however you can also use the MIC as the message 
identifier, which might be kind of handy) 

> For that matter, only a fool would
> download binaries from USENET without doing all kinds of testing.

or for that matter, from the web, or ftp, or anywhere else.

but there's such a thing as making it too easy to do harm.
it's not terribly difficult to steal a tcp connection if 
you have access to the signal path.  but when sending fragmented
files over usenet, everybody has access to the signal path.

I don't know of any good way to test binaries for possible malicious
activity - there are too many ways to do harm.

Keith

Date: Wed, 03 Apr 2002 13:05:21 -0600
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Keith Moore wrote:

> I just hope they put a MIC on the contents, otherwise there's a fairly
> good way to attack (say) compiled windows programs that get posted.

MIC?

There is definitely a big security risk here, in that any posted file can
be attacked by inserting a virus/trojan/whatever into one part. However,
that's not really any different than what can happen with any binary
posted anywhere on the Internet. For that matter, only a fool would
download binaries from USENET without doing all kinds of testing.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

From: Keith Moore <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 03 Apr 2002 14:01:08 -0500
Sender: [email-masked]

> That's pretty much the approach that has been focused on in the
> news.software.nntp newsgroup, where most of this discussion originated.
> What is being considered is an application/partial (or /whatever du jour)
> where each fragment identifies a byte-range. If a large multi-part file is
> posted and a fragment is lost, anybody can repost the fragment and the gap
> will be filled.

I just hope they put a MIC on the contents, otherwise there's a fairly
good way to attack (say) compiled windows programs that get posted.

Date: Wed, 03 Apr 2002 12:35:12 -0600
From: "Eric A. Hall" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Keith Moore wrote:

> IMHO, message/partial shouldn't be used for fragmented messages that
> use 8bit encodings, a new content-type should be defined for this
> purpose.  but we shouldn't feel compelled to adhere to the "no nested
> encodings" rule for this new type, because the normal implementation
> of this type would be to decode the c-t-e for the fragment, store the
> decoded fragment in a file, and when all of the fragments are present,
> concatenate them into a message.

That's pretty much the approach that has been focused on in the
news.software.nntp newsgroup, where most of this discussion originated.
What is being considered is an application/partial (or /whatever du jour)
where each fragment identifies a byte-range. If a large multi-part file is
posted and a fragment is lost, anybody can repost the fragment and the gap
will be filled. Benefits to this approach are that any CTE can be used to
encode the fragments, application/partial will be treated as opaque by
non-conformant MUAs, and so forth. At this point it is arguing about
details and worrying about problems.

http://www.exit109.com/~jeremy/news/binaries/ has an essay

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 03 Apr 2002 09:31:10 -0800 (PST)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Keith Moore <[email-masked]>
Cc: [email-masked], Ian Bell <[email-masked]>, [email-masked]
Sender: [email-masked]

> > Hmm. This one is much, much trickier. Although one can argue that the rules for
> > message/partial are too email-centric right now, that doesn't necessarily mean
> > changing them in another domain is a good idea. Defining another type for this
> > purpose might be feasible as an alternative, but that's not necessarily good
> > either since it would fail to leverage existing support for message/partial.
> > (Assuming there's a significant amount of it.)

> IMHO, message/partial shouldn't be used for fragmented messages that use 8bit
> encodings, a new content-type should be defined for this purpose.  but we
> shouldn't feel compelled to adhere to the "no nested encodings" rule for this
> new type, because the normal implementation of this type would be to decode
> the c-t-e for the fragment, store the decoded fragment in a file, and when
> all of the fragments are present, concatenate them into a message.

Good point, and one that finesses the problem of how to downgrade a fragment
quite nicely. I like it.

> that, and the state of programing has changed a bit since the early 1990s -
> threads make it easy to handle nested encodings even without the intermediate
> step of copying to a file, and thread support is widely available these days.

Interesting. I use threads in almost all the programming I do, and it never
occurred to me to use them for this. (It is obvious how it would be done,
of course.) I guess this would be OK for simple utilities or even for most
clients, but for any sort of server, forget it. Thread overhead is just too
great to waste them on this sort of thing.

OTOH, I coded support for nested encodings sitting in sessions at the Atlanta
IETF just to see how hard it would be. And it wasn't especially difficult.

Nevertheless, I think your first argument regarding the effective separatness
of message reassembly is much more compelling. I wish it had been made
at the time message/partial was specified.

> so while I don't think we should retroactively allow nested encodings with
> existing content-types,  I don't think there should be a problem with allowing
> them in a new content-type for fragmented messages.

Yup. And given that it is the obvious way to move forward.

				Ned

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: Ian Bell <[email-masked]>, [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 03 Apr 2002 12:11:34 -0500
Sender: [email-masked]

> Hmm. This one is much, much trickier. Although one can argue that the rules for
> message/partial are too email-centric right now, that doesn't necessarily mean
> changing them in another domain is a good idea. Defining another type for this
> purpose might be feasible as an alternative, but that's not necessarily good
> either since it would fail to leverage existing support for message/partial.
> (Assuming there's a significant amount of it.)

IMHO, message/partial shouldn't be used for fragmented messages that use 8bit 
encodings, a new content-type should be defined for this purpose.  but we 
shouldn't feel compelled to adhere to the "no nested encodings" rule for this 
new type, because the normal implementation of this type would be to decode 
the c-t-e for the fragment, store the decoded fragment in a file, and when 
all of the fragments are present, concatenate them into a message.    

that, and the state of programing has changed a bit since the early 1990s - 
threads make it easy to handle nested encodings even without the intermediate 
step of copying to a file, and thread support is widely available these days.

so while I don't think we should retroactively allow nested encodings with 
existing content-types,  I don't think there should be a problem with allowing 
them in a new content-type for fragmented messages.

Keith

To: [email-masked]
Xref: clerew local.mime:1197
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Content-Transfer-Encoding and yEnc
Date: Wed, 3 Apr 2002 09:33:50 GMT
Lines: 107
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>Charles has left out a number of important relevant issues
>which have been discussed on the usefor mailing list.

Yes, it has been discussed on the Usefor list. No, there is NO intention
to include this in the current Usefor document. 

>First, the encoding method is not at issue. The issue isn't
>yEnc vs. base64, or message size, transmission bandwidth, or
>storage space.

On the contrary, that is EXACTLY the point at issue. 'yEnc' is just
another encoding, which might or might not stand alongside base64 and Q-P.

>One plan for the relevant headers runs roughly:

>MIME-Version: 1.0
>Content-Type: application/x-yyencoded
>Content-Disposition: attachment; filename="foo.yenc"
>Content-Features: ( Type="video/mpeg" )
>Content-Transfer-Encoding: 8bit

Which is a gross misuse of the Content-Type header. If the object really
is a video/mpeg, then normal MIME usage requires that the Content-Type
say so.

Yes, you might shoe horn this through existing channels by this trick, but
it leave no convenient possibility to convert it to CTE: base64 on the fly
so that it may continue its travels as a normal MIME object.

>The other proposal (Charles') runs roughly:

>MIME-Version: 1.0
>Content-Type: video/mpeg
>Content-Transfer-Encoding: yenc
>Content-Disposition: attachment

>where there also might be additional parameters and/or a
>Content-Features header.  Call this plan B.

And observe that Plan B follows both the spirit and the letter of the
overall MIME scheme.

>Plan B would require an RFC to amend RFC 2045. That RFC would
>need to specify "yenc" as a new token for use as a
>Content-Transfer-Encoding value and would need to specify
>the domain associated with that value as 8bit.  That may
>lead to interoperability problems:
>1. RFC 2045 does not specify what domain should be used for
>   unrecognized Content-Transfer-Encoding values. The domain
>   is needed by transfer agents so that they can determine
>   whether or not the message can be transported via a
>   given transport mechanism, and if so, what options (e.g.
>   8BITMIME for ESMTP) might be required. 

Yes, there is a small glitch in RFC 2045, insofar as it does not specify a
default domain to be assumed when the CTE is unregognized. A 'domain=8bit'
parameter for CTE might have been a nice idea.

> This also applies
>   to news, since there is no widely available transport for
>   binary data.

But this problem most emphatically DOES NOT apply to news, since all known
transports (notably NNTP) support at least 8bit and some (notably UUCP)
support binary. Either of these is adequate to transport yEnc. Since this
CTE is primarily intended for News, no problem arises until it comes to be
gatewayed into Email. That is a problem for the gateway to sort out, and
at least Plan B provides a possibility to turn it into something
immediately acceptable to the existing Email system. Not that I expect
such gatewaying to be commonly required.

>Another disadvantage of Plan B is that it is inherently
>inapplicable to composite media types (multipart and message)
>as only "7bit", "8bit", and "binary" values may be used with
>those. As plan A uses "8bit", it is more flexible than plan B
>in that respect.

I regard that as a considerable AD-vantage of Plan B. Composite media
types SHOULD NOT be encoded as a whole. That is not current MIME practice,
and I don't want to make it so.

>======== new but related topic ===========

>The successor to RFC 2045 should probably specify a default
>domain where an unrecognized Content-Transfer-Encoding value
>is encountered. To ensure interoperability, the worst case
>should probably be used, viz. "binary".

The obvious default is '8bit'. 

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 03 Apr 2002 02:19:39 -0800 (PST)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Ian Bell <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> On Tue, 2 Apr 2002, [email-masked] wrote:

> >>     So the question I am asking of this list is whether a proposal for a
> >>     CTE for encoding binaries within an 8bit domain would stand a
> >>     reasonable chance of being accepted by the IESG (assuming for the
> >>     moment that it was technically sound and properly specified)?
> >
> >The key, of course, is restricting this to an 8bit domain like netnews. Do
> >that and the problem is vastly simplified. And the answer is that a
> >reasonable proposal would have a good chance of being accepted.

> The definition of a new CTE for netnews is only the first part of the
> problem. For yEnc to be generally usable, there will need to be a
> mechanism for splitting messages and probably checksumming the split
> parts.

I see the connection as well as the rationale for a move to using MIME
mechanisms for postings split into multiple messages, but to some extent this
issue is orthogonal.

> The latter could be fairly simply accomplished by extending Content-MD5
> in some way so as to apply to an entire message/partial part rather than
> just leaf parts.

A message/partial part in isolation is technically a leaf, so this is really
not a problem. However, you could also put the checksums on the parts inside
the message/partial. Or both.

> However, the use of message/partial for splitting yEnc messages in the
> first place hits RFC2046 Section 5.2.2.

>     "If [non-7bit] messages were encountered at a gateway into a 7bit
>     transport environment, there would be no way to properly encode them
>     for the 7bit world, aside from waiting for all of the fragments,
>     reassembling the inner message, and then encoding the reassembled
>     data in base64 or quoted-printable.  Since it is possible that
>     different fragments might go through different gateways, even this
>     is not an acceptable solution.  For this reason, it is specified
>     that entities of type "message/partial" must always have a
>     content-transfer-encoding of 7bit (the default).  In particular,
>     even in environments that support binary or 8bit transport, the use
>     of a content- transfer-encoding of "8bit" or "binary" is explicitly
>     prohibited for MIME entities of type "message/partial". This in turn
>     implies that the inner message must not use "8bit" or "binary"
>     encoding."

> The answer to Charles'  question "[would] a proposal for a CTE for
> encoding binaries within an 8bit domain stand a reasonable chance of
> being accepted by the IESG" has been quite positive. What chance is
> there of a relaxation of the above message/partial restriction being
> acceptable for use only within netnews, again assuming a technically
> sound and properly specified proposal?

Hmm. This one is much, much trickier. Although one can argue that the rules for
message/partial are too email-centric right now, that doesn't necessarily mean
changing them in another domain is a good idea. Defining another type for this
purpose might be feasible as an alternative, but that's not necessarily good
either since it would fail to leverage existing support for message/partial.
(Assuming there's a significant amount of it.)

And it isn't possible to argue that netnews isn't gatewayed to email; we all
know it is done all the time. I suppose we could simply prohibit gatewaying of
objects of this sort unless you're willing to reassemble them as part of the
gateway process.

This one definitely needs further discussion and thought. At a minimum it is
more than I can wrap my head around at this late hour ;-)

				Ned

Date: Wed, 3 Apr 2002 10:29:18 +0100
To: [email-masked]
From: Ian Bell <[email-masked]>
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

On Tue, 2 Apr 2002, [email-masked] wrote:

>>     So the question I am asking of this list is whether a proposal for a
>>     CTE for encoding binaries within an 8bit domain would stand a
>>     reasonable chance of being accepted by the IESG (assuming for the
>>     moment that it was technically sound and properly specified)?
>
>The key, of course, is restricting this to an 8bit domain like netnews. Do
>that and the problem is vastly simplified. And the answer is that a
>reasonable proposal would have a good chance of being accepted.

The definition of a new CTE for netnews is only the first part of the
problem. For yEnc to be generally usable, there will need to be a
mechanism for splitting messages and probably checksumming the split
parts.

The latter could be fairly simply accomplished by extending Content-MD5
in some way so as to apply to an entire message/partial part rather than
just leaf parts.

However, the use of message/partial for splitting yEnc messages in the
first place hits RFC2046 Section 5.2.2.

    "If [non-7bit] messages were encountered at a gateway into a 7bit
    transport environment, there would be no way to properly encode them
    for the 7bit world, aside from waiting for all of the fragments,
    reassembling the inner message, and then encoding the reassembled
    data in base64 or quoted-printable.  Since it is possible that
    different fragments might go through different gateways, even this
    is not an acceptable solution.  For this reason, it is specified
    that entities of type "message/partial" must always have a
    content-transfer-encoding of 7bit (the default).  In particular,
    even in environments that support binary or 8bit transport, the use
    of a content- transfer-encoding of "8bit" or "binary" is explicitly
    prohibited for MIME entities of type "message/partial". This in turn
    implies that the inner message must not use "8bit" or "binary"
    encoding."

The answer to Charles'  question "[would] a proposal for a CTE for
encoding binaries within an 8bit domain stand a reasonable chance of
being accepted by the IESG" has been quite positive. What chance is
there of a relaxation of the above message/partial restriction being
acceptable for use only within netnews, again assuming a technically
sound and properly specified proposal?

-- 
Ian Bell                                           T U R N P I K E

From: Marc Mutz <[email-masked]>
Organization: KDE
To: Bruce Lilly <[email-masked]>
Subject: Re: Content-Transfer-Encoding and yEnc
Date: Wed, 3 Apr 2002 10:36:58 +0200
Cc: [email-masked]
Sender: [email-masked]

On Wednesday 03 April 2002 04:28, Bruce Lilly wrote:
<snip>
> Plan A requires no changes to existing RFCs and does not
> mandate changes to existing transfer agents.
<snip>
> Plan B would require an RFC to amend RFC 2045. That RFC would
> need to specify "yenc" as a new token for use as a
> Content-Transfer-Encoding value and would need to specify
> the domain associated with that value as 8bit.  That may
> lead to interoperability problems:
<snip>

I know rfc2049 is vague about this, but 
    (3)   Must treat any unrecognized Content-Transfer-Encoding
          as if it had a Content-Type of "application/octet-
          stream", regardless of whether or not the actual
          Content-Type is recognized.

seems to indicate to me that on encountering an unknown encoding, the dafault 
should be to treat the part as a blob, ie. cte = binary (interpreted) and ct 
= app/octet-stream.

So the problem boils down as to what extend the deployed mailers use this 
interpretation of MIME.

> No necessity for a new Content-Transfer-Encoding value has been
> established, as Plan A does everything that Plan B would do
> without the need for a new value and without requiring wholesale
> modification of transport agents.  No benefit has been established,
> as the encoding method and its compactness are not at issue.
> Indeed, there are a number of disadvantages to Plan B, not
> least of which is hindrance of interoperability.
<snip>

<insider>
I wonder how a person so picky about strict rfc compliance in mailers can 
support such ugly misuses of the MIME mechanism. ;-)
</insider>

It's bad enough that there is no mechanism to correctly label {,g,b}zip'ed 
parts, but now using the same broken behaviour for something that clearly is 
a cte.

Marc

- -- 
Marc Mutz <[email-masked]>

Date: Tue, 02 Apr 2002 23:12:06 -0800 (PST)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Bruce Lilly <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> Charles Lindsey wrote:
> >
> > There is a proposal afoot, known as "yEnc", for encoding binary Netnews
> > articles so that they will pass through 8bit transport systems, such as
> > NNTP. See <http://www.yenc.org>. The perceived benefit is that it reduces
> > the size of an article by around 30%, as compared with Base64 or uuencode,
> > and that is a useful benefit not only for transport, but also for storage
> > space on newsservers.
> [...]

> Charles has left out a number of important relevant issues
> which have been discussed on the usefor mailing list.

> First, the encoding method is not at issue. The issue isn't
> yEnc vs. base64, or message size, transmission bandwidth, or
> storage space.

> The difference of opinion boils down to the means of wrapping
> and tagging the encoded data.

> One plan for the relevant headers runs roughly:

> MIME-Version: 1.0
> Content-Type: application/x-yyencoded
> Content-Disposition: attachment; filename="foo.yenc"
> Content-Features: ( Type="video/mpeg" )
> Content-Transfer-Encoding: 8bit

> where Content-Type parameters and/or additional
> Content-Disposition parameters might be used to
> supplement the Content-Features header for the benefit
> of user agents that do not yet handle Content-Features.
> Call this (or any of the equivalent variations on the theme)
> plan A.

This is flatly prohibited by RFC 2048 section 2.2.1.

> The other proposal (Charles') runs roughly:

> MIME-Version: 1.0
> Content-Type: video/mpeg
> Content-Transfer-Encoding: yenc
> Content-Disposition: attachment

> where there also might be additional parameters and/or a
> Content-Features header.  Call this plan B.

> Plan A requires no changes to existing RFCs and does not
> mandate changes to existing transfer agents.

On the contrary, it is a gross violation of existing RFCs. See above.

> Optionally,
> the media type could be officially registered rather than
> using an experimental subtype, but that does not affect
> the gist of the plan.  Note that the encoding information
> relative to the original file which was encoded via yEnc is
> carried implicitly in the Content-Type header, while the
> Content-Transfer-Encoding header explicitly specifies the
> domain of the body content as 8bit. Content-Features and/or
> parameters used with Content-Type and/or Content-Disposition
> convey information to regenerate the original (decoded)
> file ancillary information (type and name).  Obviously,
> applications which wish to be able to handle this form of
> encoding need to be able to recognize the media type and
> features and/or parameters, however (and this is an
> important point) no transport agent need know about the
> details.

> Plan B would require an RFC to amend RFC 2045. That RFC would
> need to specify "yenc" as a new token for use as a
> Content-Transfer-Encoding value and would need to specify
> the domain associated with that value as 8bit.

Wrong again. The MIME RFCs explicitly lay the groundwork for additional
encodings to be defined and registered. There is no need to amend anything.

> That may
> lead to interoperability problems:
> 1. RFC 2045 does not specify what domain should be used for
>    unrecognized Content-Transfer-Encoding values. The domain
>    is needed by transfer agents so that they can determine
>    whether or not the message can be transported via a
>    given transport mechanism, and if so, what options (e.g.
>    8BITMIME for ESMTP) might be required.  This also applies
>    to news, since there is no widely available transport for
>    binary data.  Currently, standard Content-Transfer-Encoding
>    values of "binary" and "8bit" specify the respective domains;
>    everything else (including the default when no
>    Content-Transfer-Encoding header is present) has a 7bit
>    domain.  No doubt some transfer agents will assume a 7bit
>    domain given an unrecognized value, and interoperability
>    cannot be expected if 8bit (yEnc) data is sent over (an
>    assumed) 7bit transport mechanism.

> 2. As far as I know, there are no current implementations of
>    transport agents which recognize a Content-Transfer-Encoding
>    value of "yenc".  Leaving aside the requirement for at
>    implementations for standardization, there is the formidable
>    task of updating ALL transport agents (including embedded
>    systems) to recognize a new value and its associated encoding
>    method and domain. That task would certainly take years. That
>    is why the RFC 2045 advice should not be brushed aside casually:
> "Unlike media subtypes, a proliferation of Content-Transfer-Encoding
>    values is both undesirable and unnecessary."
> and
> "Unlike media types and subtypes, the creation of new Content-
>    Transfer-Encoding values is STRONGLY discouraged, as it seems likely
>    to hinder interoperability with little potential benefit"

The need to add support to transfer agents is certainly an issue. But
it exists in both cases; slipping something through isn't the same
as handling it properly.

Your plan A, however, requires agents be updated to understand that
content type information is now in a different place. So instead of
exploiting what amounts to a simple plugin interface in any reasonably
constructed agent, you are now talking about creating a entirely new
way of expressing type information. Not only that, but the old mechanism
has to be supported as well.

The result is a huge increase in complexity.

> No necessity for a new Content-Transfer-Encoding value has been
> established, as Plan A does everything that Plan B would do
> without the need for a new value and without requiring wholesale
> modification of transport agents.  No benefit has been established,
> as the encoding method and its compactness are not at issue.
> Indeed, there are a number of disadvantages to Plan B, not
> least of which is hindrance of interoperability.

> Another disadvantage of Plan B is that it is inherently
> inapplicable to composite media types (multipart and message)
> as only "7bit", "8bit", and "binary" values may be used with
> those. As plan A uses "8bit", it is more flexible than plan B
> in that respect.

I dispute every point in your analysis.

				Ned

Date: Tue, 02 Apr 2002 22:56:19 -0800 (PST)
From: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> There is a proposal afoot, known as "yEnc", for encoding binary Netnews
> articles so that they will pass through 8bit transport systems, such as
> NNTP. See <http://www.yenc.org>. The perceived benefit is that it reduces
> the size of an article by around 30%, as compared with Base64 or uuencode,
> and that is a useful benefit not only for transport, but also for storage
> space on newsservers.

> If you look at the spec, you will see that it is a perfectly reasonable
> encoding. Roughly speaking, octets appear as themselves, except for NUL,
> CR and LF (and '='), which are escaped by putting an '=' in front and
> adding a constant. OK, it is a bit more complicated than that, and there
> are extra Bells and Whistles such as a byte count and a CRC check, and
> provision for other extensions.

> So it could perfectly well have been formulated as a
> Content-Transfer-Encoding, but instead it is being promoted as Yet Another
> ad hoc protocol, and they have even persuaded some browser implementors to
> support it. And when asked why they are doing it that way, they say

>     "because we raised this in the MIME newsgroups, and we were told that
>     it is not possible to add new CTEs, and in any case it would take
>     years, and the Gods of the Internet would never agree to it."

> Which is as poor an excuse as they come, even though there may be a grain
> of truth behind it.

There isn't even that. I was approached about this. I explained that
it is indeed possible to define new CTEs and how to do it. But for whatever
reason I was unable to get the point across. There's a significant amount
of misunderstanding of MIME, of deployment issues, and how to manage a
transition plan out there, it seems.

> Now, if you look at RFC 2045 and RFC 2048, you will see that there ARE
> mechanisms for establishing new CTEs, either by a standards track RFC, or
> on a private basis using an x-token. BUT these mechanisms are accompanied
> by a dire warning against "The standardization of a large number of
> different transfer encodings". Which is fair enough, but does not amount
> to an absolute "NEVER" prohibition.

Exactly.

>     So the question I am asking of this list is whether a proposal for a
>     CTE for encoding binaries within an 8bit domain would stand a
>     reasonable chance of being accepted by the IESG (assuming for the
>     moment that it was technically sound and properly specified)?

The key, of course, is restricting this to an 8bit domain like netnews. Do that
and the problem is vastly simplified. And the answer is that a reasonable
proposal would have a good chance of being accepted.

It would also be possible to specify this for domains like email where
8bit is not guaranteed. But for that you probably have to have a
negotiation/downgrade mechanism.

> The benefit is a potential saving of around 30% in bandwidth and storage.
> It would be primarily aimed at Usenet (over 90% by volume of which is
> currently binaries), though obviously it could be used for Email too. It
> would work immediately over the NNTP transport, though gateways to email
> might have to downgrade it to Base64 until such time as email systems had
> caught on. But I doubt that any significant quantity of the present binary
> bulk of Usenet ever finds its way into email.

> The downside of not doing it is that it will probably go ahead on an ad
> hoc basis anyway. There is mention of including "yEnc:" in the Subject, or
> mis-using Content-Type: application/yencoded with CTE: 8bit, or even of
> resurrecting the obsolete "conversions" parameter. The mind boggles, but
> mind-boggling is unlikely to stop the browser implementors whose loyalty
> to IETF standards is tenuous at best :-( .

FWIW, such misuse of a content type is explicitly forbidden.

But as you say, the mind boggles.

> But "where will all this stop" you ask. Indeed it is difficult to see what
> further encodings would be needed once you had this one which, I reckon,
> would barely increase the size of the transported message by more that a
> couple of percent beyond a pure binary encoding. Indeed, the only other
> CTE I can envisage would be one for 'gzip', or other compression methods
> (but those do not apply to the present situation, because most image and
> audio media types are already compressed as tight as they will go).

Well, yEnc as presently formulated has a failure mode where some messages
will grow to 2X their original size. I also don't like the fact that
it shifts the range of almost every character. It is possible to fix both
of these problems and have an encoding with an upper bound guarantee that
leaves the original data mostly untouched.

Just my personal take -- neither of these are necessarily showstoppers.

				Ned

From: Keith Moore <[email-masked]>
To: Bruce Lilly <[email-masked]>
cc: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc 
Date: Wed, 03 Apr 2002 00:46:41 -0500
Sender: [email-masked]

IMHO - if you're defining a new encoding, call it an encoding.
don't bury it in a new content-type.  I don't expect news to
gateway directly to email anyway - too many differences between
the formats - so if gateways have to convert encodings at the
boundaries in addition to other conversions I don't see a big
problem with that.  

Keith

Date: Tue, 02 Apr 2002 21:28:47 -0500
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
CC: [email-masked]
Subject: Re: Content-Transfer-Encoding and yEnc
Sender: [email-masked]

Charles Lindsey wrote:
> 
> There is a proposal afoot, known as "yEnc", for encoding binary Netnews
> articles so that they will pass through 8bit transport systems, such as
> NNTP. See <http://www.yenc.org>. The perceived benefit is that it reduces
> the size of an article by around 30%, as compared with Base64 or uuencode,
> and that is a useful benefit not only for transport, but also for storage
> space on newsservers.
[...]

Charles has left out a number of important relevant issues
which have been discussed on the usefor mailing list.

First, the encoding method is not at issue. The issue isn't
yEnc vs. base64, or message size, transmission bandwidth, or
storage space.

The difference of opinion boils down to the means of wrapping
and tagging the encoded data.

One plan for the relevant headers runs roughly:

MIME-Version: 1.0
Content-Type: application/x-yyencoded
Content-Disposition: attachment; filename="foo.yenc"
Content-Features: ( Type="video/mpeg" )
Content-Transfer-Encoding: 8bit

where Content-Type parameters and/or additional
Content-Disposition parameters might be used to
supplement the Content-Features header for the benefit
of user agents that do not yet handle Content-Features.
Call this (or any of the equivalent variations on the theme)
plan A.

The other proposal (Charles') runs roughly:

MIME-Version: 1.0
Content-Type: video/mpeg
Content-Transfer-Encoding: yenc
Content-Disposition: attachment

where there also might be additional parameters and/or a
Content-Features header.  Call this plan B.

Plan A requires no changes to existing RFCs and does not
mandate changes to existing transfer agents.  Optionally,
the media type could be officially registered rather than
using an experimental subtype, but that does not affect
the gist of the plan.  Note that the encoding information
relative to the original file which was encoded via yEnc is
carried implicitly in the Content-Type header, while the
Content-Transfer-Encoding header explicitly specifies the
domain of the body content as 8bit. Content-Features and/or
parameters used with Content-Type and/or Content-Disposition
convey information to regenerate the original (decoded)
file ancillary information (type and name).  Obviously,
applications which wish to be able to handle this form of
encoding need to be able to recognize the media type and
features and/or parameters, however (and this is an
important point) no transport agent need know about the
details.

Plan B would require an RFC to amend RFC 2045. That RFC would
need to specify "yenc" as a new token for use as a
Content-Transfer-Encoding value and would need to specify
the domain associated with that value as 8bit.  That may
lead to interoperability problems:
1. RFC 2045 does not specify what domain should be used for
   unrecognized Content-Transfer-Encoding values. The domain
   is needed by transfer agents so that they can determine
   whether or not the message can be transported via a
   given transport mechanism, and if so, what options (e.g.
   8BITMIME for ESMTP) might be required.  This also applies
   to news, since there is no widely available transport for
   binary data.  Currently, standard Content-Transfer-Encoding
   values of "binary" and "8bit" specify the respective domains;
   everything else (including the default when no
   Content-Transfer-Encoding header is present) has a 7bit
   domain.  No doubt some transfer agents will assume a 7bit
   domain given an unrecognized value, and interoperability
   cannot be expected if 8bit (yEnc) data is sent over (an
   assumed) 7bit transport mechanism.
2. As far as I know, there are no current implementations of
   transport agents which recognize a Content-Transfer-Encoding
   value of "yenc".  Leaving aside the requirement for at
   implementations for standardization, there is the formidable
   task of updating ALL transport agents (including embedded
   systems) to recognize a new value and its associated encoding
   method and domain. That task would certainly take years. That
   is why the RFC 2045 advice should not be brushed aside casually:
"Unlike media subtypes, a proliferation of Content-Transfer-Encoding
   values is both undesirable and unnecessary."
and
"Unlike media types and subtypes, the creation of new Content-
   Transfer-Encoding values is STRONGLY discouraged, as it seems likely
   to hinder interoperability with little potential benefit"

No necessity for a new Content-Transfer-Encoding value has been
established, as Plan A does everything that Plan B would do
without the need for a new value and without requiring wholesale
modification of transport agents.  No benefit has been established,
as the encoding method and its compactness are not at issue.
Indeed, there are a number of disadvantages to Plan B, not
least of which is hindrance of interoperability.

Another disadvantage of Plan B is that it is inherently
inapplicable to composite media types (multipart and message)
as only "7bit", "8bit", and "binary" values may be used with
those. As plan A uses "8bit", it is more flexible than plan B
in that respect.

======== new but related topic ===========

The successor to RFC 2045 should probably specify a default
domain where an unrecognized Content-Transfer-Encoding value
is encountered. To ensure interoperability, the worst case
should probably be used, viz. "binary".

Best regards,
  Bruce Lilly

Date: Tue, 02 Apr 2002 18:09:21 -0500
From: Bruce Lilly <[email-masked]>
Organization: Bruce Lilly
To: [email-masked]
Subject: What is a header? (draft-klyne-msghdr-registry-04.txt)
Sender: [email-masked]

A definition of what constitutes a header is probably
necessary in order for a header registry to be
optimally useful. (note that I did not say "message
header" as RFC 2046 distinguishes messages from MIME
entities, and the latter also have headers)  It
should be clear what is to be registered as well as
what does not qualify for registration.  For the
moment, let's consider headers in text messages and
ignore HTTP "headers".

Certainly RFC 2822 is clear with respect to message
headers, but it does not explicitly cover MIME
entities.  The MIME RFCs were reasonably clear when
originally written, but some of the extension RFCs
have muddied the waters a bit.

Some of the murky areas:

1. RFC 1894 defines the message/delivery-status media
   type whose ''body [...] consists of one or more "fields"
   formatted according to the ABNF of RFC 822 header
   "fields"''.  So, for the purpose of the registry,
   are the per-message fields defined in 1894 considered
   header fields or not?  If so, there's the added
   complication of the per-recipient fields, which
   begin following a blank line after the per-message
   fields.  Normally a blank line separates headers
   and body; are the per-recipient fields then body text
   or should they also be considered header fields? If
   headers, then the message/delivery-status media type
   has no body as such; only headers separated into
   groups by blank lines.

   RFC 2298's message/disposition-notification is
   similar.

   If the DSN and MDN fields are to be considered header
   fields, a definition based on the location of the
   fields within a message is likely to be troublesome
   or convoluted.

2. RFC 2425 defines a media type text/directory which
   also uses structured fields.  However, the structure
   differs somewhat from RFC 822/2822 and one would not
   normally expect to find headers in a discrete text media
   type -- were it not for RFC 1892's text/rfc822-headers
   media type.

   Crafting a suitable definition that takes these cases
   into consideration is also challenging.

3. RFC 2156 defines a number of header fields for X.400
   gateway use and cautions against general use. Nevertheless,
   RFC 2421 uses Importance and Sensitivity. Supersedes is
   in widespread use in usenet news article headers and
   appears in the usefor draft, but with a subset of the 2156
   syntax.

Most of the 2156 header fields, and the 1894 and 2298
fields (and 822's "Encrypted") are not listed in Jacob Palme's
draft-palme-mailext-headers-06.txt document. [incidentally,
reference 41 points to an older version]

To: [email-masked]
Xref: clerew local.mime:1194
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Content-Transfer-Encoding and yEnc
Date: Tue, 2 Apr 2002 13:43:52 GMT
Lines: 71
Sender: [email-masked]

There is a proposal afoot, known as "yEnc", for encoding binary Netnews
articles so that they will pass through 8bit transport systems, such as
NNTP. See <http://www.yenc.org>. The perceived benefit is that it reduces
the size of an article by around 30%, as compared with Base64 or uuencode,
and that is a useful benefit not only for transport, but also for storage
space on newsservers.

If you look at the spec, you will see that it is a perfectly reasonable
encoding. Roughly speaking, octets appear as themselves, except for NUL,
CR and LF (and '='), which are escaped by putting an '=' in front and
adding a constant. OK, it is a bit more complicated than that, and there
are extra Bells and Whistles such as a byte count and a CRC check, and
provision for other extensions.

So it could perfectly well have been formulated as a
Content-Transfer-Encoding, but instead it is being promoted as Yet Another
ad hoc protocol, and they have even persuaded some browser implementors to
support it. And when asked why they are doing it that way, they say

    "because we raised this in the MIME newsgroups, and we were told that
    it is not possible to add new CTEs, and in any case it would take
    years, and the Gods of the Internet would never agree to it."

Which is as poor an excuse as they come, even though there may be a grain
of truth behind it.

Now, if you look at RFC 2045 and RFC 2048, you will see that there ARE
mechanisms for establishing new CTEs, either by a standards track RFC, or
on a private basis using an x-token. BUT these mechanisms are accompanied
by a dire warning against "The standardization of a large number of
different transfer encodings". Which is fair enough, but does not amount
to an absolute "NEVER" prohibition.

    So the question I am asking of this list is whether a proposal for a
    CTE for encoding binaries within an 8bit domain would stand a
    reasonable chance of being accepted by the IESG (assuming for the
    moment that it was technically sound and properly specified)?

The benefit is a potential saving of around 30% in bandwidth and storage.
It would be primarily aimed at Usenet (over 90% by volume of which is
currently binaries), though obviously it could be used for Email too. It
would work immediately over the NNTP transport, though gateways to email
might have to downgrade it to Base64 until such time as email systems had
caught on. But I doubt that any significant quantity of the present binary
bulk of Usenet ever finds its way into email.

The downside of not doing it is that it will probably go ahead on an ad
hoc basis anyway. There is mention of including "yEnc:" in the Subject, or
mis-using Content-Type: application/yencoded with CTE: 8bit, or even of
resurrecting the obsolete "conversions" parameter. The mind boggles, but
mind-boggling is unlikely to stop the browser implementors whose loyalty
to IETF standards is tenuous at best :-( .

But "where will all this stop" you ask. Indeed it is difficult to see what
further encodings would be needed once you had this one which, I reckon,
would barely increase the size of the transported message by more that a
couple of percent beyond a pure binary encoding. Indeed, the only other
CTE I can envisage would be one for 'gzip', or other compression methods
(but those do not apply to the present situation, because most image and
audio media types are already compressed as tight as they will go).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 01 Apr 2002 15:04:00 +0100
To: [email-masked]
From: Graham Klyne <[email-masked]>
Subject: Fwd: I-D ACTION:draft-klyne-msghdr-registry-04.txt
Sender: [email-masked]

All,

While not claiming it's to everyone's perfect satisfaction, I think we've 
paid good attention to the various issues that have been raised.  This 
revision mainly folds in the smaller issues noted here since the previous 
release.

#g
--

>To: IETF-Announce: ;
>From: [email-masked]
>Reply-to: [email-masked]
>Subject: I-D ACTION:draft-klyne-msghdr-registry-04.txt
>Date: Thu, 28 Mar 2002 07:08:31 -0500
>Sender: [email-masked]
>
>A New Internet-Draft is available from the on-line Internet-Drafts 
>directories.
>
>
>         Title           : Registration procedures for message header fields
>         Author(s)       : G. Klyne et al.
>         Filename        : draft-klyne-msghdr-registry-04.txt
>         Pages           : 20
>         Date            : 27-Mar-02
>
>This specification defines registration procedures for the message
>header fields used by Internet mail, HTTP, news and other
>applications.
>
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-klyne-msghdr-registry-04.txt
>
>To remove yourself from the IETF Announcement list, send a message to
>ietf-announce-request with the word unsubscribe in the body of the message.
>
>Internet-Drafts are also available by anonymous FTP. Login with the username
>"anonymous" and a password of your e-mail address. After logging in,
>type "cd internet-drafts" and then
>         "get draft-klyne-msghdr-registry-04.txt".
>
>A list of Internet-Drafts directories can be found in
>http://www.ietf.org/shadow.html
>or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>
>Internet-Drafts can also be obtained by e-mail.
>
>Send a message to:
>         [email-masked].
>In the body type:
>         "FILE /internet-drafts/draft-klyne-msghdr-registry-04.txt".
>
>NOTE:   The mail server at ietf.org can return the document in
>         MIME-encoded form by using the "mpack" utility.  To use this
>         feature, insert the command "ENCODING mime" before the "FILE"
>         command.  To decode the response(s), you will need "munpack" or
>         a MIME-compliant mail reader.  Different MIME-compliant mail readers
>         exhibit different behavior, especially when dealing with
>         "multipart" MIME messages (i.e. documents which have been split
>         up into multiple messages), so check your local documentation on
>         how to manipulate these messages.
>
>
>Below is the data which will enable a MIME compliant mail reader
>implementation to automatically retrieve the ASCII version of the
>Internet-Draft.
>Content-Type: text/plain
>Content-ID:     <[email-masked]>
>
>ENCODING mime
>FILE /internet-drafts/draft-klyne-msghdr-registry-04.txt
>
><ftp://ftp.ietf.org/internet-drafts/draft-klyne-msghdr-registry-04.txt>

-------------------
Graham Klyne
<[email-masked]>