Subject: Re: UTF-8 in headers
From: Ned Freed <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Fri, 29 Jan 1999 09:42:14 -0800 (PST)

> In <[email-masked]> Chris Newman <[email-masked]> writes:

> >On Sun, 24 Jan 1999, Charles Lindsey wrote:

> >> As regards language names, I would presume that for bodies this is a MIME
> >> matter. Does there exist (or is there a proposal for) a Content-Language:
> >> header, or an equivalent parameter for one of the other MIME headers?

> >Language in MIME headers is defined in RFC 2231 as an extension to RFC
> >2047.

> Yes, but the question I had in mind was how to specify the language being
> used in a body part. Suppose I am writing my body in french. Do I put in a
> header like
> 	Content-Type: text/plain; charset=iso-8859-1; language=FR
> I was not aware of such a language parameter in RFC-2046.

This is done with a Content-Language field. See RFC1766 for details.

Note, however, that this is only a solution for body information (and not
necessarily even text), not text in header fields.

> I have now looked at RFC 2231, and what a can of worms! Well it does give
> a way to specify charsets and languages in Mime parameters, and adds
> languages to RFC 2047, though the syntactic sugar is not perticularly
> sweet :-( .

> However, my concern is that it seems to provide yet another way to
> downgrade 8bits to 7bits. It seems that, for downgrading from 8bit to 7bit
> when headers are written in UTF-8 (as now proposed for news, and soon to
> be proposed for email) we have to distinguish 3 cases:

> 1. Comments (...), Phrases (e.g. as in "Charles H. Lindsey" <chl@...>),
> Unstructured text (as in Subject:s), "extension message header fields"
> (not quite sure what that means) and all "X-" headers:
> 	downgrading is by RFC 2047

> 2. Parameters of Mime headers (e.g. Content-Distribution: attachment;
> filename="some-name-written-in-funny-characters")
> 	downgrading is by RFC 2231

> 3. All other cases
> 	there is no downgrading mechanism specified yet

> Now, within (3.) we can distinguish
> 	a) Protocol words. I would be happy to see these remain forever in
> 	   ASCII.
> 	b) Parameters as in "keyword=parameter" that are part of headers
> 	   that are not Mime headers, but have borrowed the Mime syntax.
> 	c) Other 'tokens' in assorted headers, including those not
> 	   invented yet. Newsgroup-names in news is the obvious example
> 	   here, but is fixable because we know about it. The worrying
> 	   ones are the ones we do not know about.

> So I can see a great danger that the grand canonical method of downgrading
> UTF-8 headers (which is subject of this thread) is likely to degenerate
> into a large collection of special cases, all done differently. And I do
> not see any immediate clean answer to this :-( .

There is no clean answer short of writing out explicit rules, which is what
I've been saying from the start has to be done.

				Ned

To: [email-masked]
Xref: clerew local.mime:23
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Fri, 29 Jan 1999 16:21:45 GMT
Lines: 59
Sender: [email-masked]

In <[email-masked]> Chris Newman <[email-masked]> writes:

>On Sun, 24 Jan 1999, Charles Lindsey wrote:

>> As regards language names, I would presume that for bodies this is a MIME
>> matter. Does there exist (or is there a proposal for) a Content-Language:
>> header, or an equivalent parameter for one of the other MIME headers?

>Language in MIME headers is defined in RFC 2231 as an extension to RFC
>2047.

Yes, but the question I had in mind was how to specify the language being
used in a body part. Suppose I am writing my body in french. Do I put in a
header like
	Content-Type: text/plain; charset=iso-8859-1; language=FR
I was not aware of such a language parameter in RFC-2046.

I have now looked at RFC 2231, and what a can of worms! Well it does give
a way to specify charsets and languages in Mime parameters, and adds
languages to RFC 2047, though the syntactic sugar is not perticularly
sweet :-( .

However, my concern is that it seems to provide yet another way to
downgrade 8bits to 7bits. It seems that, for downgrading from 8bit to 7bit
when headers are written in UTF-8 (as now proposed for news, and soon to
be proposed for email) we have to distinguish 3 cases:

1. Comments (...), Phrases (e.g. as in "Charles H. Lindsey" <chl@...>),
Unstructured text (as in Subject:s), "extension message header fields"
(not quite sure what that means) and all "X-" headers:
	downgrading is by RFC 2047

2. Parameters of Mime headers (e.g. Content-Distribution: attachment;
filename="some-name-written-in-funny-characters")
	downgrading is by RFC 2231

3. All other cases
	there is no downgrading mechanism specified yet

Now, within (3.) we can distinguish
	a) Protocol words. I would be happy to see these remain forever in
	   ASCII.
	b) Parameters as in "keyword=parameter" that are part of headers
	   that are not Mime headers, but have borrowed the Mime syntax.
	c) Other 'tokens' in assorted headers, including those not
	   invented yet. Newsgroup-names in news is the obvious example
	   here, but is fixable because we know about it. The worrying
	   ones are the ones we do not know about.

So I can see a great danger that the grand canonical method of downgrading
UTF-8 headers (which is subject of this thread) is likely to degenerate
into a large collection of special cases, all done differently. And I do
not see any immediate clean answer to this :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Thu, 28 Jan 1999 10:30:27 -0800 (PST)
From: Chris Newman <[email-masked]>
Subject: Re: UTF-8 in headers
Originator-info: login-id=chris; server=THOR.INNOSOFT.COM
To: [email-masked]
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Sender: [email-masked]

On Thu, 28 Jan 1999 [email-masked] wrote:
> On Wed, 27 Jan 1999 15:05:15 PST, Chris Newman said:
> > It's RFC 2482 or RFC 2231.  Take your pick.  Of course, language tags are 
> > a MUST accept, MAY generate case.
> 
> Umm.. I admit I was only paying half attention here - is this a
> "MUST accept and process correctly" or a "MUST accept, syntax check,
> and then discard if you're not up to handling it"?
> 
> In other words, are they required to handled, or merely not flagged
> as an error?  And if "accepted and ignored", does that have any implications
> for funky output on the screen?  Or does it just mean we're going to
> discard the information on (say) whether it's the Korean or Japanese
> variant of a given glyph, and for the iso8859-1 limited people it will
> look the same either way?

I'd say a basic implementation MUST be capable of handling received
language tags in such a way that they don't intrude into the user's
experience.  A more advanced implementation might provide a way to see the
language tag or act on it in terms of glyph rendering or text-to-speech
algorithm (or both).

		- Chris

To: Chris Newman <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: UTF-8 in headers 
From: [email-masked]
Date: Thu, 28 Jan 1999 09:59:06 -0500
Sender: [email-masked]

--==_Exmh_-1818933092P
Content-Type: text/plain; charset=us-ascii

On Wed, 27 Jan 1999 15:05:15 PST, Chris Newman said:
> It's RFC 2482 or RFC 2231.  Take your pick.  Of course, language tags are 
> a MUST accept, MAY generate case.

Umm.. I admit I was only paying half attention here - is this a
"MUST accept and process correctly" or a "MUST accept, syntax check,
and then discard if you're not up to handling it"?

In other words, are they required to handled, or merely not flagged
as an error?  And if "accepted and ignored", does that have any implications
for funky output on the screen?  Or does it just mean we're going to
discard the information on (say) whether it's the Korean or Japanese
variant of a given glyph, and for the iso8859-1 limited people it will
look the same either way?

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_-1818933092P
Content-Type: application/pgp-signature

--==_Exmh_-1818933092P--

Date: Wed, 27 Jan 1999 15:05:15 -0800 (PST)
From: Chris Newman <[email-masked]>
Subject: Re: UTF-8 in headers
Originator-info: login-id=chris; server=THOR.INNOSOFT.COM
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On Sun, 24 Jan 1999, Charles Lindsey wrote:
> Thanks. I have looked at those two RFCs now. I see that there is provision
> for certain "names" to be declared a part of the protocol, and that would
> apply to all the things I said should be in ASCII (Header names, and the
> like). It seems we need a wording somewhere to draw attention to that, but
> otherwise I think we comply with that bit.

Yes.

> As regards language names, I would presume that for bodies this is a MIME
> matter. Does there exist (or is there a proposal for) a Content-Language:
> header, or an equivalent parameter for one of the other MIME headers?

Language in MIME headers is defined in RFC 2231 as an extension to RFC
2047.

> 1. A Language header. That would be simplest, but not so suitable for the
> man who wants to give his real name (in a From:) in Chinese, his Subject:
> in Arabic, and his Keywords: in Hebrew. Should we care?

This is not an option given headers can include more than one human name,
and human names are probably the touchiest subject for correct language
labelling.  In addition, it requires two-pass parsing of headers.

> 2. Use RFC 2482 (language tags embedded in UTF-8 text). Extremely
> flexible, but would undoubtedly raise howls of protest from users whose
> existing agents saw them as a sequence of garbage characters (people who
> read news can get exceedingly irate when shown such things - as witness
> the railings against HTML in news, or even against any form of Mime).

It's RFC 2482 or RFC 2231.  Take your pick.  Of course, language tags are 
a MUST accept, MAY generate case.

> >P.S. I think your client has a bug such that when you both post to the
> >local.mime newsgroup and email to the mailing list, your client omits the
> >"To" header from the email message
> 
> Is it OK now?

Working fine.

		- Chris

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: UTF-8 in headers 
Date: Tue, 26 Jan 1999 13:24:25 -0500
Sender: [email-masked]

> >> 2. Use RFC 2482 (language tags embedded in UTF-8 text). Extremely
> >> flexible, but would undoubtedly raise howls of protest from users whose
> >> existing agents saw them as a sequence of garbage characters (people who
> >> read news can get exceedingly irate when shown such things - as witness
> >> the railings against HTML in news, or even against any form of Mime).
>  
> >essentially nobody's existing UA supports UTF-8, so if you're
> >using UTF-8 anyway, including language tags in UTF-8 doesn't make 
> >the situation much worse.
> 
> Well there are lots of existing UAs that support a well-known subset of
> UTF-8 :-). Their users will indoubtedly complain when they see
> gobbledegook appearing. But that is just a fact of life (or will be) :-( .
> 
> However, when _real_ UTF-8 agents start to appear, they will support only
> a subset of the available characters sets. And in particular, many will
> not recognise the RFC 2482 stuff. So you get more users complaining about
> more gobbledegook.

I'm curious why this should be the case.  If we define the spec for
utf-8-enhanced mail/news in such a way that support RFC 2482 is 
required (at least in the sense that you must not display the
langugage tags as gobbledegook), why would large numbers of 
"real" UAs fail to implement the spec?

> I think the real point is that inclusion of language information is a MAY.
> You do not include it because you have bought this nice new toy that
> allows it, or because the latest Billyware does it by default. You use it
> when there are useful clues which could indeed be helpful to displays which
> know about that particular language. Which means you will hardly ever need
> to use it for the language EN.

I disagree - you may very well need to use it for the language EN,
not so much for display, than for text-to-speech converters.
(why should they assume English pronounciation as a default?)

*I* think the real point is that you should not lie about the language.
If you don't have reliable knowledge of the language, you should leave 
off the language tag.

Keith

To: [email-masked]
Xref: clerew local.mime:18
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Mon, 25 Jan 1999 17:19:39 GMT
Lines: 41
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>> 2. Use RFC 2482 (language tags embedded in UTF-8 text). Extremely
>> flexible, but would undoubtedly raise howls of protest from users whose
>> existing agents saw them as a sequence of garbage characters (people who
>> read news can get exceedingly irate when shown such things - as witness
>> the railings against HTML in news, or even against any form of Mime).
 
>essentially nobody's existing UA supports UTF-8, so if you're
>using UTF-8 anyway, including language tags in UTF-8 doesn't make 
>the situation much worse.

Well there are lots of existing UAs that support a well-known subset of
UTF-8 :-). Their users will indoubtedly complain when they see
gobbledegook appearing. But that is just a fact of life (or will be) :-( .

However, when _real_ UTF-8 agents start to appear, they will support only
a subset of the available characters sets. And in particular, many will
not recognise the RFC 2482 stuff. So you get more users complaining about
more gobbledegook.

I think the real point is that inclusion of language information is a MAY.
You do not include it because you have bought this nice new toy that
allows it, or because the latest Billyware does it by default. You use it
when there are useful clues which could indeed be helpful to displays which
know about that particular language. Which means you will hardly ever need
to use it for the language EN.

>I imagine that lots of clients will fail to include the language
>tags -- and that's probably better than insisting that the clients
>include tags which they're likely to get wrong.  the main thing
>is to make sure there's a provision for language tags.

Yes, that sounds about right.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

From: Keith Moore <[email-masked]>
To: [email-masked] (Charles Lindsey)
cc: [email-masked]
Subject: Re: UTF-8 in headers 
Date: Sun, 24 Jan 1999 13:00:22 -0500
Sender: [email-masked]

> As to the language of headers, I can see two possibilities:
> 
> 1. A Language header. That would be simplest, but not so suitable for the
> man who wants to give his real name (in a From:) in Chinese, his Subject:
> in Arabic, and his Keywords: in Hebrew. Should we care?

yes.  If I'm replying to a message with a subject in language A, but 
my name is in language B, we need separate language tags for each.

more generally, in an email message which can have names for
each from address and each recipient,  the names may all be
in different languages.  I realize we're just talking about
news but eventually we will have to addresses these issues
for email and we may as well get it right now.

> 2. Use RFC 2482 (language tags embedded in UTF-8 text). Extremely
> flexible, but would undoubtedly raise howls of protest from users whose
> existing agents saw them as a sequence of garbage characters (people who
> read news can get exceedingly irate when shown such things - as witness
> the railings against HTML in news, or even against any form of Mime).

essentially nobody's existing UA supports UTF-8, so if you're
using UTF-8 anyway, including language tags in UTF-8 doesn't make 
the situation much worse.

> AFAICS the only reason why a newsreader would care about knowing the
> language would be in deciding whether to display the characters
> left-to-right or right-to-left. 

that's not the reason.  the newsreader needs to use different
glyphs to display certain Unicode characters depending on the language 
used.  also, text-to-speech converters need language information to know 
how to pronounce the words (but even this doesn't work for all languages).

I imagine that lots of clients will fail to include the language
tags -- and that's probably better than insisting that the clients
include tags which they're likely to get wrong.  the main thing
is to make sure there's a provision for language tags.

> >Any news client which is also an SMTP client, MUST support RFC 2047
> >encoding if it uses non-ASCII characters in any headers sent over SMTP. 
> >If a "Newsgroups" header is included in an SMTP message, it MUST use RFC
> >2047 encoding for UTF-8 characters.
> 
> I think that is a gatewaying issue. What I would like to see is that if
> you downgrade to RFC2047-charset=utf-8, you MUST upgrade to UTF-8
> if/when it comes back into the news system. In particular, you do not
> downgrade to anything other than utf-8 (even if you think you know how)
> and you do not attempt to upgrade anything other than utf-8 (even if you
> think you know how - that is for users agents only when they are ready to
> display). The point is that UTF-8 <-> RFC2047-charset=utf-8 is a simple
> algorithm. Anything else may require knowledge which not all agents
> possess.

this makes sense to me.  people might want to convert 2047 from
utf-8 to other charsets, but they should only do so for mail
to be delivered locally, not for messages that might be gatewayed
back into news.

(there really needs to be a standard for mail<>news gateways anyway,
to address lots of other issues besides format conversion)

Keith

To: [email-masked]
Xref: clerew local.mime:16
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: UTF-8 in headers
Date: Sun, 24 Jan 1999 12:26:37 GMT
Lines: 107
Sender: [email-masked]

In <[email-masked]> Chris Newman <[email-masked]> writes:

>What you have is mostly right, but it's only a start to the complete
>problem.  You'll note in RFC 2277 there's a requirement for language
>labelling of i18n text.  You'll probably have to use RFC 2482 and define
>all the rules for default language and when a language reset happens.

Thanks. I have looked at those two RFCs now. I see that there is provision
for certain "names" to be declared a part of the protocol, and that would
apply to all the things I said should be in ASCII (Header names, and the
like). It seems we need a wording somewhere to draw attention to that, but
otherwise I think we comply with that bit.

As regards language names, I would presume that for bodies this is a MIME
matter. Does there exist (or is there a proposal for) a Content-Language:
header, or an equivalent parameter for one of the other MIME headers?

As to the language of headers, I can see two possibilities:

1. A Language header. That would be simplest, but not so suitable for the
man who wants to give his real name (in a From:) in Chinese, his Subject:
in Arabic, and his Keywords: in Hebrew. Should we care?

2. Use RFC 2482 (language tags embedded in UTF-8 text). Extremely
flexible, but would undoubtedly raise howls of protest from users whose
existing agents saw them as a sequence of garbage characters (people who
read news can get exceedingly irate when shown such things - as witness
the railings against HTML in news, or even against any form of Mime).

I think I prefer 1. so far, but clearly this needs to be looked at in both
mail and news circles. Opinions anyone?

AFAICS the only reason why a newsreader would care about knowing the
language would be in deciding whether to display the characters
left-to-right or right-to-left. Or is that to be determined by the
charset?  Unicode seems confused on that issue.

>On Wed, 20 Jan 1999, Charles Lindsey wrote:
>> 2. Header-names are strictly ascii (in fact, the only characters allowed
>> are ALPHA / DIGIT / "-", which is more restrictive than DRUMS).

>I'd prefer if you followed the rules from the Message Format draft.

Regard it as a declaration by the news community that they do not intend
to invent headers outside of those characters. I doubt the mail community
does either, but if they do they would be accepted (news is committed to
accepting, and usually ignoring, any properly-defined mail header).

>> 7. Tokens can use full UTF-8 (but that probably needs reviewing).

>Which tokens?

'token' is defined in RFC2045 so, for example, in "charset=iso8859-1" both
"charset" and "iso8859-1" are tokens. The question is whether some day
Mime might allow non-ascii characters in such tokens. That is for Mime to
decide. The only token so far defined in news is for our new User-Agent:
header, in which the name of the agent (e.g. "Mozilla") can use non-ascii
characters if it wants (no harm done, beacuse it is not a protocol word).
As I said, our text on that issue probably needs reviewing.

>In some of these cases, it may be acceptable to drop the
>`case-insensitive' rules.  Your 450kB number is misleading.  The 450kB
>table is a US-ASCII form of a fairly complete table of several character
>attributes and character names for Unicode.

Yes, I knew I was overstating the argument, but even a bitmap would be
quite large.

> Compressed, it's only 70kB
>and for case conversion you probably only need a fraction of that.  So I
>suspect the case-conversion table would be of negligable size in practice. 
>The hard part is not the size of the table, but the fact that it has to be
>periodically updated as new characters are added to Unicode so it's a
>nasty maintenance issue.

Yes, that is the killer.

>> Indeed, in the Newgroups header it is a definite MUST NOT be used. Granted
>> it may have to be used when downgrading to mail, but in that case it would
>> have to be restored on the upgrade.

>Any news client which is also an SMTP client, MUST support RFC 2047
>encoding if it uses non-ASCII characters in any headers sent over SMTP. 
>If a "Newsgroups" header is included in an SMTP message, it MUST use RFC
>2047 encoding for UTF-8 characters.

I think that is a gatewaying issue. What I would like to see is that if
you downgrade to RFC2047-charset=utf-8, you MUST upgrade to UTF-8
if/when it comes back into the news system. In particular, you do not
downgrade to anything other than utf-8 (even if you think you know how)
and you do not attempt to upgrade anything other than utf-8 (even if you
think you know how - that is for users agents only when they are ready to
display). The point is that UTF-8 <-> RFC2047-charset=utf-8 is a simple
algorithm. Anything else may require knowledge which not all agents
possess.

>P.S. I think your client has a bug such that when you both post to the
>local.mime newsgroup and email to the mailing list, your client omits the
>"To" header from the email message

Is it OK now?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Fri, 22 Jan 1999 09:24:33 -0800 (PST)
From: Chris Newman <[email-masked]>
Subject: Re: UTF-8 in headers
Originator-info: login-id=chris; server=THOR.INNOSOFT.COM
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

What you have is mostly right, but it's only a start to the complete
problem.  You'll note in RFC 2277 there's a requirement for language
labelling of i18n text.  You'll probably have to use RFC 2482 and define
all the rules for default language and when a language reset happens.

On Wed, 20 Jan 1999, Charles Lindsey wrote:
> 2. Header-names are strictly ascii (in fact, the only characters allowed
> are ALPHA / DIGIT / "-", which is more restrictive than DRUMS).

I'd prefer if you followed the rules from the Message Format draft.

> 7. Tokens can use full UTF-8 (but that probably needs reviewing).

Which tokens?

> One of the problems with working with UTF-8 is that some of the character
> sets make no distinction between upper- and lower-case letters, and some
> have an extra title-case, whatever that might mean. But worse, there is no
> algorithmic method of converting upper- to lower-case; it can only be
> done, in general, by table lookup, and the table is 450kB in length. So
> you cannot allow UTF-8 in any place where some token is said to be
> 'case-insensitive' and if it is then necessary for agents to be able to
> detect and act on it.

In some of these cases, it may be acceptable to drop the
`case-insensitive' rules.  Your 450kB number is misleading.  The 450kB
table is a US-ASCII form of a fairly complete table of several character
attributes and character names for Unicode.  Compressed, it's only 70kB
and for case conversion you probably only need a fraction of that.  So I
suspect the case-conversion table would be of negligable size in practice. 
The hard part is not the size of the table, but the fact that it has to be
periodically updated as new characters are added to Unicode so it's a
nasty maintenance issue.

> As regards RFC-2047, it is a SHOULD accept, but SHOULD NOT generate.

For news, this is a practical compromise.  I might add that for a news
client which is also an IMAP or POP client, RFC 2047 should be a MUST
accept.

> Indeed, in the Newgroups header it is a definite MUST NOT be used. Granted
> it may have to be used when downgrading to mail, but in that case it would
> have to be restored on the upgrade.

Any news client which is also an SMTP client, MUST support RFC 2047
encoding if it uses non-ASCII characters in any headers sent over SMTP. 
If a "Newsgroups" header is included in an SMTP message, it MUST use RFC
2047 encoding for UTF-8 characters.

		- Chris

P.S. I think your client has a bug such that when you both post to the
local.mime newsgroup and email to the mailing list, your client omits the
"To" header from the email message -- thus the message appears to come
directly to me rather than to the list.  I thus had to manually add the
"[email-masked]" address to the headers on reply.  If you're using a
news client and newsgroup, it'd be a bug in your news->email gateway.

Xref: clerew local.mime:14
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: UTF-8 in headers
Date: Wed, 20 Jan 1999 14:41:59 GMT
Lines: 87
Apparently-To: [email-masked]
Sender: [email-masked]

In <[email-masked]> Charles Lindsey <[email-masked]> writes:

Newman:
>> >* Create rules for UTF-8 headers and downgrading thereof.
>> 
Lindsey:
>> We have worked this out for news. Would you like me to post an outline of
>> what we have done here?

Newman:
>Sure.

OK. So here is a short description of UTF-8 in headers in the present
USEFOR draft. I can privide a fuller description, complete with all the
syntax, if you really want it, but I doubt that would be useful just yet.

1. All headers are in the character set UTF-8. But, since US-ASCII is a
strict subset of UTF-8, anything that is permissable at the present time
remains so. However, that is not to say that Non-Ascii characters are now
to be allowed absolutely anywhere - only where the syntax specifically
allows them.

2. Header-names are strictly ascii (in fact, the only characters allowed
are ALPHA / DIGIT / "-", which is more restrictive than DRUMS).

3. Addr-specs are strictly ascii. If ever the DNS changes to allow UTF-8
in domain names, that might change, but I guess that is a looong way off.
OTOH phrases can use the full UTF-8, so a person's "real name" in an
address can be in Greek/whatever. Actually, we currently allow full UTF-8
in any quoted-string (qtext), but I think that needs fixing.

4. Message-IDs are strictly ascii, wherever they occur. Likewise Dates.

5. Comments (ctext) can use full UTF-8 (but comments SHOULD NOT be used
in news except in the few places that are allowed at present - they MUST
be accepted, of course).

6. Subject, Keywords, Summary, etc allow full UTF-8.

7. Tokens can use full UTF-8 (but that probably needs reviewing).

8. Newsgroup-names can use full UTF-8. Indeed, this was our reason for
admitting UTF-8 in the first place. The Scandinavians wanted to use their
extra characters in newsgroup names, and were all for making iso-8859-1
the default, but we managed to talk them into UTF-8 instead.

9. The Path header is strict ascii.

One of the problems with working with UTF-8 is that some of the character
sets make no distinction between upper- and lower-case letters, and some
have an extra title-case, whatever that might mean. But worse, there is no
algorithmic method of converting upper- to lower-case; it can only be
done, in general, by table lookup, and the table is 450kB in length. So
you cannot allow UTF-8 in any place where some token is said to be
'case-insensitive' and if it is then necessary for agents to be able to
detect and act on it.

Also, one cannot use UTF-8 in situations where transports (that is
relayers and servers in the news context) need to be able to decode and
understand UTF-8 characters (we get away with it in the Newsgroups header
only because the UTF-8 characters are themselves regarded as the canonical
form of the newsgroup-name). The most that a transport agent can be
expected to do is to convert UTF-8 characters into some escaped notation
so that the admin-guy can look at them for diagnostic purposes.

User agents, OTOH, do need to know about UTF-8 characters so that they can
display them or generate them from the keyboard. But there is no need for
every user agent to understand all characters. If you want to read Chinese
newsgroups, then you better buy yourself a user agent that understands
that part of the character space.

As regards RFC-2047, it is a SHOULD accept, but SHOULD NOT generate.
Indeed, in the Newgroups header it is a definite MUST NOT be used. Granted
it may have to be used when downgrading to mail, but in that case it would
have to be restored on the upgrade.

HTH

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Fri, 15 Jan 1999 08:43:36 -0800 (PST)
From: List Manager of ietf-822 <[email-masked]>
To: [email-masked]
Subject: How to be removed from this list
Sender: [email-masked]

Greetings again. Occasionally, people will sign up for a mailing list and
forget how to be removed. This message is a reminder for those folks.

First off, when you subscribe to a mailing list, you almost always get a
first message from the list owner telling you about the mailing list, and
explaining how to unsubscribe. It is always a good idea to keep those
messages, since you never know when you will need to unsubscribe. This is
particularly useful when you change email addresses, because it is difficult
to unsubscribe from a list after you have a different mailing address.

In the case of this list, the method to unsubscribe is to send a message
to:
     [email-masked]
with the single word:
     unsubscribe
in the body of the message. This is the same as it always has been.

To make this easier for you, I have crafted this message so that you should
be able to simply reply to this message, and the reply address should be
[email-masked] (although some mail clients screw this up...).
Remove everything from the body of the reply, and put in the single word:
     unsubscribe

If you have tried this method, and the mailing list software won't let you
unsubscribe, it is probably because your address has changed. In that case,
please send a message to [email-masked] stating which list (or lists) you
want to unsubscribe from, and what you think your previous address was.
There is a human (that's me!) who will then try to take care of your
request, often within a few days.

--Paul Hoffman, Director
--Internet Mail Consortium

Date: Wed, 13 Jan 1999 14:44:42 GMT
From: Charles Lindsey <[email-masked]>
To: [email-masked]
Subject: Application of Mime to Netnews
Content-MD5: QnvFjdjY0ahqVQbJTwUbsw==
Sender: [email-masked]

I recently offered to post here the current working USEFOR draft on
Mime, and Ned Freed has asked me to go ahead and do it, so here it is.

Note that it is mostly free-standing, but there are some reliances
on other parts of the USEFOR draft that might be slightly perplexing
at first sight (notably the mention of "agencies having the proper
authority" and "matters of policy" which actually have very specific
definitions).

Observe that I have written a Health Warning around the
application/news-message bit to indicate that it (or something to
replace it) is a matter of current debate.

6.x MIME headers

6.x.1 Syntax

        The following headers, as defined within [RFC-2045] and its
        extensions, may be used within articles conforming to this
        Document.

             MIME-Version:
             Content-Type:
             Content-Transfer-Encoding:
             Content-ID:
             Content-Description:
             Content-Disposition:
             Content-MD5:

        Insofar as the syntax for these headers, as given in [RFC-
        2045], does not specify precisely where whitespace and
        comments may occur (whether in the form of WS, FWS or CFWS),
        the usage defined in this Document, and failing that in
        [MESSFOR], and failing that in [RFC-822] MUST be followed. In
        particular, there MUST NOT be any WS between a header-name and
        the following colon and there MUST be a SPACE following that
        colon.

        The meaning of the various MIME headers is as defined in [RFC-
        2045] and [RFC-2046], and in extensions registered in
        accordance with [RFC-2048]. However, their usage is restricted
        as described in the following sections.

6.x.2 Content-Transfer-Encoding

        Posting agents SHOULD specify "Content-Transfer-Encoding:
        8bit" for all articles not written in pure ASCII and not
        requiring full binary. They MAY use "8bit" encoding even when
        "7bit" encoding would have sufficed. They SHOULD specify
        "base64" when the content type implies binary (i.e. content
        intended for machine, rather than human, consumption).

        NOTE: If a future extension to the MIME standards were to
        provide a more compact encoding of binary suited to transport
        over an 8bit channel, it could be considered as an alternative
        to base64 once it had gained widespread acceptance.

        Posting agents SHOULD NOT specify encoding "quoted-printable",
        but reading agents MUST interpret that encoding correctly.
        Encoding "binary" MUST NOT be used (except in closed networks
        with alternative transport arrangements) because this Document
        does not mandate a transport mechanism that could support it.

        Injecting and relaying agents MUST NOT change the encoding of
        articles passed to them. Gateways SHOULD ONLY change the
        encoding if absolutely necessary.

6.x.3 Content-Type

        There exist many content types which, whilst being technically
        unexceptionable, are politically undesirable. It is therefore
        anticipated that agencies having the proper authority will, as
        a matter of policy, place restrictions upon the content types
        to be used within particular hierarchies or particular groups.
        In the absence of such explicit policies, the following
        defaults are provided as an indication of current best
        practice.

        The Content-Type: "text/plain" is the default type for any
        news article. Other text types (such as "HTML") are thus
        inappropriate unless and until policy or custom has
        established otherwise.

        In the event that such other text types do get used then, for
        the benefit of readers who see it only in its transmitted
        form, the material SHOULD be "pretty-printed" so as to keep it
        (including any sequences which control its layout or style)
        within the recommendations and limits on line lengths set out
        in section 4.6, and so as to keep such sequences, as far as is
        possible, separate from the meaningful text.

        Content-Types requiring special processing for their display,
        such as "application", "image", "audio", "video" and
        "multipart/related" are deprecated except in groups
        specifically intended (by policy or custom) to include them.
        Exceptionally, those application types defined in [RFC-2015]
        and elsewhere for use within "multipart/signed" articles, and
        the type "application/pgp-keys" (or other similar types
        containing digital certificates) may be used freely.

        Reading agents SHOULD NOT, unless explicitly configured
        otherwise, act automatically on Application types which could
        change the state of that agent (e.g. by writing or modifying
        files), except in the case of those prescribed for use in
        control messages (6.5).

        The Content-Type "message/partial" MAY be used to split a long
        news article into several smaller ones, but this usage is
        deprecated on the grounds that modern transport agents should
        have no difficulty in handling articles of arbitrary length.

        IF this feature is used, then the "id" parameter SHOULD be in
        the form of a unique message-id (but different from the
        Message-ID of any of the parts).  Contrary to the requirements
        specified in [RFC-2046], the Transfer-Encoding SHOULD be set
        to 8bit at least in each part that requires it.  The second
        and subsequent parts SHOULD contain References headers
        referring to all the previous parts, thus enabling reading
        agents with threading capabilities to present them in the
        correct order. Reading agents MAY then provide a facility to
        recombine the parts into a single article (but this document
        does not require them to do so).

        The Content-Type "message/external-body" could be apropriate
        for texts which it would be uneconomic (in view of the likely
        readership) to distribute to the entire network.

        The Content-Type: "multipart/mixed" (also
        "multipart/parallel") may be used freely in news articles.

        By default, the use of the Content-Type:
        "multipart/alternative" is deprecated (on account of the extra
        bandwidth consumed and the difficulty of quoting in
        followups).

        The Content-Type: "multipart/digest" is commended for any
        article composed of multiple messages more conveniently viewed
        as separate entities. The "boundary" should be composed of 28
        hyphens (ASCII 45) (which makes each boundary delimiter 30
        hyphens, or 32 for the final one) so as to accord with current
        practice for digests [RFC-1153].
[Actually, this conflicts with some present digest usage (such as the
news.answers rules), but should still be the right way to go. I
suggest this is left in for now (just to stake a claim), while we
discuss the matter with the news.answers moderators and the faq-
maintainers.]

        The Content-Type: "multipart/signed" [RFC-1847, RFC-2015] is
        the preferred method for the bodies of news articles that are
        to be digitally signed.  However, encryption (as in
        "multipart/encrypted") is deprecated.

6.x.4 Character Sets

        In principle, any character set may be specified in the
        "charset=" parameter of a content type. However, character
        sets other than "us-ascii", "iso-8859-1" (and the
        corresponding parts of UTF-8) ought only to be used in
        hierarchies where the language customarily used so requires
        (and whose readers could be expected to possess agents capable
        of displaying them).

6.x.5 Definition of some new Content-Types

        This document defines (or redefines) several new Content-
        Types, which require to be registered with IANA as provided
        for in [RFC-2048]. For "multipart/news-groupinfo" see 6.6.1.1,
        for "application/news-groupinfo" see 6.6.1.2, for
        "application/news-checkgroups" see 6.6.5, and for the
        remainder see this present section.

6.x.5.1 Application/news-transmission

        The Content-Type "application/news-transmission" is intended
        for the encapsulation of complete news articles where the
        intention is that the recipient should then inject them into
        Netnews. This Application type SHOULD be used when mailing
        articles to moderators and to mail-to-news gateways.
[The remarks about sending articles to moderators should perhaps be
made in a more appropriate place in our standard.]

        NOTE: The benefit of such encapsulation is that it removes
        possible conflict between news and email headers and it
        provides a convenient way of "tunnelling" a news article
        through a transport medium that does not support 8bit
        characters.

        The MIME content type definition of "application/news-
        transmission" is:

        MIME type name:           application
        MIME subtype name:        news-transmission
        Required parameters:      none
        Optional parameters:      moderate; inject; relay
        Encoding considerations:  A transfer-encoding (such as
                                  Quoted-Printable or Base64) different
                                  from that of the article transmitted
                                  MAY be supplied (perhaps en route) to
                                  ensure correct transmission over some
                                  7bit transport medium.
        Security considerations:  A news article may be a "control
                                  message", which could have effects on
                                  the recipient host's system beyond
                                  just storage of the article. However,
                                  such control messages also occur in
                                  normal news flow, so most hosts will
                                  already be suitably defended against
                                  undesired effects.
        Published specification:  [USEFOR]
        Body part:                A complete article or proto-article,
                                  ready for injection into Netnews, or a
                                  batch of such articles.

        NOTE: It is likely that the recipient of an
        "application/news-transmission" will be a specialised gateway
        (e.g. a moderator's submission address) able to accept
        articles with only one of the three parameter types
        "moderate", "inject" and "relay", hence the reason why they
        are optional, being redundant in most situations.
        Nevertheless, they MAY be used to signify the originator's
        intention with regard to the transmission, so removing any
        possible doubt.

        When the parameter "relay" is used, or implied, the body part
        MAY be a batch of articles to be transmitted together, in
        which case the following syntax MUST be used.

        batch        = 1*( batch-header article )
        batch-header = "#!" SPACE "rnews" SPACE article-size CRLF
        article-size = 1*digit

        where the "rnews" is case-sensitive. Thus a batch is a
        sequence of articles, each prefixed by a header line that
        includes its size. The article-size is a decimal count of the
        octets in the article, counting each CRLF as one octet
        regardless of how it is actually represented.

        NOTE: Despite the similarity of this format to an executable
        UNIX script, it is EXTREMELY unwise to feed such a batch into
        a command interpreter in anticipation of it running a command
        named "rnews"; the security implications of so doing would be
        disastrous.

6.x.5.2 Application/news-message
[WARNING: The application/mews-message type as described here has been
subject to much adverse criticism. Thus it is liable to be replaced by
an alternative method of encapsulation in a future draft of this
document.]

        The Content-Type "application/news-message" is intended for
        the encapsulation of complete news articles which have already
        been posted to Netnews and which are for the information of
        the recipient, and do not constitute a request to repost them.
        This Message type SHOULD be used when a courtesy copy of a
        followup is mailed to the author of its precursor, and when
        gatewaying from news-to-mail.

        NOTE: The benefits of such encapsulation are
        a) that it will be apparent to the recipient that what he sees
        is a copy of a news article rather than an original email;
        b) that it will survive tramsmission intact even when the
        transport medium does not support 8bit characters and has to
        employ some special Content-Transfer-Encoding;
        c) likewise, that UTF-8 characters in headers will survive
        transmission;
        c) likewise, that any digital signature applied to it will
        survive tramsmission intact.

        The MIME content type definition of "application/news-message"
        is:

        MIME type name:           application
        MIME subtype name:        news-message
        Required parameters:      none
        Optional parameters:      none
        Encoding considerations:  A transfer-encoding (such as
                                  Quoted-Printable or Base64) different
                                  from that of the article transmitted
                                  MAY be supplied (perhaps en route) to
                                  ensure correct transmission over some
                                  7bit transport medium.
        Disposition:              By default, presentation agents SHOULD
                                  display news-messages inline, except
                                  where overriden by a
                                  Content-Disposition header.
        Published specification:  [USEFOR]
        Body part:                A complete news article, as already
                                  injected into Netnews.

6.x.5.3 Message/news obsoleted
[WARNING: In view pf the possible withdrawal of the proposed
application/news-message (see above), it should not be assumed that
the obsoletion of message/news will necessarily remain in future
drafts of this document.]

        The Content-Type "message/news", as previously registered with
        IANA, is hereby obsoleted and should be withdrawn. The reason
        is that, according to RFC-2045, it is expressly forbidden for
        any "message" type to use a transfer-encoding other than
        "7bit", "8bit" or "binary", thus preventing a news article
        which makes full use of UTF-8 characters in its headers, or
        which uses (as it SHOULD) "8bit" as its own transfer-encoding,
        from being passed through a 7bit transport medium (recall that
        it is not permitted by this document for the transfer-encoding
        of a news article to be changed).

6.x.6 MIME within headers

        Since the headers of news articles are expected to use the
        UTF-8 character set, they SHOULD NOT normally be encoded using
        the MIME mechanism defined in [RFC-2047]. Nevertheless,
        reading agents SHOULD support that usage.

        It is to be noted, however, that RFC-2047 encoding would be
        required were a news article to be transmitted as a mail
        message without first encapsulating it as an
        "application/news-message" as suggested above.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Tue, 12 Jan 1999 11:11:09 -0800 (PST)
From: Chris Newman <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
Originator-info: login-id=chris; server=THOR.INNOSOFT.COM
To: Charles Lindsey <[email-masked]>
Content-transfer-encoding: 7BIT

[For some reason, the message you sent appeared to be private rather than
 from the list.  You may redirect this reply to the list if you intended
 this to be public debate.]

On Tue, 12 Jan 1999, Charles Lindsey wrote:
> In <[email-masked]> Chris Newman 
<[email-masked]> writes:
> 
> >I suspect application/news-message is a complete non-starter.  The last
> >thing MIME needs is a new composite type.  I suppose multipart/news is
> >possible, but ugly.  I doubt there will be sufficient motivation for MUA
> >authors to add good support for such an optional and ugly type.
> 
> AIUI there are three possibilities for encapsulating news articles:
> 
> 1. Application/news-message. This happens to be what is written into our
> current proto-draft, but it is not cast in concrete. It works, from a
> strict technical point of view, but it is agreed to be ugly. Exceedingly
> ugly if you like.

While you can put any _data_ in an application type that you wish, you
can't change the rules which govern how a MIME agent will process
application types in general.  Application types are processed as a single
opaque entity which is either saved to disk or passed directly to a viewer
application.  In particular, a MIME agent does not show an application
type to the user.  Since news messages sent through email should be
readable by users, using application is NOT an option.

Also, what if a news message has a MIME attachment?  If you use an
application type, then the attachment is inherently invisible to the MUA.

> 2. Multipart/news. This is the one I described briefly a few days ago.
> Details are not fully worked out. It would certainly work, and might even
> give an acceptable interface on present news/mail readers. But it is not
> pretty and certainly not ideal. Regard it as the fallback position for
> now.

Very ugly and will probably be less functional than the current email/news
infrastructure.  But it is an option, unlike #1.

> >* Create rules for UTF-8 headers and downgrading thereof.
> 
> We have worked this out for news. Would you like me to post an outline of
> what we have done here?

Sure.

> >* Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
> >  both top level headers and nested message/rfc822 headers.
> 
> May I suggest calling it "8BITHEADER" extension.

IMHO, it should either be UTF8HEADER with the simple obvious meaning, or
it would be 8BITHEADER and add an extra "HEADERCHARSET=<blah>" parameter
to MAIL FROM.  Given the latter is more complex and doesn't seem to be
significantly more functional, I prefer the former.

> applications for it. Presumably easily (trivially?) implemented on top of
> 8BITMIME.

No.  In order to interoperate with existing 7-bit infrastructure and
clients (or worse, clients which assume some random localized 8-bit
charset), an MTA which advertized UTF8HEADER MUST support RFC 2047
downconverting.  Otherwise you break things left and right.

> However, RFC 2047 downgrading may not be the answer. RFC 2047 is
> much disliked in USEFOR. In our draft it is a MUST accept, but MUST NOT
> generate. I think the point about any downgrading is that it must be
> reversible, which means you need to know how it started out so you can
> translate it back to the same thing again. One of our problems with RFC
> 2047 is that it is a one-to-many mapping (and, we suspect, frequently
> incorrectly implemented).

I don't think anyone likes RFC 2047.  But the only interoperable way to do
internationalized headers with today's email software is RFC 2047.  Any
new/better/simpler solution to internationalized headers will have to
support RFC 2047 downconversion for interoperability with deployed
software.  There is no alternative.

		- Chris

------------- End Forwarded Message -------------

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Xref: clerew local.mime:10
Newsgroups: local.mime
Path: clerew!daemon
From: Charles Lindsey <[email-masked]>
Subject: Re: Should mail and news converge or diverge?
>received: from mail.proper.com by m1.cs.man.ac.uk (8.8.8/AL/MJK-2.0) id RAA06632; Tue, 12 Jan 1999 17:44:46 GMT
Date: Tue, 12 Jan 1999 16:13:16 GMT
Lines: 82
Apparently-To: [email-masked]
Sender: [email-masked]

--66d3_3b37-5bc0_7d2c-7c5a_564b
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-MD5: nY/jP53t/vO6PA8YhUaO3g==
X-Sun-Data-Type: text

Sorry, my attempt to Cc this between two lists failed, and my mailer
does not give me much choice as to what Content-type it goes out as :-( .

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5
--66d3_3b37-5bc0_7d2c-7c5a_564b
Content-Type: application/octet-stream; name=0.9161514270.t
Content-Transfer-Encoding: 7bit
Content-MD5: L7vhbAx3BHYRzr0A/ncYQg==
Content-Description: 0.9161514270.t
Content-Disposition: attachment; filename=0.9161514270.t

Newsgroups: local.usenet
Path: local.mime!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Should mail and news converge or diverge?
Message-ID: <[email-masked]>
Cc: [email-masked]
X-Newsreader: NN version 6.5.2 (NOV)
References: <Pine.SUN.3.96.990111102700.12755B-100000@kira>
Date: Tue, 12 Jan 1999 14:30:19 GMT
Lines: 48

In <Pine.SUN.3.96.990111102700.12755B-100000@kira> John Stanley <[email-masked]> writes:

>Just the opposite is true. In a unified standard, there will be clear ways
>of telling which route a message takes to get to you, because both mail
>and news will be aware of the other and will know what to do, or not to
>do, with each other's transport-specific headers. E.g., mail will know
>that "Newsgroups" in a message it processes is meaningless and can remove
>it. News will know that "To:" in news is meaningless and remove it.

I think it would be bad for servers of transports ever to remove any
headers. In USEFOR we have steadfastly taken a strict anti-munging
position on the grounds that "well meaning" attempts to "improve" headers
have in the past done more harm than good.

But if a news article has a To: (or a Cc:) header, whould the user's agent
treat it as an intent to mail it as well as post it? Some newsreaders
offer this as a service. We could write it into our standard as a MAY.

As to Newsgroups headers in email, my preference it to encapsulate a news
article that is also mailed using application/news-message (or
whatever it turns out to be after the current discussions with Ned Freed
have stabilised).

But it would be too much to expect the whole world to start doing this, so
the other option of leaving the Newsgroups: header in the email ought also
to behave. Indeed, when gatewaying into a mailing list that is probably a
sensible idea (the recipients of the mailing list can then see where the
article was also posted).

There was a suggestion earlier (Jacob Palme, I think) for a Posted: header
to be allowed in mail to indicate that the message was also a news
article, but that seems to have died.

But bear in mind that if/when we get around to digitally signed headers,
these are likely to include the Newsgroups header, so removing it would
break the signature.

I hace Cced this to [email-masked] (so we shall see whether my
newsreader believes in Cc). I have not yet persuaded [email-masked] to
subscribe me to that list. You will also observe a Newsgroups: header in
this message.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5
--66d3_3b37-5bc0_7d2c-7c5a_564b--

Date: Tue, 12 Jan 1999 22:37:30 GMT
From: Charles Lindsey <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: [email-masked]
Content-MD5: G9BLB18bNOgFtrzrrJyYSA==
Sender: [email-masked]

------------- Begin Forwarded Message -------------

Date: Tue, 12 Jan 1999 10:21:01 -0800
From: Brad Templeton <[email-masked]>
To: "J. Scheerder" <[email-masked]>, IETF mailing list on e-mail headers <[email-masked]>, IETF Usenet format mailing list <[email-masked]>
Subject: Re: Should mail and news converge or diverge?
Organization: http://www.templetons.com/brad
Sender: [email-masked]

I can't say that the audience differnce is so vast.  The line between
the perception of a mailing list and a newsgroup is pretty blurry.
The mailing list is often diverted into other folders.  It's harder
to subscribe to, usually smaller, and sometimes private, but those
are about the differences people see.

Xref: clerew local.mime:8
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Restrictions on the Content-Type "message"
Date: Tue, 12 Jan 1999 16:00:03 GMT
Lines: 73
Apparently-To: [email-masked]
Sender: [email-masked]

In <[email-masked]> Chris Newman <[email-masked]> writes:

>I suspect application/news-message is a complete non-starter.  The last
>thing MIME needs is a new composite type.  I suppose multipart/news is
>possible, but ugly.  I doubt there will be sufficient motivation for MUA
>authors to add good support for such an optional and ugly type.

AIUI there are three possibilities for encapsulating news articles:

1. Application/news-message. This happens to be what is written into our
current proto-draft, but it is not cast in concrete. It works, from a
strict technical point of view, but it is agreed to be ugly. Exceedingly
ugly if you like.

2. Multipart/news. This is the one I described briefly a few days ago.
Details are not fully worked out. It would certainly work, and might even
give an acceptable interface on present news/mail readers. But it is not
pretty and certainly not ideal. Regard it as the fallback position for
now.

3. Some new enhanced feature for email. Obviously this is the ideal
solution, except that it does not exist yet. Ned reckons it could be done
in 6 months. I predicted 5 years. Doubtless the truth lies somewhere in
between.

>The solution I prefer is:

OK, let's look at the possibilities.

>* Create rules for UTF-8 headers and downgrading thereof.

We have worked this out for news. Would you like me to post an outline of
what we have done here?

>* Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
>  both top level headers and nested message/rfc822 headers.

May I suggest calling it "8BITHEADER" extension. There might be other
applications for it. Presumably easily (trivially?) implemented on top of
8BITMIME. However, RFC 2047 downgrading may not be the answer. RFC 2047 is
much disliked in USEFOR. In our draft it is a MUST accept, but MUST NOT
generate. I think the point about any downgrading is that it must be
reversible, which means you need to know how it started out so you can
translate it back to the same thing again. One of our problems with RFC
2047 is that it is a one-to-many mapping (and, we suspect, frequently
incorrectly implemented).

>* Extend message/rfc822 to permit 8-bit headers in appropriate transports
>  and to use:  Content-Type: message/rfc822; variant=news

Yes.

>* Permit use of 8-bit "message/rfc822; variant=news" with any SMTP server
>  that advertises UTF8HEADER.

Yes.

>* a news->mail gateway can use message/rfc822; variant=news, and has to
>  have full downgrading support.
>* a mail->news gateway has to provide full upgrading to 8-bit support.
>* Support for a top-level message/rfc822 is already part of MIME, although
>  UI quality varies.  This should be sufficient incentive for MUA vendors
>  to improve support for an already mandatory facility.
>* old gateways which don't do proper downgrading/upgrading will get yelled
>  at and fixed.

All yes, but the upgrading is perhaps the tricky part.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Xref: clerew local.mime:9
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Restrictions on the Content-Type "message"
Date: Tue, 12 Jan 1999 16:07:30 GMT
Lines: 21
Apparently-To: [email-masked]
Sender: [email-masked]

In <[email-masked]> Ned Freed <[email-masked]> writes:

>I note in passing that the present USEFOR drafts apparently don't contain
>definitions of either of these types, other than to note that news-transmission
>isn't well defined.

One of our problems on the USEFOR list is that the man supposed to be
editing the drafts has been ill, or absent, or silent, or whatever. Not a
satisfactory situation, but that is our problem, not yours.

But it does mean that the draft you are likely to have seen is way behind
our current state. Would it be useful for me to post our current
MIME-related part to this list? Note that it still contains the
application/news-message proposal, but my next job is to insert a huge
meta-comment at that point to say that it is subject to further review.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Xref: clerew local.mime:7
Newsgroups: local.mime
Path: clerew!chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Restrictions on the Content-Type "message"
Date: Tue, 12 Jan 1999 15:36:17 GMT
Lines: 96
Apparently-To: [email-masked]
Sender: [email-masked]

I am now on the [email-masked] list, so no need to send separate Cc:s to
me now.

In <[email-masked]> [email-masked] (Kai Henningsen) writes:

>[email-masked] (Ned Freed)  wrote on 07.01.99 in <[email-masked]>:

I see that Ned has now agreed that he was shooting at the wrong target
here, so I will just comment on a few details and clarifications.

>> > The way I understand it, application/news-transmission is a tunneling
>> > protocol. As such, it's *requirements* are:
>>
>> > 1. Upon unpacking from the tunnel, the articles in question must be
>> > bit-by- bit *identical* to the articles that went in. That means redoing
>> > nested encodings is Right Out[tm].

Yes, absolutely so. It is meant now to be processed by news software only.
If that software finds some inner CTE that it wants to expand, then so be
it. Note that the stage at which a CTE is undone can be important if there
is a digital signature involved. Checking it on the wrong side of the CTE
will make it fail.

>>
>> ... So we wrote MIME with a no
>> nested encodings rule, and we didn't make an exemption for tunneling or
>> anything else.

No. There is no nested encoding in multipart or message types, but you can
do what you like in applications types (it is the application that gets to
sort out the mess).

>> But there is a formal rule that things have to interoperate. You cannot
>> proceed down the standards track in the presence of interoperability
>> problems and you cannot even get on the track if it can be shown that your
>> proposal is guaranteed to cause interoperability problems.

Actually, since RFC 1036 was never a standard, the formal rule may not
strictly apply. Not that we wish to avoid that problem, of course.
Everything that has been proposed in USEFOR has had its impact on existing
implementations carefully considered. We have drawn a clear distinction
between news readers (or which there are millions worldwide, with umpteen
different implementations) and news servers (which are fewer in number and
use a rather small number of different implementations). Clearly, current
newsreaders will be around a long time yet, and so must be failsafe as
regards new funtionality. OTOH, it is much easier to lean on servers that
are behind the times, and to flood around them if they remain so (news has
never been a 100% lossless medium :-( ).

>Gateways are a problem. If you transfer through a news->mail and a  
>mail->news gateway, what you get is corrupted news and people shouting at  
>you to shut down that gateway.

Yes, gateways are a problem and gateways are difficult. And maybe even
more difficult than they were before :-( . And yes, they are always going
to get shouted at.

>> > Now, the case of application/news-message is not quite so simple. The way
>> > I understand it, this one is actually meant (among others) for situations
>> > where software would need to act on these nested encodings.
>>
>> > *However*, it still runs up against the problem that
>>
>> > a. News is meant to be 8 bit clean (not binary clean, though)
>>
>> > b. News is built around the assumption that, once a news article is
>> > underway, most of the article headers and all of the body are safe from
>> > change, in stark contrast to the mail case. (Typically, the exeptions are
>> > the Path: header and the Xref: header.) There are applications around
>> > *right* now that rely on this.

Yes. Son-of-1036 made this argument strongly, and the USEFOR drafts make
it even more strongly. If you cannot handle an article for some reason,
then it is better to drop it on the floor than to try and transmit it in
munged form.

>I think I can predict something here. If nobody comes up with a better  
>solution (and I don't think your ideas about gateways would even count as  
>half as good), and the IETF/IESG decide to make a problem out of this,  
>then the news community will probably decide that, since they've gone so  
>long without an IETF/IESG-blessed standard, they can do the same in the  
>future; they'll just make their own standards, just the same as they've  
>done all the time.

Yes, that could indeed happen. But the intention and hope is to do it
right if it can be done.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Tue, 12 Jan 1999 16:13:16 GMT
From: Charles Lindsey <[email-masked]>
To: [email-masked]
Subject: Re: Should mail and news converge or diverge?
Sender: [email-masked]

--66d3_3b37-5bc0_7d2c-7c5a_564b
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-MD5: nY/jP53t/vO6PA8YhUaO3g==
X-Sun-Data-Type: text

Sorry, my attempt to Cc this between two lists failed, and my mailer
does not give me much choice as to what Content-type it goes out as :-( .

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5
--66d3_3b37-5bc0_7d2c-7c5a_564b
Content-Type: application/octet-stream; name=0.9161514270.t
Content-Transfer-Encoding: 7bit
Content-MD5: L7vhbAx3BHYRzr0A/ncYQg==
Content-Description: 0.9161514270.t
Content-Disposition: attachment; filename=0.9161514270.t

Newsgroups: local.usenet
Path: chl
From: [email-masked] (Charles Lindsey)
Subject: Re: Should mail and news converge or diverge?
Message-ID: <[email-masked]>
Cc: [email-masked]
X-Newsreader: NN version 6.5.2 (NOV)
References: <Pine.SUN.3.96.990111102700.12755B-100000@kira>
Date: Tue, 12 Jan 1999 14:30:19 GMT
Lines: 48

In <Pine.SUN.3.96.990111102700.12755B-100000@kira> John Stanley <[email-masked]> writes:

>Just the opposite is true. In a unified standard, there will be clear ways
>of telling which route a message takes to get to you, because both mail
>and news will be aware of the other and will know what to do, or not to
>do, with each other's transport-specific headers. E.g., mail will know
>that "Newsgroups" in a message it processes is meaningless and can remove
>it. News will know that "To:" in news is meaningless and remove it.

I think it would be bad for servers of transports ever to remove any
headers. In USEFOR we have steadfastly taken a strict anti-munging
position on the grounds that "well meaning" attempts to "improve" headers
have in the past done more harm than good.

But if a news article has a To: (or a Cc:) header, whould the user's agent
treat it as an intent to mail it as well as post it? Some newsreaders
offer this as a service. We could write it into our standard as a MAY.

As to Newsgroups headers in email, my preference it to encapsulate a news
article that is also mailed using application/news-message (or
whatever it turns out to be after the current discussions with Ned Freed
have stabilised).

But it would be too much to expect the whole world to start doing this, so
the other option of leaving the Newsgroups: header in the email ought also
to behave. Indeed, when gatewaying into a mailing list that is probably a
sensible idea (the recipients of the mailing list can then see where the
article was also posted).

There was a suggestion earlier (Jacob Palme, I think) for a Posted: header
to be allowed in mail to indicate that the message was also a news
article, but that seems to have died.

But bear in mind that if/when we get around to digitally signed headers,
these are likely to include the Newsgroups header, so removing it would
break the signature.

I hace Cced this to [email-masked] (so we shall see whether my
newsreader believes in Cc). I have not yet persuaded [email-masked] to
subscribe me to that list. You will also observe a Newsgroups: header in
this message.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5
--66d3_3b37-5bc0_7d2c-7c5a_564b--

Date: Tue, 12 Jan 1999 09:06:05 +0100
To: "J. Scheerder" <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Re: Should mail and news converge or diverge?
Cc: IETF mailing list on e-mail headers <[email-masked]>, IETF Usenet format mailing list <[email-masked]>
Sender: [email-masked]

At 10.55 +0100 99-01-11, J. Scheerder wrote:
> That is, however, all technology speaking.  But, and to me that's
> a thing never to be forgotten: even though the technologies may
> be similar, and the interfacing similar, the `media' are very different.
> A mail message and a news message typically have a very different
> audience.  That implies that they are written from a different outlook.
>
> What I'm advocating here: regardless of whether Usenet and mail message
> format get unified or not, at the user level it is essential to make
> a 100% crisp clear distinction.  The user must at all times *know* where
> a particular message came from (is it a personal message to me or not?),
> and where a reply (or followup) is going to (is it a personal message
> for Joe Shmoe, or am I writing for a generic audience?).

It is a well-known problem in both mail and news that
people who read and write messages/articles are often
unsure of their audience. I agree very much that user
interfaces should do more to help people in this respect.
This is however a difficult problem, there is no simple
solution.

Since IETF does usually not standardize user interfaces,
what we can do is to ensure that all the information is
there to the client software. The client software will know
whether it downloaded a message using NNTP or using POP or
IMAP. IMAP can be used to download both from private
mailboxes and public mail areas. I am no expert on IMAP,
but I hope that the client knows, when downloading a
message through IMAP, whether this is from a private
mailbox or a public area?

A particular area are messages sent to both mail and news
or gatewayed between these areas. I understand that the
USEFOR IETF group is going to propose clear identification
on such messages that they have been posted to both
carrying media lr that they have been gatewayed from mail
to news or the reverse. So this will give additional
information, which the client can use to tell the user of
the audience of a message.

The size and type of the audience need not be directly
depending on whether a message is sent to a newsgroup or a
mailing list. There are small specialized newsgroups, even
closed newsgroups within a single news server, and there
are very large mailing lists with public subscription. So
just telling a user that "this is a mail message" or "this
is a news message" need not give reliable information to
the user of whether the audience is small and closed or
large and open.

One hint on the audience is if the e-mail address of the
recipient can be found in the "To", "Cc" or "Bcc" fields.
If this is not the case, the audience is probably large.

The RFC 2369 will mean that clients more often will know
which messages come from mailing lists, and which these
lists are, which is also information the client can use
to give the user information about the audience.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

Date: Mon, 11 Jan 1999 10:55:49 +0100
From: "J. Scheerder" <[email-masked]>
To: Jacob Palme <[email-masked]>
Cc: IETF mailing list on e-mail headers <[email-masked]>, IETF Usenet format mailing list <[email-masked]>
Subject: Re: Should mail and news converge or diverge?
Sender: [email-masked]

Jacob Palme:

[...]

> Should we try to make e-mail and news
> standards as close as possible?

[comments -- snip]

[argument lists pro and con -- snip]

> Common counterargument: The separation between mail and
> news is user-friendly, because it allows users to give mail
> higher priority.
> 
> Counter-counterargument: Certainly users need filtering,
> sorting and prioritizing of the incoming message flow. But
> it is not always true that for a particular user all mail
> messages are always more important than all news articles.

$.02: there is a _very_  important level of distinction that I feel
is overlooked here.

Yes, the technology is `close'.  This is, from the viewpoint of a
programmer, a strong hint to handle the similar tasks with the
very same code.  From the user's point of view, similar tasks
(i.e. composing messages for platforms with similar prerequisites)
need to be dealt with similarly.

Yes, one could expect some consistency in interface.  And some reuse of
code -- a message editor is a message editor, and offering news message
editing and mail message editing in a consistent interface, i.e. identical
for identical tasks, may be a good thing.

That is, however, all technology speaking.  But, and to me that's
a thing never to be forgotten: even though the technologies may
be similar, and the interfacing similar, the `media' are very different.
A mail message and a news message typically have a very different
audience.  That implies that they are written from a different outlook.

What I'm advocating here: regardless of whether Usenet and mail message
format get unified or not, at the user level it is essential to make
a 100% crisp clear distinction.  The user must at all times *know* where
a particular message came from (is it a personal message to me or not?),
and where a reply (or followup) is going to (is it a personal message
for Joe Shmoe, or am I writing for a generic audience?).

There is a surprising amount of software that obliterates the
distinction, making the mistake of not seeing the functional difference
in technologically similar tasks.

By itself, that observation is not an argument.  However, one can see
that full format unification increases the risk of obliteration of the
difference between the personal email and public Usenet, at the user
level, whereas divergence may give an incentive to `share' (inherit?)
the common technology and interfacing, while doing the differences
full justice (i.e. create differring instances of an interface class,
of an editor class, and so on, depending on context).

> Conclusion
> ----------
> 
> Since there are arguments both for and against convergence
> between mail and news, we cannot conclude that we should
> strive for full integration, or that we should disregard
> the integration needs entirely. But we should strive for as
> much convergence as possible, and we should have good
> reason in cases where we intentionally define things
> differently for mail and news.

I think that's a sound principle.  The danger of `blurring the issue'
should be kept in mind, I'd argue, when considering whether or not
integration is a good and worthwhile thing.

-- J.
-- 
The ideal is impossible... the idea of the ideal is essential.
                -- P. Taylor

Date: Sun, 10 Jan 1999 10:25:56 +0100
To: IETF mailing list on e-mail headers <[email-masked]>, IETF Usenet format mailing list <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Should mail and news converge or diverge?
Sender: [email-masked]

--============_-1296163083==_ma============
Content-Type: text/plain; charset="us-ascii"

This message is intentionally sent to both the
usenet-format and the ietf-822 mailing lists.
The message is sent in the format multipart/alternative
with plain ascii and text/html in the branches. If you
have problems reading this format, please report to me.

Should we try to make e-mail and news
standards as close as possible?

Some previous comments on this issue:
------------------------------------

At 02.26 +0100 99-01-07, Keith Moore wrote:
> While I'd like to keep the two formats as similar as
> possible (and thus maximize the amount of sharable code)
> it's far too late to insist that Usenet and email use the
> same message format. Fundamental protocol elements like
> From and Reply-To have always had significant differences
> between email and usenet. Heck, usenet doesn't even
> separate the delivery envelope from content.

At 19.38 +0100 99-01-04, Ned Freed wrote:
> BTW, one argument I suspect to hear is that this is news,
> not mail, and what applies to news doesn't apply to mail.
> But all you have to do is look at the number of news to
> email gateways to see how this doesn't wash.

At 20.12 +0100 99-01-07, Ned Freed wrote:
> But there is a formal rule that things have to
> interoperate. You cannot proceed down the standards track
> in the presence of interoperability problems and you
> cannot even get on the track if it can be shown that your
> proposal is guaranteed to cause interoperability problems.
> Now, we all know that mail and news are gatewayed all the
> time. Messages pass from news to mail and from mail to
> news, sometimes multiple times.

Arguments for divergence:
------------------------

- Different needs for different applications.

- Established base.

- Restrictions needed in one protocol not need
  in another (Examples: news does not need
  content-transfer-encoding, mail does maybe not
  need mandatory Message-ID and Subject).

- Easier to get results in standards work.

Arguments for convergence:
-------------------------

- Messages are often moved through mail-news
  gateways, and these will work better, the less
  difference there is between the formats.

- The same message is sometimes sent to both mail
  and news at the same time.

- Replies to news messages are sometimes sent via mail.

- Combined news and mail agents are becoming more common.

- User filtering needs, see below.

User filtering needs
--------------------

E-mail and netnews differ from most other Internet
applications in that there is a continuos stream of new
information, and that users want to be told when there is
new items for them. At the same time, this causes a risk of
information overload.

Users find it difficult to have to regularly connect to
several different systems in order to check for news. It is
easier for them to use a unified system, where all new
items are available with the same user interface.

Common counterargument: The separation between mail and
news is user-friendly, because it allows users to give mail
higher priority.

Counter-counterargument: Certainly users need filtering,
sorting and prioritizing of the incoming message flow. But
it is not always true that for a particular user all mail
messages are always more important than all news articles.
Some mailing lists may for example have lower priority than
some newsgroups for a particular user. So filtering,
sorting and prioritizing should not be restricted by an
arbitrary rule putting all mail at higher priority than all
news.

Conclusion
----------

Since there are arguments both for and against convergence
between mail and news, we cannot conclude that we should
strive for full integration, or that we should disregard
the integration needs entirely. But we should strive for as
much convergence as possible, and we should have good
reason in cases where we intentionally define things
differently for mail and news.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme
--============_-1296163083==_ma============
Content-Type: text/html; charset="us-ascii"

<div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4" color="#0000CC">This 
  message is intentionally sent to both the<br>
  usenet-format and the ietf-822 mailing lists.</font></div>
<div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4" color="#0000CC">The 
  message is sent in the format multipart/alternative</font></div>
<div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4" color="#0000CC">with 
  plain ascii and text/html in the branches. If you</font></div>
<div> 
  <p><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4" color="#0000CC">have 
    problems reading this format, please report to me.</font></p>
  <h1><font face="Times" size="+1"> </font><font face="Times"><b><font face="Verdana, Arial, Helvetica, Geneva, sans-serif">Should 
    we try to make e-mail and<br>
    news </font></b></font><font face="Verdana, Arial, Helvetica, Geneva, sans-serif"><b>standards 
    as close as possible?</b></font><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif"><b></b></font></h1>
</div>
<div>
<h2><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif"><b><font face="Verdana, Arial, Helvetica, Geneva, sans-serif">Some 
    previous comments on this issue:</font></b></font></h2>
  <p><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"><b><font color="#0000CC">At 
    02.26 +0100 99-01-07, Keith Moore wrote:</font></b><br>
    &gt; While I'd like to keep the two formats as similar as<br>
    &gt; possible (and thus maximize the amount of sharable code)<br>
    &gt; it's far too late to insist that Usenet and email use the<br>
    &gt; same message format. Fundamental protocol elements like<br>
    &gt; From and Reply-To have always had significant differences<br>
    &gt; between email and usenet. Heck, usenet doesn't even<br>
    &gt; separate the delivery envelope from content.</font></p>
</div>
<div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"><br>
  <font color="#0000CC"><b>At 19.38 +0100 99-01-04, Ned Freed wrote:</b></font><br>
  &gt; BTW, one argument I suspect to hear is that this is news,<br>
  &gt; not mail, and what applies to news doesn't apply to mail.<br>
  &gt; But all you have to do is look at the number of news to<br>
  &gt; email gateways to see how this doesn't wash.</font></div>
<div> 
  <p><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"><br>
    <b><font color="#0000CC">At 20.12 +0100 99-01-07, Ned Freed wrote:</font></b><br>
    &gt; But there is a formal rule that things have to inter-<br>
    &gt; operate. You cannot proceed down the standards track<br>
    &gt; in the presence of interoperability problems and you<br>
    &gt; cannot even get on the track if it can be shown that your<br>
    &gt; proposal is guaranteed to cause interoperability problems.<br>
    &gt; Now, we all know that mail and news are gatewayed all the<br>
    &gt; time. Messages pass from news to mail and from mail to<br>
    &gt; news, sometimes multiple times</font></p>
  <h2><b><font face="Verdana, Arial, Helvetica, Geneva, sans-serif">Arguments 
    for divergence:</font></b></h2>
</div>
<ul>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"> 
      Different needs for different applications.<br>
      </font></div>
  </li>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">Established 
      base.<br>
      </font></div>
  </li>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"> 
      Restrictions needed in one protocol not need</font><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"><br>
      in another (Examples: news does not need</font><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"><br>
      content-transfer-encoding, mail does maybe<br>
      not need mandatory Message-ID and Subject).<br>
      </font></div>
  </li>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">Easier 
      to get results in standards work.</font></div>
  </li>
</ul>
<div> 
  <h2><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif"><b><font face="Verdana, Arial, Helvetica, Geneva, sans-serif">Arguments 
    for convergence:</font></b></font></h2>
</div>
<ul>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"> 
      Messages are often moved through mail-news </font><br>
      g<font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">ateways, 
      and these will work better, the less </font><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4"><br>
      difference there is between the formats.<br>
      </font></div>
  </li>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">The 
      same message is sometimes sent to both <br>
      mail </font>a<font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">nd 
      news at the same time.<br>
      </font></div>
  </li>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">Replies 
      to news messages are sometimes sent <br>
      via mail.<br>
      </font></div>
  </li>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">Combined 
      news and mail agents are becoming <br>
      more common.<br>
      </font></div>
  </li>
  <li> 
    <div><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">User 
      filtering needs, see below.</font> </div>
  </li>
</ul>
<div>
<h2><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif"><b><font face="Verdana, Arial, Helvetica, Geneva, sans-serif">User 
    filtering needs</font></b></font></h2>
  <p><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">E-mail 
    and netnews differ from most other Internet<br>
    applications in that there is a continuos stream of new<br>
    information, and that users want to be told when there is<br>
    new items for them. At the same time, this causes a risk of<br>
    information overload.<br>
    <br>
    Users find it difficult to have to regularly connect to<br>
    several different systems in order to check for news. It is<br>
    easier for them to use a unified system, where all new<br>
    items are available with the same user interface.<br>
    <br>
    <font color="#660000"><b>Common counterargument:</b></font> The separation 
    between mail and<br>
    news is user-friendly, because it allows users to give mail<br>
    higher priority.<br>
    <br>
    <font color="#660000"><b>Counter-counterargument: </b></font>Certainly users 
    need filtering,<br>
    sorting and prioritizing of the incoming message flow. But<br>
    it is not always true that for a particular user all mail<br>
    messages are always more important than all news articles.<br>
    Some mailing lists may for example have lower priority than<br>
    some newsgroups for a particular user. So filtering,<br>
    sorting and prioritizing should not be restricted by an<br>
    arbitrary rule putting all mail at higher priority than all<br>
    news.</font></p>
  <h2><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif"><b><font face="Verdana, Arial, Helvetica, Geneva, sans-serif">Conclusion</font></b></font></h2>
  <p><font face="Georgia, New Century Schlbk, Times, Times New Roman, serif" size="4">Since 
    there are arguments both for and against convergence<br>
    between mail and news, we cannot conclude that we should<br>
    strive for full integration, or that we should disregard<br>
    the integration needs entirely. But we should strive for as<br>
    much convergence as possible, and we should have good<br>
    reason in cases where we intentionally define things<br>
    differently for mail and news.</font></p>
</div>
  
<div> 
  <hr>
  <font size="4" face="Georgia, New Century Schlbk, Times, Times New Roman, serif">Jacob 
  Palme &lt;[email-masked]&gt; (Stockholm University and KTH)<br>
  for more info see URL: http://www.dsv.su.se/~jpalme</font></div>
--============_-1296163083==_ma============--

Date: Fri, 08 Jan 1999 22:34:24 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Chris Newman <[email-masked]>
Cc: Kai Henningsen <[email-masked]>, [email-masked]
Sender: [email-masked]

> > Uh, pay attention, please. The above is application/news-transmission, not
> > application/news-message, for which, if you recall, I said that the case
> > is far murkier.

> I see no problems with application/news-transmission, assuming it is a
> news equivalent to application/batch-SMTP (RFC 2442).  It's something
> end-users will never see.

Apologies to all -- I completely missed the point that this was talking about a
transport protocol type. (Indeed, I cannot see how discussing such a thing has
any relevance whatsoever to the topic at hand.) I thought the discussion was on
the topic of message types.

I note in passing that the present USEFOR drafts apparently don't contain
definitions of either of these types, other than to note that news-transmission
isn't well defined.

However, unlike message types, there is absolutely no problem assigning types
to transport protocol objects. The key distinction here is that transport
protocol objects have very different semantics and uses than composite message
objects.

About the only tricky part can be definition of appropriate "batching" rules.
Experience has shown that when you get this wrong, as happened in the case of
the various initial attempts to define batch SMTP, you end up with all sorts of
amusing problems. And we basically had to wait for existing use of batch SMTP
to die out before we could try again.

> I suspect application/news-message is a complete non-starter.  The last
> thing MIME needs is a new composite type.  I suppose multipart/news is
> possible, but ugly.  I doubt there will be sufficient motivation for MUA
> authors to add good support for such an optional and ugly type.

Exactly.

> The solution I prefer is:

> * Create rules for UTF-8 headers and downgrading thereof.
> * Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
>   both top level headers and nested message/rfc822 headers.
> * Extend message/rfc822 to permit 8-bit headers in appropriate transports
>   and to use:  Content-Type: message/rfc822; variant=news
> * Permit use of 8-bit "message/rfc822; variant=news" with any SMTP server
>   that advertises UTF8HEADER.

I'm not wild about the parameter name "variant", but aside from that
this is exactly what I think should happen.

> * a news->mail gateway can use message/rfc822; variant=news, and has to
>   have full downgrading support.
> * a mail->news gateway has to provide full upgrading to 8-bit support.
> * Support for a top-level message/rfc822 is already part of MIME, although
>   UI quality varies.  This should be sufficient incentive for MUA vendors
>   to improve support for an already mandatory facility.

FWIW, gateway support quality also varies greatly, but has greatly improved in
the last few years. (As has MUA/UI support quality.)

> * old gateways which don't do proper downgrading/upgrading will get yelled
>   at and fixed.

I hope so, although past experience with gateways upgrade leads me less than
optimistic that this will happen in a timely fashion.

				Ned

Date: Fri, 08 Jan 1999 20:27:47 -0800 (PST)
From: Chris Newman <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
Originator-info: login-id=chris; server=THOR.INNOSOFT.COM
To: Kai Henningsen <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On Fri, 8 Jan 1999, Kai Henningsen wrote:
> Uh, pay attention, please. The above is application/news-transmission, not  
> application/news-message, for which, if you recall, I said that the case  
> is far murkier.

I see no problems with application/news-transmission, assuming it is a
news equivalent to application/batch-SMTP (RFC 2442).  It's something
end-users will never see.

I suspect application/news-message is a complete non-starter.  The last
thing MIME needs is a new composite type.  I suppose multipart/news is
possible, but ugly.  I doubt there will be sufficient motivation for MUA
authors to add good support for such an optional and ugly type.

The solution I prefer is:

* Create rules for UTF-8 headers and downgrading thereof.
* Create UTF8HEADER SMTP extension.  Provides RFC 2047 downgrading for
  both top level headers and nested message/rfc822 headers.
* Extend message/rfc822 to permit 8-bit headers in appropriate transports
  and to use:  Content-Type: message/rfc822; variant=news
* Permit use of 8-bit "message/rfc822; variant=news" with any SMTP server
  that advertises UTF8HEADER.

* a news->mail gateway can use message/rfc822; variant=news, and has to
  have full downgrading support.
* a mail->news gateway has to provide full upgrading to 8-bit support.
* Support for a top-level message/rfc822 is already part of MIME, although
  UI quality varies.  This should be sufficient incentive for MUA vendors
  to improve support for an already mandatory facility.
* old gateways which don't do proper downgrading/upgrading will get yelled
  at and fixed.

		- Chris

Date: 08 Jan 1999 18:24:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: Restrictions on the Content-Type "message"
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Ned Freed)  wrote on 07.01.99 in <[email-masked]>:

> > The way I understand it, application/news-transmission is a tunneling
> > protocol. As such, it's *requirements* are:
>
> > 1. Upon unpacking from the tunnel, the articles in question must be
> > bit-by- bit *identical* to the articles that went in. That means redoing
> > nested encodings is Right Out[tm].
>
> > 2. It being news, and news wanting no 7 bit restriction for very good
> > reasons (such as those being known to lead to flamewars all the time), the
> > mail restriction to 7 bit means that some sort of CTE is necessary here.
>
> > Now, the combination of 1 and 2 means there is no possible way to do this
> > without using nested encodings. OTOH, 1 also means that nested encodings
> > should not be a problem, since nobody is expected to *handle* nested
> > encodings - the mail and gateway parts only ever handle the outer
> > encodings, and other news software only ever sees the inner encodings.
>
> There's nothing new here -- these are exactly the same arguments we heard 7
> years ago when we first did MIME, and I understand them all too well. The
> point you don't seem to be getting is that we considered all these choices
> way back when MIME was designed, and we decided that we would not allow such
> requirements to be met. Let me repeat that: We would not allow it. We
> understood that situations would arise where such requirements would be
> given, but we decided not to cater to them. So we wrote MIME with a no
> nested encodings rule, and we didn't make an exemption for tunneling or
> anything else.

Well, it's true. I do have problems getting the point that people would  
act that way without a good reason.

Which good reason, frankly, nobody has given yet.

> You're attempting to go back on a decision that was made some time ago,
> rightly or wrongly, and which implementations have gone along with in good
> faith, and which has resulted in a situation where if you're allowed to
> proceed along the path you've chosen the result becomes untenable.

Uh, there's a problem with your assertions there.

You seem to think that there will be implementations negatively affected  
by implementing the obvious solution to the above problem.

I completely fail to see which implementations that would be. In fact,  
given the nature of the problem, it seems just about impossible for such  
implementations to exist.

Maybe you could explain this?

> But there is a formal rule that things have to interoperate. You cannot
> proceed down the standards track in the presence of interoperability
> problems and you cannot even get on the track if it can be shown that your
> proposal is guaranteed to cause interoperability problems.

The proposal - this part of it, at least - is *guaranteed* *not* to cause  
interoperability problems. IMHO.

> Now, we all know
> that mail and news are gatewayed all the time. Messages pass from news to
> mail and from mail to news, sometimes multiple times.
>
> So what happens when a application/news-message object passes from news to

Uh, pay attention, please. The above is application/news-transmission, not  
application/news-message, for which, if you recall, I said that the case  
is far murkier.

> mail, one which contains 8bit characters in the data, and the mail system
> has to downgrade to 7bit? There are two choices:
>
> (1) The mail system treats the object as opaque and slaps on a CTE. But then
>     when someone tries to read this object using a mail reader, they see an
>     opaque blob. The content, which likely contains perfectly reasonable
> MIME     objects, is unreadable.

Sounds perfectly ok to me. It's not meant to be viewed in a MUA. It's  
meant to be fed to a news server. There's no good reason why a mail reader  
*should* know how to interpret this. (Incidentally, this is the exact same  
case as with application/batch-smtp - you can't dig down to embedded MIME  
objects with that one, either.)

> The result is a failure to interoperate,
> one which     would effectively prevent any sort of move to draft standard.

Seems extremely silly to me. "Hey, this doesn't do something that there's  
no good reason to try to do." So what?

> Either way you have a violation of the intent, if not the letter, of the
> standards process. And this is why I'm saying that the course you're on
> isn't viable.

Except I'm claiming that it certainly doesn't violate the intent, whereas  
your line, to me, looks like intentionally trying to subvert the standards  
process.

Think about it: what you're saying is, in effect, "you can't do a standard  
that does what you need, you need to do it with non-standard means".  
Doesn't that sound silly to you?

> The long and short of it is that you need to give up on this idea of strict
> tunneling. And once you do, a number of approaches open up that I think will
> meet your real requirements without breaking things and which are viable as
> standards.

Nope. Without strict tunnelling, this cannot do what it is meant to do.

> The most obvious way to do this is to treat this as a gateway problem and

Gateways are a problem. If you transfer through a news->mail and a  
mail->news gateway, what you get is corrupted news and people shouting at  
you to shut down that gateway.

> define the operation of mail <--> news gateways to get what you want. For

That's a *far* harder nut than this MIME issue. Except if you just handle  
it as a different, non-MIME way of encapsulation, in which case I must see  
I see even less of a point.

> > Now, the case of application/news-message is not quite so simple. The way
> > I understand it, this one is actually meant (among others) for situations
> > where software would need to act on these nested encodings.
>
> > *However*, it still runs up against the problem that
>
> > a. News is meant to be 8 bit clean (not binary clean, though)
>
> > b. News is built around the assumption that, once a news article is
> > underway, most of the article headers and all of the body are safe from
> > change, in stark contrast to the mail case. (Typically, the exeptions are
> > the Path: header and the Xref: header.) There are applications around
> > *right* now that rely on this.
>
> > Obviously, a means that the headers of a news message, if sent by mail,
> > may still need to have a CTE applied. And b means that redoing CTEs is
> > still a problem.
>
> > So, it seems to me that we have here a case where the design goals of the
> > news communbity are actually incompatible with the design goals of the
> > MIME community, and there is no technical solution that will completely
> > satisfy both sides.
>
> I disagree. I don't think you've explored all the options for moving
> forward. I think that you've picked a way forward that solves one set of
> problems but not a bunch of others (you've admitted your solution doesn't
> handle the outermost header of a news message moving through mail), and

Huh?! That sounds like just about the opposite of what I've said. The  
application/news-message thing *does* solve that problem; the message/rfc- 
822 doesn't, the way I understand it.

> moreover it causes lots of new problems, many of them of a sort that are
> going to give you a great deal of trouble in the standards process. But
> you've gotten hung up on how this one solution works and made the conditions

I find it interesting that you claim I'm hung up on a solution, just  
because I take the trouble of explaining it to you. Also above you claim  
I'm "just not getting" a point.

Maybe you confuse me with someone else?

> of it absolute.

I think the conditions are just about absolute, and in any case, they're  
older than the solution.

Oh, and you seem just as (if not far more) hung up on some absolute, and  
far less justified, conditions.

> I think that with a bit of compromise on your part we'll

"Hey, you forget about your necessary functionality, and I'll keep my  
unreasonable demands. Now isn't that a nice compromise?"

Why doesn't that sound reasonable to me?

I think I can predict something here. If nobody comes up with a better  
solution (and I don't think your ideas about gateways would even count as  
half as good), and the IETF/IESG decide to make a problem out of this,  
then the news community will probably decide that, since they've gone so  
long without an IETF/IESG-blessed standard, they can do the same in the  
future; they'll just make their own standards, just the same as they've  
done all the time.

No, I don't speak for the news community. But I still think I know what  
their reaction to unreasonable opposition will be.

And I have no problems at all labelling what I see here as unreasonable.

> have a way to move forward that works and which satisfies the requirements
> of both communities. It likely will dump on gateways, but hey, those of us
> who write gateways are used to it.

I know more about news-to-X gateways that I ever wanted to, and the best  
you can do is not having them. If you do have them, correct software is  
only part of any solution; it also takes correct administration, and quite  
a lot of that, as well.

"Gateway", except in very precise circumstances, has become a sort of four  
letter word for news people. And for good reasons.

MfG Kai

Date: Fri, 8 Jan 1999 09:39:46 -0600 (CST)
From: Rick Troth <[email-masked]>
To: [email-masked]
Subject: canonicalization [was: Re: Restrictions on ...] 
Sender: [email-masked]

	Regarding the CTE issue in this thread,  I think we've 
again missed a significant but subtle characteristic of MIME: 
it is processed off-line.   We've gone to great lengths to ensure that 
encoded streams survive the various gyrations they get put through, 
but we still as a group seem to think of MIME as an on-the-wire thing. 
But in my experience it is most often NOT processed on-line. 
 
	We might seem to be in danger of overstepping our charter. 
IETF standards typically define the interactions between Internet hosts, 
which would imply SMTP rather than MIME.   Nevertheless,  what gets 
carried over SMTP (eg: MIME) clearly impacts the effectiveness of SMTP. 
And we MUST leave the IMPLEMENTOR the FREEDOM to process MIME as s/he 
sees best on whatever platform s/he is coding (on-line or off-line). 
So MIME might be a TCP creature,  or it might not. 
 
	Looking at NVT,  lines of text consist of printable ASCII 
[here you can substitute a wide range of character sets]  delimited by 
CR/LF.   It is convenient for Microsoft that their definition of a 
plain text file looks the same.   On UNIX,  this is not the case; 
lines are ASCII delimited by NL (or LF).   I am fond of a particular 
system where the character set is NOT ASCII,  nor any of its variants, 
and lines are delimited out-of-band.   VMS defines lines (records) OOB 
though it again uses good old ASCII.   Embedded ESC, TAB, BS, etc. are 
generally allowed and sometimes required. 
 
	Keep this in mind when you think about how new types 
will get processed.   Not all the world is an MS-DOS filesystem. 
 
	</soapbox> 
 
-- 
Rick Troth at La Casita, Houston, Texas, USA 
 
"We build our computers the way we build our cities -- 
over time, without a plan, on top of ruins. 
	-- Ellen Ullman, "The Dumbing Down of Programming" 

Date: Thu, 07 Jan 1999 16:32:00 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Charles Lindsey <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> > I'm sorry, but this just doesn't wash. For one thing, I fail to see
> > the urgency -- the format for news was last updated well over a decade
> > ago, if memory serves. 822bis is nearly done and should be out in
> > another month or two. I fail to see how any sense of urgency could
> > exist that warrants this kind of haste after a decade of inactivity.

> Well the course of action you were suggesting appeared to involve a
> 5-year wait to develop new protocols before the news people could even
> begin to address their part of the problem. I think the USEFOR group
> would rather find a solution that can be worked within the existing
> constraints.

I have no idea where you got the notion 822bis was going to take 5 years to do.
The course of action that's likely to take that long, if past experience is any
indicator, is the one you're on. (In case you didn't know, more or less the
same thing, with the same claims, was tried by the SMTPEXT group. And  the
result was that the proposal eventually went down in flames, but not before
wasting three whole years of time. And the IETF was much, much smaller then,
and much faster at getting things done.)

> > Your implicit assumption here is that you'll be allowed to move
> > forward with a specification that doesn't give detailed, specific, and
> > up to date syntax rules and without downgrading rules. Now, I'm not
> > in a position to say that you cannot do this. But I'll tell you right
> > now that I'll be one of the people reviewing your documents when they
> > make it to IETF last call, and I guarantee that I will object if these
> > matters aren't covered. I suspect many others will as well.

> With respect, you haven't been following the discussions on the USEFOR
> list. We are well aware that such additional functionality as we propose
> must degrade gracefully when it meets older systems. We are also well
> aware of what older systems are actually capable, and we think we
> have devised acceptable solutions within the news environment.

As I said before, once you reach last call I'll be one of the people who
will review your document. And then we'll see.

> When it comes to gatewaying with other environments we find some
> problems, and the solution we have proposed so far is admittedly
> sub-optimal. So if you people are able to point us at a better solution,
> then that would clearly be welcomed.

That is precisely what I tried to do in my previous message.

> > Again, I said that I see no problem with using message/rfc822 for this, and as
> > yet the only objection you've managed to raise is that such usage cannot be
> > accomplished on your timeline. If you have a technical reason why this cannot
> > be done I'd like to hear it.

> You are asking me to prove a negative, which is not always easy.

Where? I asked you for technical objections to using message/rfc822. Thus far
you haven't given any. You've only given arguments based on a timeline which
is, as far as I can tell, entirely fantastic. I see no place where I'm asking
you to prove a negative.

> Are
> you saying that message/rfc822 as it stands can do it? Surely not. News
> headers will be liable to contain characters with the top bit set.
> Message/rfc822 must follow the syntax of RFC-822 (or drums if you like)
> which forbids such headers (though allowing some email headers to be
> omitted).

Please reread my ealier response where I covered how to do this. Repeating:

First we finished 822bis so we have a stable, coherent specification to work
with. Then we develop a very detailed, very concrete specification for how to
do 8bit in headers along with downgrading rules. (Perhaps you have already done
this, but I cannot see how it is possible in the absence of 822bis.) 

Then we develop an SMTP extension for transmission of 8bit headers. (This is
easy and I have offered to do it.)

Then we formally extend the definition of message/rfc822 to include such
things. (This can happen independent of the SMTP extension. All it takes is a
citable specification.)

I figure all this will take six months to do if there's sufficient motivation
to actually do the work quickly. By contrast, I see no chance of your approach
producing any useful result in less than several years, assuming it ever does.

> If you are suggesting some extension of message/rfc822, then please
> indicate the sort of thing that might be allowed. The 8th bit could be
> admitted at the stroke of a pen, and one can always put CTE: 8bit on
> the outside, but that is not much help if you offer it to a transport
> that does not support 8BITMIME. AIUI, such a transport could properly
> re-encode the _body_ of the message and create/modify the CTE in the
> inner headers, but to re-encode those headers themselves would seem to
> need rather a large and incompatible extension to the existing stuff.
> Please correct me if I am wrong.

I'm fairly confident you're wrong. I cannot prove it since I don't have
a concrete proposal that addresses all the issues, but I think I see how
to do it, and I see no substantive technical objections to doing it. And
perhaps more to the point, you haven't come up with any either.

> > > So for the moment we have proposed
> > > application/news-message - not that we like it very much.

> > Please understand that I'm not being obstructive about this just for the heck
> > of it. I sincerely believe that this is a dreadful idea, one that will cause a
> > huge amount of pain to the installed base,...

> I agree that it is a dreadful idea, but I don't see that it will cause
> pain to the installed base. It will indeed cause pain to the users of
> the installed base because it will get treated like octet-stream and get
> shoved off to a file, which is probably not what the user wanted.

And we have a name for this: Lack of interoperability. And we have formal
rules about interoperability.

> > But even supposing we moved forward with this, the question becomes: Can you
> > get consensus around this idea faster than 822bis can complete and the job can
> > be done right?

> I would have thought so, but I would be happy to listen to arguments to
> the contrary.

It should be obvious that I cannot prove to you that the course of action I
suggest is viable. The only way to test the road is to travel down it. But I
certainly can tell you that my assessment is that the course you're on is one
of the toughest imagineable, one which I suspect is doomed to failure, and even
should it succeed will take far more time than you can imagine or afford. And
to be blunt, at least part of that I can more or less guarantee.

				Ned

Date: Thu, 7 Jan 1999 20:07:46 GMT
From: Charles Lindsey <[email-masked]>
To: [email-masked]
Subject: Re: Restrictions on the Content-Type "message"
Cc: [email-masked], [email-masked], [email-masked]
Reply-To: [email-masked], [email-masked], [email-masked]
Content-MD5: wW9VF1d8OSzT1byeyWzjPA==
Sender: [email-masked]

	On Wed, 06 Jan 1999 10:04:15 -0800 (PST)
	Ned Freed <[email-masked]> said...

> 
> I'm sorry, but this just doesn't wash. For one thing, I fail to see
> the urgency -- the format for news was last updated well over a decade
> ago, if memory serves. 822bis is nearly done and should be out in
> another month or two. I fail to see how any sense of urgency could
> exist that warrants this kind of haste after a decade of inactivity.

Well the course of action you were suggesting appeared to involve a
5-year wait to develop new protocols before the news people could even
begin to address their part of the problem. I think the USEFOR group
would rather find a solution that can be worked within the existing
constraints.
> 
> 
> Your implicit assumption here is that you'll be allowed to move
> forward with a specification that doesn't give detailed, specific, and
> up to date syntax rules and without downgrading rules. Now, I'm not
> in a position to say that you cannot do this. But I'll tell you right
> now that I'll be one of the people reviewing your documents when they
> make it to IETF last call, and I guarantee that I will object if these
> matters aren't covered. I suspect many others will as well.

With respect, you haven't been following the discussions on the USEFOR
list. We are well aware that such additional functionality as we propose
must degrade gracefully when it meets older systems. We are also well
aware of what older systems are actually capable, and we think we
have devised acceptable solutions within the news environment.

When it comes to gatewaying with other environments we find some
problems, and the solution we have proposed so far is admittedly
sub-optimal. So if you people are able to point us at a better solution,
then that would clearly be welcomed.
> 
> 
> Again, I said that I see no problem with using message/rfc822 for this, and as
> yet the only objection you've managed to raise is that such usage cannot be
> accomplished on your timeline. If you have a technical reason why this cannot
> be done I'd like to hear it.

You are asking me to prove a negative, which is not always easy. Are
you saying that message/rfc822 as it stands can do it? Surely not. News
headers will be liable to contain characters with the top bit set.
Message/rfc822 must follow the syntax of RFC-822 (or drums if you like)
which forbids such headers (though allowing some email headers to be
omitted).

If you are suggesting some extension of message/rfc822, then please
indicate the sort of thing that might be allowed. The 8th bit could be
admitted at the stroke of a pen, and one can always put CTE: 8bit on
the outside, but that is not much help if you offer it to a transport
that does not support 8BITMIME. AIUI, such a transport could properly
re-encode the _body_ of the message and create/modify the CTE in the
inner headers, but to re-encode those headers themselves would seem to
need rather a large and incompatible extension to the existing stuff.
Please correct me if I am wrong.
> 
> > So for the moment we have proposed
> > application/news-message - not that we like it very much.
> 
> 
> Please understand that I'm not being obstructive about this just for the heck
> of it. I sincerely believe that this is a dreadful idea, one that will cause a
> huge amount of pain to the installed base,...

I agree that it is a dreadful idea, but I don't see that it will cause
pain to the installed base. It will indeed cause pain to the users of
the installed base because it will get treated like octet-stream and get
shoved off to a file, which is probably not what the user wanted.
> 
> > I suppose something like the following might be possible:
> 
> > Content-Transfer-Encoding: 8bit
> > Content-Type: message/news; boundary="foofoo" (or should it be 
multipart/news?)
> > Other-Email-Headers:
> 
> > This is an encapsulated news article.
> 
> > --foofoo
> > Content-Transfer-Encoding: quoted-printable (1111)
> > Content-Type: application/news-headers; charset=utf-8 (1111)
> >     (or text/news-headers?)
> 
> > Newsgroups: various.groups.in.some.greek.hierarchy.using.utf-8.characters
> > Message-ID: <[email-masked]> (message-ids are still strict ascii)
> > Path: here!there!everywhere
> > Subject: Some Greek text, no doubt
> > Content-Transfer-Encoding: 8bit
> > Content-Type: text/plain; charset=iso-8859-6 (or whatever the Greek iso code 
is)
> > Other-News-Headers:
> 
> > --foofoo
> > Content-Transfer-Encoding: base64 (2222)
> > Content-Type: text/plain; charset=iso-8859-6 (2222)
> 
> > The body of the article, written in greek using iso-8859-6 (whatever).
> 
> > --foofoo--
> 
> An interesting idea, and certainly one that avoids obvious pitfalls. But it is
> also unspeakably ugly. And an argument can certainly be makde that
> application/news-headers should be a subtype of message.

Well precisely what you call the various types is a detail to be
discussed. I was aiming for something that would survive unscathed
through a system that tried to process it as multipart/mixed. And
I was hoping that the headers of the two parts could be omitted in
straighforward cases (I gather that CT defaults to text/plain, but I
cannot find where in RFC-2046 it says what CTE defaults to inside a
multipart).
> 
> But even supposing we moved forward with this, the question becomes: Can you
> get consensus around this idea faster than 822bis can complete and the job can
> be done right?

I would have thought so, but I would be happy to listen to arguments to
the contrary.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Thu, 07 Jan 1999 11:12:01 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: [email-masked]
Cc: [email-masked]
Sender: [email-masked]

> The way I understand it, application/news-transmission is a tunneling
> protocol. As such, it's *requirements* are:

> 1. Upon unpacking from the tunnel, the articles in question must be bit-by-
> bit *identical* to the articles that went in. That means redoing nested
> encodings is Right Out[tm].

> 2. It being news, and news wanting no 7 bit restriction for very good
> reasons (such as those being known to lead to flamewars all the time), the
> mail restriction to 7 bit means that some sort of CTE is necessary here.

> Now, the combination of 1 and 2 means there is no possible way to do this
> without using nested encodings. OTOH, 1 also means that nested encodings
> should not be a problem, since nobody is expected to *handle* nested
> encodings - the mail and gateway parts only ever handle the outer
> encodings, and other news software only ever sees the inner encodings.

There's nothing new here -- these are exactly the same arguments we heard 7
years ago when we first did MIME, and I understand them all too well. The point
you don't seem to be getting is that we considered all these choices way back
when MIME was designed, and we decided that we would not allow such
requirements to be met. Let me repeat that: We would not allow it. We
understood that situations would arise where such requirements would be given,
but we decided not to cater to them. So we wrote MIME with a no nested
encodings rule, and we didn't make an exemption for tunneling or anything else.

You can make up all the requirements you want. I can make up a requirement that
I want to be able to send packets will 50 billion bytes in them. But I cannot
do that with IPv4, and just because I've come up with a requirement to do it
doesn't mean that IPv4 has to change to be able to meet my needs. And it
doesn't mean I'll be able to get my protocol standardized when it requires such
a change in IPv4.

You're attempting to go back on a decision that was made some time ago, rightly
or wrongly, and which implementations have gone along with in good faith, and
which has resulted in a situation where if you're allowed to proceed along the
path you've chosen the result becomes untenable.

Now, the IETF has no formal rule that says one standard cannot unmake a rule
made in another standard. And there's no formal rule that says one standard
cannot change things that break old implementations. So in theory you can
register a new type that does these sorts of things to MIME.

But there is a formal rule that things have to interoperate. You cannot proceed
down the standards track in the presence of interoperability problems and you
cannot even get on the track if it can be shown that your proposal is
guaranteed to cause interoperability problems. Now, we all know that mail and
news are gatewayed all the time. Messages pass from news to mail and from mail
to news, sometimes multiple times.

So what happens when a application/news-message object passes from news to
mail, one which contains 8bit characters in the data, and the mail system
has to downgrade to 7bit? There are two choices:

(1) The mail system treats the object as opaque and slaps on a CTE. But then
    when someone tries to read this object using a mail reader, they see an
    opaque blob. The content, which likely contains perfectly reasonable MIME
    objects, is unreadable. The result is a failure to interoperate, one which
    would effectively prevent any sort of move to draft standard.
(2) The mail system treats the object as composite and encodes the inner parts.
    This puts the mail system in direct violation of your no-encoding rule.
    And a standard which has to be violated for things to work isn't acceptable.

Either way you have a violation of the intent, if not the letter, of the
standards process. And this is why I'm saying that the course you're on isn't
viable.

The long and short of it is that you need to give up on this idea of strict
tunneling. And once you do, a number of approaches open up that I think will
meet your real requirements without breaking things and which are viable as
standards.

The most obvious way to do this is to treat this as a gateway problem and
define the operation of mail <--> news gateways to get what you want. For
example, you can say that such gateways have to convert
application/news-message to message/rfc822; type=news and vice versa. Software
that adds and removes encodings is well understood, and should be no problem to
implement. It is especially easy if you deprecate use of quoted-printable and
base64 encoding of 8bit material in news (something you likely want to do
anyway). In such a case the gateway can simply remove quoted-printable and
base64 encoding of 8bit material willy-nilly; it need not consider what the
original encoding was. The result should be that headers are, modulo folding,
always returned to their original state and that bodies and their encodings are
preserved bit-for-bit. And you can further refine this with additional
restrictions, such as saying that boundary markers and such have to be
preserved, header lines cannot be moved (MIME actually says this already),
whatever.  You're never going to get to the point where exact bit-for-bit
compatibility is achieved, but there was never any chance of that happening
with mail transport of compositive MIME objects anyhow, no matter what the
standards say or how you label them, and this is something you're just going to
have to live with. (And if this causes problems for security services, well,
there are solutions to this that don't require bit-for-bit preservation of
content, which I'd be happy to elaborate on if need be.)

And once you do this, the rationale for having a new type vanishes. All you
will be doing is restricting what sorts of operations news systems can perform
on message/rfc822; type=news. But that is perfectly acceptable -- any standard
can at any time select a consistent subset of another standard.

> Now, the case of application/news-message is not quite so simple. The way
> I understand it, this one is actually meant (among others) for situations
> where software would need to act on these nested encodings.

> *However*, it still runs up against the problem that

> a. News is meant to be 8 bit clean (not binary clean, though)

> b. News is built around the assumption that, once a news article is
> underway, most of the article headers and all of the body are safe from
> change, in stark contrast to the mail case. (Typically, the exeptions are
> the Path: header and the Xref: header.) There are applications around
> *right* now that rely on this.

> Obviously, a means that the headers of a news message, if sent by mail,
> may still need to have a CTE applied. And b means that redoing CTEs is
> still a problem.

> So, it seems to me that we have here a case where the design goals of the
> news communbity are actually incompatible with the design goals of the
> MIME community, and there is no technical solution that will completely
> satisfy both sides.

I disagree. I don't think you've explored all the options for moving forward. I
think that you've picked a way forward that solves one set of problems but not
a bunch of others (you've admitted your solution doesn't handle the outermost
header of a news message moving through mail), and moreover it causes lots of
new problems, many of them of a sort that are going to give you a great deal of
trouble in the standards process. But you've gotten hung up on how this one
solution works and made the conditions of it absolute. I think that with a bit
of compromise on your part we'll have a way to move forward that works and
which satisfies the requirements of both communities. It likely will dump
on gateways, but hey, those of us who write gateways are used to it.

				Ned

Date: 07 Jan 1999 16:20:00 +0200
From: [email-masked] (Kai Henningsen)
To: [email-masked]
Subject: Re: Restrictions on the Content-Type "message"
Organization: Organisation? Me?! Are you kidding?
Comment: Unsolicited commercial mail will incur an US$100 handling fee per received mail.
Sender: [email-masked]

[email-masked] (Ned Freed)  wrote on 06.01.99 in <[email-masked]>:

> > Not so. Within the news environment, all those questions have been
> > addressed, and downgrading to 7bit is not needed. Gatewaying may get a
> > bit complicated, but application/news-transmission (which is aimed at
> > machines rather than humans) gets over some of the problems. Which just
> > leaves the matter of a human friendly way to transmit news across email
> > and have is restorable to exactly the same thing at the far end.
>
> Your implicit assumption here is that you'll be allowed to move forward with
> a specification that doesn't give detailed, specific, and up to date syntax
> rules and without downgrading rules. Now, I'm not in a position to say that

The way I understand it, application/news-transmission is a tunneling  
protocol. As such, it's *requirements* are:

1. Upon unpacking from the tunnel, the articles in question must be bit-by- 
bit *identical* to the articles that went in. That means redoing nested  
encodings is Right Out[tm].

2. It being news, and news wanting no 7 bit restriction for very good  
reasons (such as those being known to lead to flamewars all the time), the  
mail restriction to 7 bit means that some sort of CTE is necessary here.

Now, the combination of 1 and 2 means there is no possible way to do this  
without using nested encodings. OTOH, 1 also means that nested encodings  
should not be a problem, since nobody is expected to *handle* nested  
encodings - the mail and gateway parts only ever handle the outer  
encodings, and other news software only ever sees the inner encodings.

Now, the case of application/news-message is not quite so simple. The way  
I understand it, this one is actually meant (among others) for situations  
where software would need to act on these nested encodings.

*However*, it still runs up against the problem that

a. News is meant to be 8 bit clean (not binary clean, though)

b. News is built around the assumption that, once a news article is  
underway, most of the article headers and all of the body are safe from  
change, in stark contrast to the mail case. (Typically, the exeptions are  
the Path: header and the Xref: header.) There are applications around  
*right* now that rely on this.

Obviously, a means that the headers of a news message, if sent by mail,  
may still need to have a CTE applied. And b means that redoing CTEs is  
still a problem.

So, it seems to me that we have here a case where the design goals of the  
news communbity are actually incompatible with the design goals of the  
MIME community, and there is no technical solution that will completely  
satisfy both sides.

Note that "822 doesn't allow 8 bits in headers" is *not* the problem here;  
that's just a red herring. The real problem here is that (I) people want  
to transmit complex, 8 bit MIME data through an environment that has 7 bit  
body restrictions, and (II) redoing MIME encodings can lead to bad  
effects, something that, IIRC, has come up several times in the mail  
security WGs, and in this case, this includes both body and header.

I must say that I certainly do not see a good solution here. If someone  
else has a bright idea, I'm sure people will be happy to hear about it.

If all mail were 8 bit clean, this would be a non-issue. However, this is  
not the case.

MfG Kai

From: Keith Moore <[email-masked]>
To: [email-masked]
cc: Keith Moore <[email-masked]>, IETF mailing list on MIME and e-mail <[email-masked]>
Subject: Re: Supersedes versus Replaces 
Date: Thu, 07 Jan 1999 10:46:19 -0500
Sender: [email-masked]

> However, is there a good
> programmatic way to decide whether a given object is email or netnews?

No, I don't think so.  You have to evaluate the object in the context
from which it was obtained.  If it came from NNTP, it's netnews;
if it came from POP or IMAP or a mailbox, it's email.

(part of the problem is that there's no standard for email<>news
gateways, nor is there one for sending an email reply to a news article.
so there are too many things that send news-specific headers over email
to use those things as heuristics.)

To: Keith Moore <[email-masked]>
Cc: IETF mailing list on MIME and e-mail <[email-masked]>
Subject: Re: Supersedes versus Replaces 
From: [email-masked]
Date: Thu, 07 Jan 1999 10:42:22 -0500
Sender: [email-masked]

--==_Exmh_-1157436220P
Content-Type: text/plain; charset=us-ascii

On Wed, 06 Jan 1999 20:26:29 EST, Keith Moore said:
> I recommend we pursue a more limited goal of allowing the same
> engine to *display* both email and usenet messages, while realizing
> that other operations - like reply, supercedes, and others,
> are going to be fundamentally different for one than the other.

This sounds like a reasonable goal.  However, is there a good
programmatic way to decide whether a given object is email or netnews?
Since June 1998, I've received at least 656 pieces of e-mail that
contained a 'Newsgroups:' header (due to passing through news->mail
gateways, or being forwarded, etc), or about 4% of all the mail
I've bothered keeping statistics for.  I don't think 96% accuracy
is good enough here - is there a better way to tell which flavor
of MIME message a given object is?

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_-1157436220P
Content-Type: application/pgp-signature

--==_Exmh_-1157436220P--

From: Keith Moore <[email-masked]>
To: Jacob Palme <[email-masked]>
cc: IETF mailing list on MIME and e-mail <[email-masked]>
Subject: Re: Supersedes versus Replaces 
Date: Wed, 06 Jan 1999 20:26:29 -0500
Sender: [email-masked]

While I'd like to keep the two formats as similar as possible
(and thus maximize the amount of sharable code)
it's far too late to insist that Usenet and email use the same
message format.  Fundamental protocol elements like From and Reply-To
have always had significant differences between email and usenet. 
Heck, usenet doesn't even separate the delivery envelope from content.

I recommend we pursue a more limited goal of allowing the same
engine to *display* both email and usenet messages, while realizing
that other operations - like reply, supercedes, and others,
are going to be fundamentally different for one than the other.

Keith

From: Keith Moore <[email-masked]>
To: Ned Freed <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Restrictions on the Content-Type "message" 
Date: Wed, 06 Jan 1999 20:21:02 -0500
Sender: [email-masked]

    Adding a sense of urgency to a protocol specification effort usually
    slows the effort down rather than speeding it up.

bingo.

Date: Wed, 6 Jan 1999 21:16:45 +0100
To: Charles Lindsey <[email-masked]>, [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
Cc: [email-masked], [email-masked]
Sender: [email-masked]

At 15.27 +0100 99-01-05, Charles Lindsey wrote:
> Then I go to my local newsgroup and they are not there :-(. Presumably
> because, due to anti-spam measures, Ned does not have posting privileges
> to the list. Cc to Brad to ask him to fix this, so at least Ned can post
> to the list. I would also hope he would subscribe to it, at least while
> this present thread runs. (email to [email-masked] with
> subscribe your-email-address in the body).

For the mailing lists which I manage on IETF issues,
everyone can post, but contributions by non-members are
moderated by me. This means that contributions by
non-members can be delayed compared to contributions by
members.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

Date: Wed, 6 Jan 1999 21:22:02 +0100
To: IETF mailing list on MIME and e-mail  <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Supersedes versus Replaces
Sender: [email-masked]

There is some leaning in the usefor group to keep
Supersedes as it is, but let it deprecate with time, and
invent a new header "Replaces". This new header might have
a parameter with values such as "minor", "major" and "faq".

The reason for this is that many existing news server
cannot handle even the simplest change in the syntax for
Supersedes.

Since e-mail does not have any standard yet for this
header, this might mean that such a header should be named
"Replaces" and not "Supersedes" in e-mail. We  will have to
wait for the conclusions in usefor before this is decided.
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme

Date: Wed, 06 Jan 1999 10:04:15 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Charles Lindsey <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> But it is unreasonable to say that all other internet developments (news
> in this case) have to wait until 822bis is completed, and then some new
> design team is formed, before they can proceed. This is a problem we
> have to solve _now_.

So what you're saying is that the matter is urgent, and its urgency
justifies ignoring dependencies on other work.

I'm sorry, but this just doesn't wash. For one thing, I fail to see the urgency
-- the format for news was last updated well over a decade ago, if memory
serves. 822bis is nearly done and should be out in another month or two. I fail
to see how any sense of urgency could exist that warrants this kind of haste
after a decade of inactivity.

But even if your sense of urgency was justifiable, it still wouldn't wash. The
need for IPP is urgent. The need for security in mail protocols is urgent. And
various other needs in the IETF are urgent. But protocols, including these, are
routinely delayed, sometimes for _years_, because of such dependencies. Heck,
the cumulative delays to MIME because of possible dependencies in other areas
probably cost us a year if not more.

Indeed, I once devised a corrollary to one of Brook's laws, which says:

    Adding a sense of urgency to a protocol specification effort usually
    slows the effort down rather than speeding it up.

I think you would do well to heed it, because it definitely seems to apply to
USEFOR.

> > Please note that the notion that you can simply say it is OK to use
> > 8bit in headers is a dog that does not hunt. You to have specify where
> > it can be used, what its semantics are, you have to specify how to
> > downgrade it to 7bit, and you have to give a revised grammar for the
> > result. Nothing less is likely to pass muster. And while I'm fairly
> > sure that UTF-8 is the right answer here, there may be substantive
> > profiling issues with how UTF-8 is used (language tags in particular
> > come to mind).

> Not so. Within the news environment, all those questions have been
> addressed, and downgrading to 7bit is not needed. Gatewaying may get a
> bit complicated, but application/news-transmission (which is aimed at
> machines rather than humans) gets over some of the problems. Which just
> leaves the matter of a human friendly way to transmit news across email
> and have is restorable to exactly the same thing at the far end.

Your implicit assumption here is that you'll be allowed to move forward with a
specification that doesn't give detailed, specific, and up to date syntax rules
and without downgrading rules. Now, I'm not in a position to say that you
cannot do this. But I'll tell you right now that I'll be one of the people
reviewing your documents when they make it to IETF last call, and I guarantee
that I will object if these matters aren't covered. I suspect many others will
as well.

And then it will be up to the IESG. I'm not on the IESG, and even if I was I
would be only one vote, but I wouldn't wager much on the chances of getting
this sort of thing through.

> We had hoped to use message/something for this, and your earlier
> message seemed to suggest that it could be done within the existing
> constraints, but it seems not to be so.

Again, I said that I see no problem with using message/rfc822 for this, and as
yet the only objection you've managed to raise is that such usage cannot be
accomplished on your timeline. If you have a technical reason why this cannot
be done I'd like to hear it.

> So for the moment we have proposed
> application/news-message - not that we like it very much.

Now, this happens to be an area where I, as type reviewer, have an opinion that
carries weight. And in my capacity as type reviewer I'm telling you now that I
will object to the registration of this type. And that, I believe, pushes the
matter before the IESG and IAB. And that is likely to take time -- lots of it.

Please understand that I'm not being obstructive about this just for the heck
of it. I sincerely believe that this is a dreadful idea, one that will cause a
huge amount of pain to the installed base, and one which should not be allowed
to move forward. But one of the side effects of this will almost certainly
be to make use of 822bis possible. And I cannot say I'm unhappy about
that.

> I suppose something like the following might be possible:

> Content-Transfer-Encoding: 8bit
> Content-Type: message/news; boundary="foofoo" (or should it be multipart/news?)
> Other-Email-Headers:

> This is an encapsulated news article.

> --foofoo
> Content-Transfer-Encoding: quoted-printable (1111)
> Content-Type: application/news-headers; charset=utf-8 (1111)
>     (or text/news-headers?)

> Newsgroups: various.groups.in.some.greek.hierarchy.using.utf-8.characters
> Message-ID: <[email-masked]> (message-ids are still strict ascii)
> Path: here!there!everywhere
> Subject: Some Greek text, no doubt
> Content-Transfer-Encoding: 8bit
> Content-Type: text/plain; charset=iso-8859-6 (or whatever the Greek iso code is)
> Other-News-Headers:

> --foofoo
> Content-Transfer-Encoding: base64 (2222)
> Content-Type: text/plain; charset=iso-8859-6 (2222)

> The body of the article, written in greek using iso-8859-6 (whatever).

> --foofoo--

An interesting idea, and certainly one that avoids obvious pitfalls. But it is
also unspeakably ugly. And an argument can certainly be makde that
application/news-headers should be a subtype of message.

But even supposing we moved forward with this, the question becomes: Can you
get consensus around this idea faster than 822bis can complete and the job can
be done right?

> > Then we'll likely need to define an SMTP extension to negotiate
> > the use of 8bit headers in mail.

> There already is such an extension (8BITMIME) but it is only a SHOULD,
> not a MUST ;-( .

As a coauthor of 8bitMIME, I am well aware of its existance. However, it
doesn't accomodate 8bit headers.

And as a practical matter any such scheme can never be anything but a SHOULD.
And this is why you cannot avoid the downgrade issue, much as you may like to.

I'll even make you an offer: Do the 8bit in headers job right and I'll write
the 8bitheader SMTP extension document to facilitate getting this done
faster.

And HTTP gatewaying probably needs to be considered as well.

				Ned

Date: Wed, 6 Jan 1999 11:31:54 GMT
From: Charles Lindsey <[email-masked]>
To: [email-masked]
Subject: Re: Restrictions on the Content-Type "message"
Cc: [email-masked], [email-masked], [email-masked]
Reply-To: [email-masked], [email-masked], [email-masked], [email-masked]
Content-MD5: EXK8ztmUrgH8fKRCPYKbbw==
Sender: [email-masked]

	On Tue, 05 Jan 1999 09:25:00 -0800 (PST)
	Ned Freed <[email-masked]> said...

> > BTW, what is [email-masked]? Some RFC822-email list I presume. Should
> > it continue to be included in these exchanges?
> 
> ietf-822 was the list where MIME was developed. As such, it is the right
> place for discussions of MIME-related issues.

Fine. Someone has pointed me to its website where I can retrieve the
missing articles. Maybe I should join that list to continue this
discussion.

I have CCed this to the usenet-format-list, but not in the Reply-To
line, so Ned will not get any more bounces.

> > So, dealing with the points raised as best I can remember them, Ned
> > seems to be suggesting that it is possible to cope with headers with
> > 8bit characters in them within the existing message structure, even
> > within message/rfc822. But I just do not see how this is possible. So
> > please can we have an explanation of how it's done, preferably with a
> > simple example. I agree it would be far better to use a message type
> > if it can be done, and using an application type was known to be a
> > second-best kludge when it was put forward.
> 
> I already outlined the path forward: First we finish 822bis, so we
> have a coherent specification to work from. Then we form a design team
> to put together a specification for how 8bit in headers would work.

But it is unreasonable to say that all other internet developments (news
in this case) have to wait until 822bis is completed, and then some new
design team is formed, before they can proceed. This is a problem we
have to solve _now_.

> Please note that the notion that you can simply say it is OK to use
> 8bit in headers is a dog that does not hunt. You to have specify where
> it can be used, what its semantics are, you have to specify how to
> downgrade it to 7bit, and you have to give a revised grammar for the
> result. Nothing less is likely to pass muster. And while I'm fairly
> sure that UTF-8 is the right answer here, there may be substantive
> profiling issues with how UTF-8 is used (language tags in particular
> come to mind).

Not so. Within the news environment, all those questions have been
addressed, and downgrading to 7bit is not needed. Gatewaying may get a
bit complicated, but application/news-transmission (which is aimed at
machines rather than humans) gets over some of the problems. Which just
leaves the matter of a human friendly way to transmit news across email
and have is restorable to exactly the same thing at the far end.

We had hoped to use message/something for this, and your earlier
message seemed to suggest that it could be done within the existing
constraints, but it seems not to be so. So for the moment we have proposed
application/news-message - not that we like it very much.

I suppose something like the following might be possible:

Content-Transfer-Encoding: 8bit
Content-Type: message/news; boundary="foofoo" (or should it be multipart/news?)
Other-Email-Headers:

This is an encapsulated news article.

--foofoo
Content-Transfer-Encoding: quoted-printable (1111)
Content-Type: application/news-headers; charset=utf-8 (1111)
    (or text/news-headers?)

Newsgroups: various.groups.in.some.greek.hierarchy.using.utf-8.characters
Message-ID: <[email-masked]> (message-ids are still strict ascii)
Path: here!there!everywhere
Subject: Some Greek text, no doubt
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=iso-8859-6 (or whatever the Greek iso code is)
Other-News-Headers:

--foofoo
Content-Transfer-Encoding: base64 (2222)
Content-Type: text/plain; charset=iso-8859-6 (2222)

The body of the article, written in greek using iso-8859-6 (whatever).

--foofoo--

NOTE: The lines marked (1111) and (2222) would ideally use CTE: 8bit,
but could lawfully be changed to something else if a transport without
8BITMIME capability was encountered en route. It may be that they could
be omitted entirely in the default (8bit) case.

I cannot say I like such a scheme, but at least it might do something
reasonable on existing implementations, and it does not look too bad if
the recipient gets to see the text exactly as I wrote it above.

> Then we'll likely need to define an SMTP extension to negotiate
> the use of 8bit headers in mail.

There already is such an extension (8BITMIME) but it is only a SHOULD,
not a MUST ;-( .

> I don't know about NNTP. You say
> that NNTP is 8bit clean, and who am I to argue, but this leaves
> the substantive issue of how to handle NNTP gateways to other
> environments, to say nothing of client support for UTF-8.

NNTP is not a problem. Client support for UTF-8 is obviously needed
in newsreaders, but only for users who want to read Greek (whatever)
newsgroups, and they are probably going to need special clients in any
scheme. It seems that news servers do not need to understand UTF-8 at
all and will need minimal upgrading (if any).
>
> And once all this is done there should be no problem putting 8bit
> message headers in message/rfc822 parts.
> 
> 				Ned
> 

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Tue, 05 Jan 1999 09:25:00 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Charles Lindsey <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> I read all of Ned's interesting replies on this topic, observed that
> they had all been CCed to [email-masked], and deleted them,
> intending to deal with them when I dealt with the day's postings to
> the usenet-format list (which I have gated to a local newsgroup on my
> machine).

> Then I go to my local newsgroup and they are not there :-(. Presumably
> because, due to anti-spam measures, Ned does not have posting privileges
> to the list. Cc to Brad to ask him to fix this, so at least Ned can post
> to the list.

I've long since gotten sick and tired of these sorts of draconian anti-spam
measures being applied to lists, especially to IETF lists where the
operational rules forbid it.

As I operate a number of IETF lists myself, which as it happens receive a huge
amount of spam, and I manage to do it in ways that prevent of the spam from
getting to the list, I absolutely reject that things have to be this way.

Unfortunately I have neither the time nor the patience to engage in a
protracted struggle with list managers, in particular a struggle which
previously has never resulted in a satisfactory outcome. So I didn't complain,
and simply deleted the bounce messages.

However, please be aware that development of protocols has to be an open
process, and should someone file an objection/appeal of a protocol developed on
such a list, speaking as an IAB member I for one would have a difficult time
rejecting such an objection out of hand.

> I would also hope he would subscribe to it, at least while
> this present thread runs. (email to [email-masked] with
> subscribe your-email-address in the body).

Sorry. I already received well over 400 messages a day. I have limited
bandwidth and the USEFOR list is _extremely_ high volume. I have no time
to spare to deal with it.

> So could anybody who still has copies let me have them, or better post
> them to the list?

> BTW, what is [email-masked]? Some RFC822-email list I presume. Should
> it continue to be included in these exchanges?

ietf-822 was the list where MIME was developed. As such, it is the right
place for discussions of MIME-related issues.

> So, dealing with the points raised as best I can remember them, Ned
> seems to be suggesting that it is possible to cope with headers with
> 8bit characters in them within the existing message structure, even
> within message/rfc822. But I just do not see how this is possible. So
> please can we have an explanation of how it's done, preferably with a
> simple example. I agree it would be far better to use a message type
> if it can be done, and using an application type was known to be a
> second-best kludge when it was put forward.

I already outlined the path forward: First we finish 822bis, so we have a
coherent specification to work from. Then we form a design team to put together
a specification for how 8bit in headers would work. Please note that the notion
that you can simply say it is OK to use 8bit in headers is a dog that does not
hunt. You to have specify where it can be used, what its semantics are, you
have to specify how to downgrade it to 7bit, and you have to give a revised
grammar for the result. Nothing less is likely to pass muster. And while I'm
fairly sure that UTF-8 is the right answer here, there may be substantive
profiling issues with how UTF-8 is used (language tags in particular come to
mind). 

Then we'll likely need to define an SMTP extension to negotiate the use of 8bit
headers in mail. I don't know about NNTP. You say that NNTP is 8bit clean, and
who am I to argue, but this leaves the substantive issue of how to handle NNTP
gateways to other environments, to say nothing of client support for UTF-8.

And once all this is done there should be no problem putting 8bit message
headers in message/rfc822 parts.

				Ned

Date: Tue, 5 Jan 1999 14:27:16 GMT
From: Charles Lindsey <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re: Restrictions on the Content-Type "message"
Cc: [email-masked], [email-masked]
Content-MD5: WHWiQbWaTmo3cPYcIC1WyA==
Sender: [email-masked]

Damn!

I read all of Ned's interesting replies on this topic, observed that
they had all been CCed to [email-masked], and deleted them,
intending to deal with them when I dealt with the day's postings to
the usenet-format list (which I have gated to a local newsgroup on my
machine).

Then I go to my local newsgroup and they are not there :-(. Presumably
because, due to anti-spam measures, Ned does not have posting privileges
to the list. Cc to Brad to ask him to fix this, so at least Ned can post
to the list. I would also hope he would subscribe to it, at least while
this present thread runs. (email to [email-masked] with
subscribe your-email-address in the body).

So could anybody who still has copies let me have them, or better post
them to the list?

BTW, what is [email-masked]? Some RFC822-email list I presume. Should
it continue to be included in these exchanges?

So, dealing with the points raised as best I can remember them, Ned
seems to be suggesting that it is possible to cope with headers with
8bit characters in them within the existing message structure, even
within message/rfc822. But I just do not see how this is possible. So
please can we have an explanation of how it's done, preferably with a
simple example. I agree it would be far better to use a message type
if it can be done, and using an application type was known to be a
second-best kludge when it was put forward.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Tue, 05 Jan 1999 11:11:02 +0800
To: Ned Freed <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
Cc: Ned Freed <[email-masked]>, Charles Lindsey <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

At 01:46 PM 1/4/99 -0800, Ned Freed wrote:
>I completely agree, which is why I suggested a parameter to message/rfc822
>as the solution. We certainly do not want to define a flotilla of new
>composite subtypes to cover all these cases.

(my suggested passed yours in the mail.)

parameter sounds dandy.

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400 MALAYSIA
Brandenburg Consulting                                          
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464              675 Spruce Dr., Sunnyvale, CA 94086 USA

Date: Mon, 04 Jan 1999 13:46:03 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Dave Crocker <[email-masked]>
Cc: Ned Freed <[email-masked]>, Charles Lindsey <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> > This would be a valid point if message/rfc822 were strictly aligned with
> > RFC822. But it isn't -- the name, which was chosen and implemented long
> > before the exact rules for the type were put in place, is misleading.

> > Message/rfc822 was specifically designed to allow for material that isn't
> > legal according to RFC822 (and, I suspect, DRUMS). Specifically, the
> > requirements for what headers have to be present are substantially relaxed,
> > as are the syntax rules.

> I think this suggests a legitimate question about some sort of labeling
> which notes that the content really is (honest to goodness) conformant
> specifically with 822, or with netnews, or whatever.  (822 variants, such
> as an incomplete draft message, versus a fully 'received' one are useful
> distinctions, too.)

I completely agree, which is why I suggested a parameter to message/rfc822
as the solution. We certainly do not want to define a flotilla of new
composite subtypes to cover all these cases.

				Ned

Date: Tue, 05 Jan 1999 03:05:23 +0800
To: Ned Freed <[email-masked]>
From: Dave Crocker <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
Cc: Charles Lindsey <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

At 09:57 AM 1/4/99 -0800, Ned Freed wrote:
>This would be a valid point if message/rfc822 were strictly aligned with
>RFC822. But it isn't -- the name, which was chosen and implemented long
>before the exact rules for the type were put in place, is misleading.
>
>Message/rfc822 was specifically designed to allow for material that isn't
>legal according to RFC822 (and, I suspect, DRUMS). Specifically, the
>requirements for what headers have to be present are substantially relaxed,
>as are the syntax rules.

I think this suggests a legitimate question about some sort of labeling
which notes that the content really is (honest to goodness) conformant
specifically with 822, or with netnews, or whatever.  (822 variants, such
as an incomplete draft message, versus a fully 'received' one are useful
distinctions, too.)

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400 MALAYSIA
Brandenburg Consulting                                          
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464              675 Spruce Dr., Sunnyvale, CA 94086 USA

Date: Mon, 04 Jan 1999 10:38:29 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Jamie Zawinski <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> The fact is that a news message is not an RFC 822 message.  It is
> very similar to one, but it does not conform to RFC 822, and labelling
> it in such a way that implies that it does is going to cause problems.

> There is a need for a Content-Type that distinguishes RFC 822 mail
> messages from RFC 1036 news messages.

> (Yes, I'm talking about interpretation of the "Newsgroups" header,
> among other things.)

Then I would suggest adding a paremeter to message/rfc822. Again, the cost of
adding new composite types is very high, and has to be taken into account
when making these choices.

> Making the Content-Type that describes a news message not be a subtype
> of "message/" is just unspeakably lame.

I agree.

And moreover, there is no reason for doing this. Look at it this way: The
reason for wanting to put it under application is to avoid the rule about
encodings that applies to message subtypes. The reason USEFOR wants to avoid
this is so that they can pass messages to restricted transports without
breaking their "cannot change the inner CTEs" rule. In other words, they want
nested encodings.

Now, we had a strong consensus in the MIME WG that such things were not to be
allowed, at all, ever, in any context. And if that consensus still exists (and
I suspect it  does) then this won't fly no matter what top-level type you put
it under. You cannot get there from here. So there's no advantage to not using
a subtype of message.

But suppose the consensus doesn't exist any longer, and moreoever we're will to
say "screw the installed base" that implemented to this consensus in good
faith. (I don't think this going to happen, but let's suppose.) In such a case
the right thing to do would be to lift the restrictions on message/rfc822. Or
failing that, on the message type. So again there's no advantage to not using a
subtype of message.

Bottom line: Regardless of what restrictions currently exist on message, the
right path  forward is for news messages to be a subtype of message. You solve
nothing by using some other branch of the type tree.

BTW, one argument I suspect to hear is that this is news, not mail, and what
applies to news doesn't apply to mail. But all you have to do is look at the
number of news to email gateways to see how this doesn't wash.

				Ned

Date: Mon, 04 Jan 1999 10:26:21 -0800
From: Jamie Zawinski <[email-masked]>
Organization: mozilla dot org
To: Charles Lindsey <[email-masked]>
CC: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Restrictions on the Content-Type "message"
Sender: [email-masked]

The fact is that a news message is not an RFC 822 message.  It is 
very similar to one, but it does not conform to RFC 822, and labelling
it in such a way that implies that it does is going to cause problems.

There is a need for a Content-Type that distinguishes RFC 822 mail
messages from RFC 1036 news messages.

(Yes, I'm talking about interpretation of the "Newsgroups" header,
among other things.)

Charles Lindsey wrote:
>
>        application/news-message

Making the Content-Type that describes a news message not be a subtype
of "message/" is just unspeakably lame.

-- 
Jamie Zawinski
[email-masked]    http://www.mozilla.org/   (work)
[email-masked]        http://www.jwz.org/       (play)

Date: Mon, 04 Jan 1999 09:57:24 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Charles Lindsey <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> I don't think it affects e-mail standards at all, since message/rfc822
> does not have this problem (at least, not with e-mail as currently
> specified by DRUMS - they are going to run into this problem if/when
> they decide to allow UTF-8 in headers at some future date).

> IMHO RFC-2045 is too restrictive. It should have allowed each
> message/type to decide this issue for itself. In fact, RFC-2045
> contradicts itself for it says, in the first paragraph of 6.4:

This is false. See below.

> "... If an entity is of type "multipart" the Content-Transfer-Encoding
> is not permitted to have any value other than "7bit", "8bit" or
> "binary". Even more severe restrictions apply to some subtypes of the
> "message" type."

> That seems to suggest that the restriction applies to all multipart
> types (which is fine by me) but not to message types unless some
> particular message-type so specifies (as message/rfc822 indeed does).
> This view is confirmed by the 7th paragraph of 5.1 of RFC-2046 (as
> regards multipart-types) and the 3rd paragraph of section 5.2 of
> RFC-2046 which states:

> "Subtypes of "message" often impose restrictions on what encodings are
> allowed. These restrictions are described in conjunction with each
> specific subtype."

That's neither what it suggests nor says. What is says is that even
more restrictive rules apply to some subtypes of message, which is a
simple fact: The partial subtype of message is restricted to 7bit and
doesn't allow 8bit or binary.

The mistake you're making in your claim that this is contradictory is that it
refers to message/rfc822. It doesn't.

> and the followinng section of RFC-2046 then goes on to specifiy exactly
> such a restriction in the case of message/rfc822. That interpretation
> would suit me just fine.

Only as long as message/rfc822 doesn't carry news messages. However, it
currently does.

> HOWEVER, the 4th paragraph of section 6.4 of RFC-2045 then goes and
> undoes the good work by forbidding other than "7bit", "8bit" or "binary"
> with any composite media type.

Correct. And as I said before, this was a hotly contested point for which
there was strong consensus in the MIME WG. It is going to be nigh on to
impossible to undo it now.

> This is most unfortunate, since it forbids for all time any message-type
> for any kind of document which allows 8bit characters in its headers.
> This is not a problem (yet) for rfc822-style email, but it will be a
> problem for news articles constructed according to the new draft now in
> preparation, and it will be a problem for any not-yet-thought-of kind of
> internet document that might be proposed in the future.

It imposes no such restriction. The restriction is simply that CTEs can only
be applied to leaves in the MIME structure. There is nothing that says we
cannot define a composite type with 8bit or even binary headers in the future.

> So that is the reason why we have proposed to obsolete message/news.

Again, I have no problem with obsoleting message/news, and to be honest
I don't really care that the grounds you claim for doing so aren't correct.

> > Well, I certainly agree with the outcome -- message/news should never
> > have been registered. The intent was always for message/rfc822 to
> > cover News, and it does so just fine. Creating additional composite
> > types is a serious business owing to the need to add support in a
> > large number of contexts. It should never have been allowed, and would
> > not be today, but back in those days we didn't have an appropriately
> > tuned type registration mechanism in place.

> No, it is too much to expect message/rfc822 to apply to news, for that
> will constrain the news standards and the mail standards and the mime
> standards always to keep in exact step, and the whole IETF mechanisms
> are too complex to ever expect to achieve consensus across such a
> wide front at any one time. Currently, the Usenet-format group, which
> started its work a couple of years later than DRUMS, and which is in the
> fortunate position of being able to take advantage of a software base
> for news transport which is already 8bit clean, has been able to take
> the plunge and build 8bit in as a mandatory requirement for transports.
> Specifically, it is proposing to a allow UTF-8 in headers.

This would be a valid point if message/rfc822 were strictly aligned with
RFC822. But it isn't -- the name, which was chosen and implemented long
before the exact rules for the type were put in place, is misleading.

Message/rfc822 was specifically designed to allow for material that isn't
legal according to RFC822 (and, I suspect, DRUMS). Specifically, the
requirements for what headers have to be present are substantially relaxed,
as are the syntax rules.

Now, if news goes off and designs something that has different overall header
syntax that isn't line-oriented or something similar, then I can see a case for
a new type. But until and unless something like that happens I don't think
there's adequate justification for adding a new composite type. 8bit in headers
certainly doesn't come close to making it a requirement.

> > But the reasons given here are specious. For one thing, the use of UTF-8 in
> > headers is an orthogonal issue to this one. We've had a strong consensus for
> > years that this is a place we want to get to eventually, but we have to finish
> > a bunch of other stuff, most notably DRUMS, first. Only then will we have the
> > basis on which a design team can write a specification for UTF-8 headers that
> > actually works.

> At which point you will undoubtedly run into this exact same problem.

Again, the problem you're seeing just isn't there. This isn't to say there
aren't problems -- major ones -- ahead, but this isn't one of them.

> > And as for the CTE issue, it was decided long ago that CTEs cannot be
> > nested. And this is why the restrictions on message exist, and will
> > apply equally to any attempt to register any other composite type,
> > regardless of what top-level type you put it under. And this is why
> > CTEs on inner parts can and will be changed by software as messages,
> > be they from News or whereever, move from News to mail or vica versa.

> > Millions of lines of code have been written in good faith based on
> > this choice. We don't go against choices we made in the past that have
> > led to widespread good faith implementations. As such, any document
> > that says I cannot do what I've previously been specifically told I
> > have to do to messages because they happened to originate in News,
> > doesn't stand much of a chance of being standardized.

> > > How does this conclusion influence e-mail standards?

> > It isn't a conclusion, it is simply a paragraph in an Internet Draft. And one
> > which will receive strong objection if by some chance it makes it to IETF last
> > call.

> I don't see how you can object to the removal of a feature which you
> just said should not be there in the first place.

Please reread my response. I said nothing of the sort. My objection was to your
attempts to violate the no nested encodings rule.

> What you may not
> like is the thing we propose to put in its place, which is two new
> application types:

> 	application/news-tramsmission
> 	application/news-message

> These are quite legal according to RFC-2045. The first one is no problem
> (indeed it is already registered with IANA) but the second is an ugly
> compromise which is bound to lead to less-than-optimal behaviour on
> existing news and mail readers (until they are upgraded to recognise it
> as requiring disposition-display). It would have been much easier to
> have "message" recategorized as a non-composite type, but that is not
> our call :-(.

Well, you're welcome to try and move forward in this way, but I predict major
trouble when you attempt to register a composite type that allows encoding.

One final note. As those of you who were present at the time may recall, I've
always been ambivalent about nested encodings. I never objected to them nor
insisted on them. But at the time I was the one lone abstain in a large room
full of strong objections.

				Ned

Date: Mon, 4 Jan 1999 11:54:35 GMT
From: Charles Lindsey <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Restrictions on the Content-Type "message"
Cc: [email-masked]
Content-MD5: ogNTCnir5ppwb8fVTrtVfg==
Sender: [email-masked]

	On Sun, 3 Jan 1999 13:04:41 +0100
	Jacob Palme <[email-masked]> said...

> 
> Below is a quote from DRAFT Mime headers 0.4
> by Charles Lindsey <[email-masked]>, which
> is being developed by the USEFOR IETF group:
> 
>     6.x.5.3 Message/news obsoleted
> 
>         The Content-Type "message/news", as previously registered with
>         IANA, is hereby obsoleted and should be withdrawn. The reason
>         is that, according to RFC-2045, it is expressly forbidden for
>         any "message" type to use a transfer-encoding other than
>         "7bit", "8bit" or "binary", thus preventing a news article
>         which makes full use of UTF-8 characters in its headers, or
>         which uses (as it SHOULD) "8bit" as its own transfer-encoding,
>         from being passed through a 7bit transport medium (recall that
>         it is not permitted by this document for the transfer-encoding
>         of a news article to be changed).
> 
> How does this conclusion influence e-mail standards?
> ------------------------------------------------------------------------
> Jacob Palme <[email-masked]> (Stockholm University and KTH)
> for more info see URL: http://www.dsv.su.se/~jpalme
> 

I don't think it affects e-mail standards at all, since message/rfc822
does not have this problem (at least, not with e-mail as currently
specified by DRUMS - they are going to run into this problem if/when
they decide to allow UTF-8 in headers at some future date).

IMHO RFC-2045 is too restrictive. It should have allowed each
message/type to decide this issue for itself. In fact, RFC-2045
contradicts itself for it says, in the first paragraph of 6.4:

"... If an entity is of type "multipart" the Content-Transfer-Encoding
is not permitted to have any value other than "7bit", "8bit" or
"binary". Even more severe restrictions apply to some subtypes of the
"message" type."

That seems to suggest that the restriction applies to all multipart
types (which is fine by me) but not to message types unless some
particular message-type so specifies (as message/rfc822 indeed does).
This view is confirmed by the 7th paragraph of 5.1 of RFC-2046 (as
regards multipart-types) and the 3rd paragraph of section 5.2 of
RFC-2046 which states:

"Subtypes of "message" often impose restrictions on what encodings are
allowed. These restrictions are described in conjunction with each
specific subtype."

and the followinng section of RFC-2046 then goes on to specifiy exactly
such a restriction in the case of message/rfc822. That interpretation
would suit me just fine.

HOWEVER, the 4th paragraph of section 6.4 of RFC-2045 then goes and
undoes the good work by forbidding other than "7bit", "8bit" or "binary"
with any composite media type.

This is most unfortunate, since it forbids for all time any message-type
for any kind of document which allows 8bit characters in its headers.
This is not a problem (yet) for rfc822-style email, but it will be a
problem for news articles constructed according to the new draft now in
preparation, and it will be a problem for any not-yet-thought-of kind of
internet document that might be proposed in the future.

So that is the reason why we have proposed to obsolete message/news.

	On Sun, 03 Jan 1999 11:18:35 -0800 (PST)
	Ned Freed <[email-masked]> said...

> 
> > Below is a quote from DRAFT Mime headers 0.4
> > by Charles Lindsey <[email-masked]>, which
> > is being developed by the USEFOR IETF group:
> 
> >     6.x.5.3 Message/news obsoleted
> 
[snipped]
> 
> Well, I certainly agree with the outcome -- message/news should never
> have been registered. The intent was always for message/rfc822 to
> cover News, and it does so just fine. Creating additional composite
> types is a serious business owing to the need to add support in a
> large number of contexts. It should never have been allowed, and would
> not be today, but back in those days we didn't have an appropriately
> tuned type registration mechanism in place.

No, it is too much to expect message/rfc822 to apply to news, for that
will constrain the news standards and the mail standards and the mime
standards always to keep in exact step, and the whole IETF mechanisms
are too complex to ever expect to achieve consensus across such a
wide front at any one time. Currently, the Usenet-format group, which
started its work a couple of years later than DRUMS, and which is in the
fortunate position of being able to take advantage of a software base
for news transport which is already 8bit clean, has been able to take
the plunge and build 8bit in as a mandatory requirement for transports.
Specifically, it is proposing to a allow UTF-8 in headers.

UTF-8 is currently a favoured option for future developments within
IETF, and it looks like news is going to be the first major application
to take it on board. This does, however, mean that we get to bear the
brunt of the teething problems :-(.

> 
> But the reasons given here are specious. For one thing, the use of UTF-8 in
> headers is an orthogonal issue to this one. We've had a strong consensus for
> years that this is a place we want to get to eventually, but we have to finish
> a bunch of other stuff, most notably DRUMS, first. Only then will we have the
> basis on which a design team can write a specification for UTF-8 headers that
> actually works.

At which point you will undoubtedly run into this exact same problem.
> 
> And as for the CTE issue, it was decided long ago that CTEs cannot be
> nested. And this is why the restrictions on message exist, and will
> apply equally to any attempt to register any other composite type,
> regardless of what top-level type you put it under. And this is why
> CTEs on inner parts can and will be changed by software as messages,
> be they from News or whereever, move from News to mail or vica versa.
> 
> Millions of lines of code have been written in good faith based on
> this choice. We don't go against choices we made in the past that have
> led to widespread good faith implementations. As such, any document
> that says I cannot do what I've previously been specifically told I
> have to do to messages because they happened to originate in News,
> doesn't stand much of a chance of being standardized.
> 
> > How does this conclusion influence e-mail standards?
> 
> It isn't a conclusion, it is simply a paragraph in an Internet Draft. And one
> which will receive strong objection if by some chance it makes it to IETF last
> call.

I don't see how you can object to the removal of a feature which you
just said should not be there in the first place. What you may not
like is the thing we propose to put in its place, which is two new
application types:

	application/news-tramsmission
	application/news-message

These are quite legal according to RFC-2045. The first one is no problem
(indeed it is already registered with IANA) but the second is an ugly
compromise which is bound to lead to less-than-optimal behaviour on
existing news and mail readers (until they are upgraded to recognise it
as requiring disposition-display). It would have been much easier to
have "message" recategorized as a non-composite type, but that is not
our call :-(.
> 
> 					Ned
> 
> 

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Email:     [email-masked]  Web:   http://www.cs.man.ac.uk/~chl
Voice/Fax: +44 161 437 4506      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9     Fingerprint: 73 6D C2 51 93 A0 01 E7  65 E8 64 7E 14 A4 AB A5

Date: Sun, 03 Jan 1999 11:18:35 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Restrictions on the Content-Type "message"
To: Jacob Palme <[email-masked]>
Cc: [email-masked], Charles Lindsey <[email-masked]>
Sender: [email-masked]

> Below is a quote from DRAFT Mime headers 0.4
> by Charles Lindsey <[email-masked]>, which
> is being developed by the USEFOR IETF group:

>     6.x.5.3 Message/news obsoleted

>         The Content-Type "message/news", as previously registered with
>         IANA, is hereby obsoleted and should be withdrawn. The reason
>         is that, according to RFC-2045, it is expressly forbidden for
>         any "message" type to use a transfer-encoding other than
>         "7bit", "8bit" or "binary", thus preventing a news article
>         which makes full use of UTF-8 characters in its headers, or
>         which uses (as it SHOULD) "8bit" as its own transfer-encoding,
>         from being passed through a 7bit transport medium (recall that
>         it is not permitted by this document for the transfer-encoding
>         of a news article to be changed).

Well, I certainly agree with the outcome -- message/news should never have been
registered. The intent was always for message/rfc822 to cover News, and it does
so just fine. Creating additional composite types is a serious business owing
to the need to add support in a large number of contexts. It should never have
been allowed, and would not be today, but back in those days we didn't have an
appropriately tuned type registration mechanism in place.

But the reasons given here are specious. For one thing, the use of UTF-8 in
headers is an orthogonal issue to this one. We've had a strong consensus for
years that this is a place we want to get to eventually, but we have to finish
a bunch of other stuff, most notably DRUMS, first. Only then will we have the
basis on which a design team can write a specification for UTF-8 headers that
actually works.

And as for the CTE issue, it was decided long ago that CTEs cannot be nested.
And this is why the restrictions on message exist, and will apply equally to
any attempt to register any other composite type, regardless of what top-level
type you put it under. And this is why CTEs  on inner parts can and will be
changed by software as messages, be they from News or whereever, move from News
to mail or vica versa.

Millions of lines of code have been written in good faith based on this choice.
We don't go against choices we made in the past that have led to widespread
good faith implementations. As such, any document that says I cannot do what
I've previously been specifically told I have to do to messages because they
happened to originate in News, doesn't stand much of a chance of being
standardized.

> How does this conclusion influence e-mail standards?

It isn't a conclusion, it is simply a paragraph in an Internet Draft. And one
which will receive strong objection if by some chance it makes it to IETF last
call.

					Ned

Date: Sun, 3 Jan 1999 13:04:41 +0100
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Restrictions on the Content-Type "message"
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

Below is a quote from DRAFT Mime headers 0.4
by Charles Lindsey <[email-masked]>, which
is being developed by the USEFOR IETF group:

    6.x.5.3 Message/news obsoleted

        The Content-Type "message/news", as previously registered with
        IANA, is hereby obsoleted and should be withdrawn. The reason
        is that, according to RFC-2045, it is expressly forbidden for
        any "message" type to use a transfer-encoding other than
        "7bit", "8bit" or "binary", thus preventing a news article
        which makes full use of UTF-8 characters in its headers, or
        which uses (as it SHOULD) "8bit" as its own transfer-encoding,
        from being passed through a 7bit transport medium (recall that
        it is not permitted by this document for the transfer-encoding
        of a news article to be changed).

How does this conclusion influence e-mail standards?
------------------------------------------------------------------------
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/~jpalme