Subject: Re: Possible error in RFC 2822
From: "Christopher Yorke" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Date: Wed, 19 Mar 2003 08:14:07 -0400
List-Id: <ietf-822.imc.org>

Bruce Lilly wrote:
> Christopher Yorke wrote:
> > The confusion on my
> > part arises when I consult the obs-mbox-list production:
> >
> > obs-mbox-list = 1*([mailbox] [CFWS] "," [CFWS]) [mailbox]
> >
> > Again, if my reading of RFC 2234 is correct, this means that
obs-mbox-list
> > must consist of at least one comma, but that's it. It doesn't seem to
> > require any addresses at all.
>
> The obs- syntax in 2822 is intended to allow parsing of old messages,
> while the regular (non-obs-) syntax is for generation of new messages.

Yes, I understood that. However, I failed to consider RFC 724 in my
definition of old messages, only RFC 822.

> If you go back to RFC 724, you'll see that a From header field body
> could in fact be completely empty:
>
>           <any-from-field>  ::=   "From"        ":" <address-list>
> ...
>           <address-list>    ::=   <null>
>                                 | <address-item>
>                                 | <address-item> "," <address-list>
>
> In order to be able to handle old messages, a parser should be able
> to handle lists with empty elements.  The degenerate case, as you
> point out, has no mailboxes at all.  A robust parser would not fail
> even on an empty field body, such as was legal under RFC 724 (i.e. not
> even a comma).

If my understanding of this production is correct, the "From" field could
contain no addresses, one address, one address followed by a comma, address
comma address, and so on. At the very least, obs-mbox-list doesn't seem to
allow an empty field body since everything is optional *except* for the
comma.

> Of course, generating an RFC 2822-compliant From header field requires
> at least one mailbox, and no null list elements (including at the
> start or end of the list).

At least I understand this.

Date: Wed, 19 Mar 2003 01:28:24 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Christopher Yorke <[email-masked]>
CC: [email-masked]
Subject: Re: Possible error in RFC 2822
Sender: [email-masked]

Christopher Yorke wrote:
> I have a question about a couple of the productions in section 4.4 of RFC
> 2822 entitled "Obsolete Addressing." Based on my reading of RFC (2)822,
> every message is required to have a "From:" address.
[...]
> The confusion on my
> part arises when I consult the obs-mbox-list production:
> 
> obs-mbox-list = 1*([mailbox] [CFWS] "," [CFWS]) [mailbox]
> 
> Again, if my reading of RFC 2234 is correct, this means that obs-mbox-list
> must consist of at least one comma, but that's it. It doesn't seem to
> require any addresses at all.

The obs- syntax in 2822 is intended to allow parsing of old messages,
while the regular (non-obs-) syntax is for generation of new messages.
If you go back to RFC 724, you'll see that a From header field body
could in fact be completely empty:

          <any-from-field>  ::=   "From"        ":" <address-list>
...
          <address-list>    ::=   <null>
                                | <address-item>
                                | <address-item> "," <address-list>

In order to be able to handle old messages, a parser should be able
to handle lists with empty elements.  The degenerate case, as you
point out, has no mailboxes at all.  A robust parser would not fail
even on an empty field body, such as was legal under RFC 724 (i.e. not
even a comma).

Of course, generating an RFC 2822-compliant From header field requires
at least one mailbox, and no null list elements (including at the
start or end of the list).

From: "Christopher Yorke" <[email-masked]>
To: <[email-masked]>
Subject: Possible error in RFC 2822
Date: Tue, 18 Mar 2003 18:45:36 -0400
Sender: [email-masked]

I have a question about a couple of the productions in section 4.4 of RFC
2822 entitled "Obsolete Addressing." Based on my reading of RFC (2)822,
every message is required to have a "From:" address. The relevant production
from RFC 822 is:

authentic = "From" ":" mailbox      ; Single author
          / ( "Sender" ":" mailbox  ; Actual submittor
              "From" ":" 1#mailbox) ; Multiple authors
                                    ; or not sender

This production, along with section 2.7 of RFC 822, indicate that this means
"From:" contains one or more mailboxes, separated by commas; however, RFC
2822 seems to indicate otherwise. The relevant productions from RFC 2822
are:

from = "From:" mailbox-list CRLF
mailbox-list = (mailbox *("," mailbox)) / obs-mbox-list

The first option for the mailbox-list production agrees with the RFC 822
production but for the fact that it doesn't allow for consecutive ','s (i.e.
null elements, as described in section 2.7 of RFC 822). The confusion on my
part arises when I consult the obs-mbox-list production:

obs-mbox-list = 1*([mailbox] [CFWS] "," [CFWS]) [mailbox]

Again, if my reading of RFC 2234 is correct, this means that obs-mbox-list
must consist of at least one comma, but that's it. It doesn't seem to
require any addresses at all. If it is supposed to match the syntax of RFC
822, I would think it should be something closer to:

obs-mbox-list = mailbox *([CFWS] "," [CFWS] [mailbox])

I often have troubles divining what RFC 822 intends so I'm not sure if the
<l>#<m>element syntax allows the list to end (or start) with a comma, but I
still don't see how the obs-mbox-list production from RFC 2822 is correct.
The same problem exists in the obs-addr-list production.

I hope that I have made myself understood and that I haven't missed
something obvious. If so, I apologize for wasting everyone's time.

To: [email-masked]
Xref: clerew local.mime:2537
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Comments on draft-gellens-format-bis-00.txt
Date: Thu, 13 Mar 2003 10:31:39 GMT
Lines: 51
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>"Charles Lindsey" <[email-masked]> writes:

>> If the original message was not format flowed, then it Ought to 
>  have
>> started out with a line length of 72 or thereabouts to allow 
>  room for
>> quoting. So in a well-ordered thread the situation should not 
>  arise, and
>> if it does, then things are no worse than they ar at present 
>  with unflowed
>> messages.

>Your message illustrates quite well that this isn't always true 
>(your first line is 82 characters long, now it is 84...).

Yes, it is the way nn construct its attribution lines :-( .

However, the way your system quoted my original message raises some
further problems. My system does not understnad format=flowed (and this
message is not flowed, so you can deduce exactly how it was displayed to
me).  All the lines you quoted were then splt, by you, into two, which
produced even more bizarre effects when I quoted them back at you.

OTOH, even if my reader had understood flowed, I don't see how it could
have displayed those lines any better, or quoted them any better when
composing this message.

I think this is one aspect of the problem that you are trying to solve,
though I am not sure exactly how your proposal would fix it.

>This is what my implemention does too, using the filling functions 
>in emacs, which often makes good guesses.  But, and this triggered 
>my review of the implementation, it sometimes produces horrible 
>results.

Which is why maybe it needs to be under user control.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Cc: [email-masked]
To: "Charles Lindsey" <[email-masked]>
From: Simon Josefsson <[email-masked]>
Subject: Re: Comments on draft-gellens-format-bis-00.txt
Date: Wed, 12 Mar 2003 13:54:20 +0100
Sender: [email-masked]

"Charles Lindsey" <[email-masked]> writes:

> In <[email-masked]> Simon Josefsson 
  <[email-masked]> writes:
>
>>Section 5.1 "Generating Format=Flowed" currently says "lines
>>SHOULD be shorter than 80 characters"....
>
>
>> 2) Leave the original message as is, and potentially break the
>>    SHOULD about lines being shorter than 80 characters.
>
> If the original message was not format flowed, then it Ought to 
  have
> started out with a line length of 72 or thereabouts to allow 
  room for
> quoting. So in a well-ordered thread the situation should not 
  arise, and
> if it does, then things are no worse than they ar at present 
  with unflowed
> messages.

Your message illustrates quite well that this isn't always true 
(your first line is 82 characters long, now it is 84...).  The 
longest of the lines I quote is now 76 characters, which is only a 
few quote iterations away from breaking the limit.  The problem is 
that a format=flowed implementation has to break a SHOULD in order 
to quote this message.  The normative text in the specification 
should be more realistic, IMHO.

> But I agree that letting it expand beyond 80 characters is the 
  correct
> thing to do if it arises.

Good, that's what I'd like to see added.

> However, an alternative would be to reformat the quoted text (I 
  sometime
> do this using par, which has nice facilities for the 
  purpose). This should
> be under user control, though, as it could make a mess in 
  inappropriate
> cases. However, if it is done then the format could then be made 
  flowed.

This is what my implemention does too, using the filling functions 
in emacs, which often makes good guesses.  But, and this triggered 
my review of the implementation, it sometimes produces horrible 
results.

> So if the draft is to mention this at all, maybe that 
  alternative could be
> mentioned also.

I agree.

To: [email-masked]
Xref: clerew local.mime:2535
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Comments on draft-gellens-format-bis-00.txt
Date: Wed, 12 Mar 2003 11:27:19 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>Section 5.1 "Generating Format=Flowed" currently says "lines 
>SHOULD be shorter than 80 characters"....

> 2) Leave the original message as is, and potentially break the 
>    SHOULD about lines being shorter than 80 characters.

If the original message was not format flowed, then it Ought to have
started out with a line length of 72 or thereabouts to allow room for
quoting. So in a well-ordered thread the situation should not arise, and
if it does, then things are no worse than they ar at present with unflowed
messages.

But I agree that letting it expand beyond 80 characters is the correct
thing to do if it arises.

However, an alternative would be to reformat the quoted text (I sometime
do this using par, which has nice facilities for the purpose). This should
be under user control, though, as it could make a mess in inappropriate
cases. However, if it is done then the format could then be made flowed.

So if the draft is to mention this at all, maybe that alternative could be
mentioned also.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Subject: Comments on draft-gellens-format-bis-00.txt
From: Simon Josefsson <[email-masked]>
Date: Tue, 11 Mar 2003 21:21:00 +0100
Sender: [email-masked]

I'm revising my rfc2646 implementation based on this draft, and 
I'd like to see some things clarified in the document.

It is currently problematic to generate a conforming format=flowed 
message when replying to (i.e., quoting) a non-flowed message.  Is 
this scenario supported at all?  I'm actually not sure whether 
2646 allows this or not.  If it isn't allowed, making that 
explicit would be good.  On the other hand, I think it is possible 
to generate useful format=flowed messages when replying to 
non-flowed messages if some changes are made to the specification.

Section 5.1 "Generating Format=Flowed" currently says "lines 
SHOULD be shorter than 80 characters".  It discusses how to 
achieve this by wrapping, and references to section 5.5 "Quoting" 
on how to handle quoted text.  However, section 5.5 is only 
applicaple if the message that is being replied to is 
format=flowed -- the entire section starts with "In Format=Flowed, 
the canonical quote indicator (or quote mark) is one or more close 
angle bracket (">") characters ..." and this assumption is used 
throughout the section.  The assumption does not hold for 
non-format=flowed text, so a implementation has two choices:

 1) Apply the quoting algorithm to fixed messages too.  This works 
    for well formed messages, but can generate horible output.

 2) Leave the original message as is, and potentially break the 
    SHOULD about lines being shorter than 80 characters.

I think it would be useful to add a new section "Generating 
Format=Flowed Quoting Format=Fixed Messages" that blesses the 
second choice.  It should say that if the incoming message is not 
format=flowed, the client MUST NOT make any attempt at filling the 
line to fit under 80 characters.  Some examples to illustrate that 
this cannot be done reliably (which I belive is the case) would be 
useful.  The section should go on to say that performing wrapping 
of any lines produced by the local, format=flowed aware, client 
SHOULD be done.

What do you think?  I have not finished implemented this, so there 
may be some hidden things that invalidates my idea.

To: [email-masked]
Xref: clerew local.mime:2533
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Problems with RFC 2047
Date: Mon, 10 Mar 2003 15:37:56 GMT
Lines: 46
Sender: [email-masked]

In <[email-masked]> "Dan Kohn" <[email-masked]> writes:

>"Mail User Agents SHOULD unfold headers that are displayed to the
end-user (such as Subject, To, and Cc), since line breaks in headers do
not have semantic content (unlike in message bodies, where they often
denote paragraph or list breaks)."

Somebody has not been wrapping his lines at sensible places :-(.

Anyway, I disagree. Users who write long headers may carefully fold them
to give best legibility. User agents SHOULD NOT throw away that
information wantonly (OK, if the lines are longer than 78/whatever then
refolding might be in order).

Here is an example header produced by Spamassassin:

X-Spam-Report: Detailed Report
 SPAM: -------------------- Start SpamAssassin results ----------------------
 SPAM: This mail is probably spam.  The original message has been altered
 SPAM: so you can recognise or block similar unwanted mail in future.
 SPAM: See http://spamassassin.org/tag/ for more details.
 SPAM: 
 SPAM: Content analysis details:   (8.00 hits, 5 required)
 SPAM: SUBJ_HAS_SPACES    (2.6 points)  Subject contains lots of white space
 SPAM: NO_REAL_NAME       (1.3 points)  From: does not include a real name
 SPAM: FORGED_RCVD_FOUND  (0.8 points)  Possibly-forged 'Received:' header found
 SPAM: SPAM_PHRASE_05_08  (1.6 points)  BODY: Spam phrases score is 05 to 08 (medium)
 SPAM:                    [score: 5]
 SPAM: SUBJ_HAS_UNIQ_ID   (1.1 points)  Subject contains a unique ID
 SPAM: MSG_ID_ADDED_BY_MTA_2 (0.1 points)  'Message-Id' was added by a relay (2)
 SPAM: MISSING_MIMEOLE    (0.5 points)  Message has X-MSMail-Priority, but no X-MimeOLE
 SPAM: 
 SPAM: -------------------- End of SpamAssassin results ---

Try refolding that, and it becomes unreadable.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

content-class: urn:content-classes:message
Subject: RE: Problems with RFC 2047
Importance: normal
Date: Sun, 9 Mar 2003 13:01:20 -0800
From: "Dan Kohn" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "Jacob Palme" <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

Jacob Palme <[email-masked]> wrote:

> Subject: Tid fär
>  nästa
>  möte med CMC-forskargruppen

> The cause of the problems seems to be that RFC2047 makes
> the subject much longer,

I believe the issue is not a RFC 2047 one, but an issue of semantic confusion about line breaks in Subject headers.

Specifically, I would suggest to Jacob that he write a few paragraph long informational RFC of MUA implementation advice with the following paragraph:

"Mail User Agents SHOULD unfold headers that are displayed to the end-user (such as Subject, To, and Cc), since line breaks in headers do not have semantic content (unlike in message bodies, where they often denote paragraph or list breaks)."

See these messages:

http://www.imc.org/ietf-822/mail-archive/msg02572.html
http://www.imc.org/ietf-822/mail-archive/msg02573.html
http://www.imc.org/ietf-822/mail-archive/msg02574.html

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Sun, 09 Mar 2003 15:57:59 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Jacob Palme <[email-masked]>
CC: [email-masked]
Subject: Re: Problems with RFC 2047
Sender: [email-masked]

Jacob Palme wrote:
> A message I sent recently contained the following subject:
> 
> Subject: Tid fär nästa möte med CMC-forskargruppen
> 
> It was sent as follows:
> 
> Subject: Tid =?iso-8859-1?Q?f=E4r?= =?iso-8859-1?Q?_?=
>  =?iso-8859-1?Q?n=E4sta?= =?iso-8859-1?Q?_?=
>  =?iso-8859-1?Q?m=F6te?= med CMC-forskargruppen

Subject: Tid =?ISO-8859-1?q?f=E4r_n=E4sta_m=F6te?= med CMC-forskargruppen

would be simpler. Whatever encoded your text did a pretty bad job
(it's legal, but awful).

> And when I received the same message, the subject looked 
> like follows:
> 
> Subject: Tid fär 
>  nästa 
>  möte med CMC-forskargruppen

That looks like a decoding problem; line folding and whitespace
between encoded-words is supposed to be ignored for display, and the
line breaks are only between encoded-words.

> The cause of the problems seems to be that RFC2047 makes 
> the subject much longer,

If done well, it's slightly longer (in characters) than the number
of octets in the original.  Best case depends on how long the
charset tag is (and language tag, if used (see RFC 2231)).

 > and the sending mailer is of the
> opinion that no header line should be longer than 78 
> characters,

Actually, any header line containing any encoded-word is required
to be no longer than 76 octets (not counting the CRLF [RFC 2047
section 2]). The more compact version above is just shy of that
limit.  It's still possible in this case to fit everything on one
line, even with a language tag:

Subject: =?ISO-8859-1*se?q?Tid_f=E4r_n=E4sta_m=F6te_med_CMC-forskargruppen?=

which is exactly 76 octets.

 > and thus splits the subject on multiple lines.
> And the sending mailer had to encode each line with its
> own =?iso-8859-1?Q?, making the problem worse.

Neither is neccessary, as shown above.

> This, then, will be a problem with any encoding scheme 
> which increases the number of bytes in a header. And also 
> UTF-8 can increase the number of bytes, and thus cause the 
> same problem.

It should not be a problem in any case; any header field may be
continued, Subject included, and user agents need to be able
to deal with that appropriately (e.g. by unfolding for display).

> The problem would have been solved if RFC 2047 included a 
> facility to split a header into multiple lines, which are 
> not to be received as multiple lines, like the "=" at the 
> end of a line in Quoted-Printabe.

Actually that is already part of RFC 2047 (via a somewhat different
mechanism); see the examples and discussion in RFC 2047 section 8.

Date: Sun, 9 Mar 2003 14:23:03 -0500
From: Keith Moore <[email-masked]>
To: Jacob Palme <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Problems with RFC 2047
Sender: [email-masked]

On Sun, 9 Mar 2003 20:13:52 +0100
Jacob Palme <[email-masked]> wrote:

> 
> A message I sent recently contained the following subject:
> 
> Subject: Tid fär nästa möte med CMC-forskargruppen
> 
> It was sent as follows:
> 
> Subject: Tid =?iso-8859-1?Q?f=E4r?= =?iso-8859-1?Q?_?=
>  =?iso-8859-1?Q?n=E4sta?= =?iso-8859-1?Q?_?=
>  =?iso-8859-1?Q?m=F6te?= med CMC-forskargruppen
> 
> And when I received the same message, the subject looked 
> like follows:
> 
> Subject: Tid fär 
>  nästa 
>  möte med CMC-forskargruppen
> 
> The cause of the problems seems to be that RFC2047 makes 
> the subject much longer,

well, RFC 2047 states that any white space between adjacent encoded-words
is to be ignored when decoding.  so this is a bug in your mail reader.

I'll also note that it's encoding routines, while correct, seem to be
producing very sub-optimal output.

Subject: Tid =?iso-8859-1?Q?f=E4r_N=E4sta_m=F6te?= med CMC-forskargruppen

is a valid encoding of the same string.

Keith

Date: Sun, 9 Mar 2003 20:13:52 +0100
To: [email-masked]
From: Jacob Palme <[email-masked]>
Subject: Problems with RFC 2047
Sender: [email-masked]

A message I sent recently contained the following subject:

Subject: Tid fär nästa möte med CMC-forskargruppen

It was sent as follows:

Subject: Tid =?iso-8859-1?Q?f=E4r?= =?iso-8859-1?Q?_?=
 =?iso-8859-1?Q?n=E4sta?= =?iso-8859-1?Q?_?=
 =?iso-8859-1?Q?m=F6te?= med CMC-forskargruppen

And when I received the same message, the subject looked 
like follows:

Subject: Tid fär 
 nästa 
 möte med CMC-forskargruppen

The cause of the problems seems to be that RFC2047 makes 
the subject much longer, and the sending mailer is of the 
opinion that no header line should be longer than 78 
characters, and thus splits the subject on multiple lines.
And the sending mailer had to encode each line with its
own =?iso-8859-1?Q?, making the problem worse.

This, then, will be a problem with any encoding scheme 
which increases the number of bytes in a header. And also 
UTF-8 can increase the number of bytes, and thus cause the 
same problem.

The problem would have been solved if RFC 2047 included a 
facility to split a header into multiple lines, which are 
not to be received as multiple lines, like the "=" at the 
end of a line in Quoted-Printabe.
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/

content-class: urn:content-classes:message
Subject: draft-kohn-news-article-01a.txt
Importance: normal
Date: Sun, 9 Mar 2003 10:42:29 -0800
From: "Dan Kohn" <[email-masked]>
To: "Usefor Mailing List" <[email-masked]>
Cc: "IMAP Interest List" <[email-masked]>, "IETF RFC-822 list" <[email-masked]>
Sender: [email-masked]

A new version of draft-kohn-news-article is now available on my website
(as I missed the I-D cut-off).

http://www.dankohn.com/draft-kohn-news-article-01a.txt
http://www.dankohn.com/draft-kohn-news-article-01a.html
http://www.dankohn.com/draft-kohn-news-article-01a.xml

This attempts to more fully flesh out what Russ Allbery describes
<http://www.washington.edu/imap/listarch/current/msg00303.html> as
"Option C", a fully backward-compatible approach that uses RFC 2047 and
punycode to accomplish i18n.  Sections on cancel messages and
responsibilities for news agents are still missing, but could be fairly
easily added if usefor moves forward using this approach.

The Kohn draft can be compared to Charles Lindsey's "Option A" approach
of using raw UTF-8, as described at
<http://www.ietf.org/internet-drafts/draft-ietf-usefor-article-09.txt>.

Edits to the text are welcome either to the usefor list (where any
follow-up should occur) or to the author (me).

          - dan
--
Dan Kohn <mailto:[email-masked]>
<http://www.dankohn.com/>  <tel:+1-650-327-2600>

Date: Wed, 05 Mar 2003 09:15:17 -0600
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
CC: Bruce Lilly <[email-masked]>, IETF RFC-822 list <[email-masked]>
Subject: Re: gzip-8bit
Sender: [email-masked]

on 3/3/2003 1:06 PM [email-masked] wrote:

> And all this borders on the more general issue of how to represent
> complex package formats in MIME, especially those that store
> information about their content that MIME cannot. These are always
> judgement calls.

One thing that I seem to recall having been discussed at some point was to
send compression formats as multiparts. The first part would contain a
catalog of the contents (possibly with offsets or some other applicable
reference point), the second part would contain the compressed entity.
This kind of model would make it feasible (in theory anyway) to structure
the first part as references to subordinate remote entities.

The feasibility of this is of course dependant on several things, not the
least of which is the ability for the compression format to provide
meaningful pointers. Another issue is that you don't want clients thinking
that they can download just one of the subordinate entities, unless such a
feature is specifically supported (eg, having an IMAP server extract the
referenced entity and returning it alone, on demand).

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 4 Mar 2003 23:54:39 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Claus Färber <[email-masked]> wrote:

> Content-Type: application/message-partial; number=1; total=1;
>         id="[email-masked]"
> Content-Transfer-Encoding: 8bit
> 
> Content-Type: application/postscript
> Content-Disposition: attachment; filename="hello.ps"
> Content-Transfer-Encoding: quoted-binary; foo="bar"
> Content-Encoding: gzip (from HTTP)
> Content-Required-Fields: Type, Transfer-Encoding, Encoding (list of
>         fields which a EMIME parser must know to fully decode the
>         entity)
> 
> The advantage of this is: If it crosses the border to the non-8bit
> world, everything inside the enclosing entity can be encoded with qp
> or base64.

...by an agent that doesn't know about quoted-binary.  That's a nice
feature.  And you can encapsulate another application/message-partial if
you want to apply another Content-Encoding.  The drawback there is that
you won't be able to see the ultimate Content-Type without undoing the
encodings of all but the innermost encapsulation.

I don't know whether I prefer the encapsulation approach or the
meta approach, but we might as well see how good we can make them
both.  Here's another revision of the meta approach that provides the
recodability feature:

Content-Type: meta/encoded
Content-Decoded-Type: application/postscript
Content-Encodings:
  foo; param=value; param=value,
  bar; param=value; param=value,
  gzip
Content-Transfer-Encodings:
  blah; param=value,
  escaped-8bit
Content-Disposition: attachment; filename="hello.ps.foo.bar.gz"
Content-Transfer-Encoding: 8bit

The type meta/encoded means that the object type, object encodings, and
message encodings are all supplied by three new header fields.  The very
last message encoding goes in the Content-Transfer-Encoding field (and
is omitted from the Content-Transfer-EncodingS field) only if it is
one of the original well-known encodings (quoted-printable or base64),
otherwise the Content-Transfer-Encoding field is used only to describe
the body format (7bit, 8bit, or binary).

AMC

Date: 04 Mar 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Adam M. Costello <[email-masked]> schrieb/wrote:
> Content-Type: meta/encoded
> Content-Encodings:
>   foo; param=value; param=value,
>   bar; param=value; param=value,
>   gzip
> Content-Decoded-Type: application/postscript
> Content-Disposition: attachment; filename="hello.ps.foo.bar.gz"
> Content-Transfer-Encoding: escaped-8bit

Well, that would be quit similar to an encapsulation within a new
message-like body type:

Content-Type: application/message-partial; number=1; total=1;
        id="[email-masked]"
Content-Transfer-Encoding: 8bit

Content-Type: application/postscript
Content-Disposition: attachment; filename="hello.ps"
Content-Transfer-Encoding: quoted-binary; foo="bar"
Content-Encoding: gzip (from HTTP)
Content-Required-Fields: Type, Transfer-Encoding, Encoding (list of
        fields which a EMIME parser must know to fully decode the
        entity)

The advantage of this is: If it crosses the border to the non-8bit
world, everything inside the enclosing entity can be encoded with qp or
base64.

Claus
-- 
http://www.faerber.muc.de/

From: Marc Mutz <[email-masked]>
Organization: "Old Europe" - and proud
To: [email-masked]
Subject: Re: gzip-8bit
Date: Tue, 4 Mar 2003 11:51:48 +0000
Sender: [email-masked]

--Boundary-02=_eNJZ+qbzB/yEIKP
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Monday 03 March 2003 17:33, Bruce Lilly wrote:
> > The the UA should record the capabilities of the SMTP server for
> > that account or default to a 7bit domain CTE.
>
> There may be multiple MXs with differing capabilities.

UAs usually don't find their injection SMTP server by MX, do they? But I=20
get your point...

> Moreover, a UA on a portable device may have multiple
> connection options, e.g. an ISPs SMTP server used as a
> relay when on the road, and an office SMTP relay when
> at home base; the capabilities may differ, and it may
> not be known which will be used to send if the message
> is being composed when truly off-line.

The recorded capabilities are used as an educated guess. If the actual=20
server caps are lacking, the client needs to re-code down to something=20
the server can manage (or be conservative and only use 7bit-domain=20
CTEs).

Marc

=2D-=20
It's good fortune for the government that the masses don't think.
                                                         -- Adolf Hitler

--Boundary-02=_eNJZ+qbzB/yEIKP
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_eNJZ+qbzB/yEIKP--

Date: Tue, 4 Mar 2003 01:58:58 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> It's actually worse than that; there are other transformations
> unrelated to transport transparency that one might perform on content.
>
> List syntax might be a cleaner fit to existing paradigms

Good points.

> An approach along the lines of RFC 2633 could be used...
> Yet another approach would be...via a multipart encapsulation.

Correct me if I'm wrong, but I think these approaches would obscure the
actual media type.  For example, given a Postscript attachment that is
transformed somehow (signed, say) and then compressed, the receiver
cannot know that it's Postscript without first decompressing it.

Here's a revision of my earlier proposal that tries to address your
concerns above.

Content-Type: meta/encoded
Content-Encodings:
  foo; param=value; param=value,
  bar; param=value; param=value,
  gzip
Content-Decoded-Type: application/postscript
Content-Disposition: attachment; filename="hello.ps.foo.bar.gz"
Content-Transfer-Encoding: escaped-8bit

The new top-level media type "meta" is for messages whose type
information is to be found in new Content-* fields.  The type
meta/encoded means that the actual type information is in the fields
Content-Encodings and Content-Decoded-Type.  A receiving MUA unaware of
meta/encoded would treat it as application/octet-stream, which is safe.

AMC

Date: 04 Mar 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: gzip-8bit
Organization: Bruce Lilly
Sender: [email-masked]

Bruce Lilly <[email-masked]> schrieb/wrote:
>> Content-Transfer-Encoding: gzip|escaped-8bit

> List syntax might be a cleaner fit to existing paradigms, and
> of course that would in any event be a different type of
> Content-Transfer-Encoding field, which in turn would
> necessitate a newer MIME-Version.

No, not necessarily. As long as the separators fit RFC 2045's definition  
ot a ``token'', the whole list is the name of a single unknown CTE from  
the perspective of old software.

For example, the example above is compatible with the syntax from RFC  
2045 and old parsers should read a CTE named ``gzip|escaped-8bit''

Claus
-- 
http://www.faerber.muc.de/

Date: Mon, 03 Mar 2003 11:06:01 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: Bruce Lilly <[email-masked]>
Cc: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

> Well, there's already application/vnd.ms-tnef and
> application/mac-binhex40; of course that hides the media
> type of what has been compressed.

Application.mac-binhex40 barely squeaked by because it is more than a
encoding/checksum/compression mechanism; it includes facilities for
representing the resource fork of Macintosh files in addition to the data fork.
The same applies to application/applesingle. But the preferred format for Mac
appication data is multipart/appledouble.

As for application/vnd.ms-tnef, while it is true it can be operated in a mode
where it incorporates both the content of the message as well as markup of that
content, it can also be operated in a mode where the content is external. This
sort of puts it in the applesingle/appledouble camp. Regardless, these issues
are dwarfed by the proprietary nature of this format.

And all this borders on the more general issue of how to represent complex
package formats in MIME, especially those that store information about their
content that MIME cannot. These are always judgement calls.

				Ned

Date: Mon, 03 Mar 2003 13:42:36 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: IETF RFC-822 list <[email-masked]>
Organization: Bruce Lilly
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip-8bit
Sender: [email-masked]

Adam M. Costello wrote:

> And it's a bit gross to have to define N*M transfer
> encodings to support N escape mechanisms (quoted-printable, base64,
> escaped-8bit) and M compression schemes (gzip, deflate, bzip2, ...).

It's actually worse than that; there are other transformations
unrelated to transport transparency that one might perform on
content. E.g. encryption (more about that below).  If all such
transformations *must* be bundled into CTE for the sake of old
clients, the combinatorial explosion will be tremendous. Something
will have to give; either there will be a need for a large number
of CTEs in contradiction to RFC 2045, or the mechanism will have
to be adapted, or the current plethora of ad-hoc approaches will
worsen.

> Here's a way to separate the compression from the escaping in a
> backward-compatible way.  I don't claim that this is the best way, but
> it's something to consider.
> 
> Content-Type: encoded/gzip
> Content-Decoded-Type: application/postscript
> Content-Disposition: attachment; filename="foo.ps.gz"
> Content-Transfer-Encoding: escaped-8bit

Well, there's already application/vnd.ms-tnef and
application/mac-binhex40; of course that hides the media
type of what has been compressed. An approach along the
lines of RFC 2633 could be used, with suitable definitions
for application/gzip, application/deflate, etc.

> Another approach would be to introduce structure into the
> transfer-encoding, and recommend that transport agents ignore all but
> the last component:
> 
> Content-Type: application/postscript
> Content-Disposition: attachment; filename="foo.ps"
> Content-Transfer-Encoding: gzip|escaped-8bit

List syntax might be a cleaner fit to existing paradigms, and
of course that would in any event be a different type of
Content-Transfer-Encoding field, which in turn would
necessitate a newer MIME-Version.

Yet another approach would be something similar to the way
signing and encrypting transformations are applied per RFC
1847, viz. via a multipart encapsulation. So, following the
RFC 1847 model, with the appropriate media type definitions
one could have:

Content-Type: multipart/compressed ; method=application/gzip ; boundary=foo
Content-Transfer-Encoding: escaped-8bit

--foo
Content-Type: application/gzip
Content-Disposition: attachment ; filename="foo.ps.gz"

Control information for application/gzip
--foo
Content-Type: application/octet-stream

     [compressed]Content-Type: application/postscript
     Content-Disposition: attachment ; filename="foo.ps"

     %!blah[end of compressed data]
--foo--

That would allow the recipient's UA to present a choice of
saving in compressed or uncompressed format (if the
compression method is recognized!).  It would also allow
flexibility in composition, e.g.
compress | encrypt | sign
or
encrypt | sign | compress
etc.  I'd expect old client software to work pretty much
as it works with RFC 1847 signed/encrypted multiparts, i.e.
since the multipart/compressed isn't recognized, it's treated
as multipart/mixed and the unrecognized application/gzip
would be treated as application/octet-stream.  If there
will never be a need for control information for any
compression method, that could be simplified to a 1-part
"multipart" wrapper around the application/compression-method
type.

Date: Mon, 03 Mar 2003 12:33:30 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: Marc Mutz <[email-masked]>
CC: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

Marc Mutz wrote:
> On Friday 28 February 2003 20:03, Bruce Lilly wrote:

>>With a larger choice for binary attachment encodings, things
>>get a bit more complicated.  The UA can't necessarily
>>determine whether the MTA will support 8BITMIME (the user
>>may be off-line).
> 
> 
> The the UA should record the capabilities of the SMTP server for that 
> account or default to a 7bit domain CTE.

There may be multiple MXs with differing capabilities.
Moreover, a UA on a portable device may have multiple
connection options, e.g. an ISPs SMTP server used as a
relay when on the road, and an office SMTP relay when
at home base; the capabilities may differ, and it may
not be known which will be used to send if the message
is being composed when truly off-line.

Date: Mon, 03 Mar 2003 12:12:37 -0500
From: Bruce Lilly <[email-masked]>
Reply-To: Usefor Mailing List <[email-masked]>, IMAP protocol mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Organization: Bruce Lilly
To: Arnt Gulbrandsen <[email-masked]>
CC: Usefor Mailing List <[email-masked]>, IMAP protocol mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Arnt Gulbrandsen wrote:

> Ten or eleven years ago I was told (on ietf-smtp?) that I had 
> misunderstood that stuff, and that it really had to do with RFC 
> handwaving, not accepting clearly broken input.
> 
> The explanation went something like this: "The RFCs sometimes are 
> readable instead of precise. Say an RFC specifies a maximum line length, 
> but it's not entirely clear whether CRLF is included in or added to the 
> specified length. In that case you should accept max-plus-CRLF and send 
> only max-including-CRLF."
> 
> I forget who that was and no longer have any mail older than 1994. Was 
> he right or wrong?

The original, with some discussion, is in RFC 791, section3.2.
It is referenced, among other places, in RFC 1958, section
3.9.  There's more to it than RFC handwaving.

From: Arnt Gulbrandsen <[email-masked]>
To: Dave Barr <[email-masked]>
Subject: Re: Unicode newsgroup name options
Cc: Usefor Mailing List <[email-masked]>, IMAP protocol mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Date: Mon, 3 Mar 2003 10:40:44 +0100
Sender: [email-masked]

Dave Barr writes:

> On Sat, 2003-03-01 at 17:13, Keith Moore wrote:
>>   But I don't
>>  see any obligation for MTAs or UAs to deal with whatever random 
>>  trash is thrown at them.
>
> "Be conservative in what you accept" and all that, right?

Ten or eleven years ago I was told (on ietf-smtp?) that I had 
misunderstood that stuff, and that it really had to do with RFC 
handwaving, not accepting clearly broken input.

The explanation went something like this: "The RFCs sometimes are 
readable instead of precise. Say an RFC specifies a maximum line 
length, but it's not entirely clear whether CRLF is included in or 
added to the specified length. In that case you should accept 
max-plus-CRLF and send only max-including-CRLF."

I forget who that was and no longer have any mail older than 1994. Was 
he right or wrong?

--Arnt

Subject: Re: Unicode newsgroup name options
From: Dave Barr <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Organization: 
Date: 02 Mar 2003 22:47:32 -0600
Sender: [email-masked]

On Sat, 2003-03-01 at 17:13, Keith Moore wrote:
>  But I don't
> see any obligation for MTAs or UAs to deal with whatever random trash is
> thrown at them.

"Be conservative in what you accept" and all that, right?

--Dave

Date: Sun, 02 Mar 2003 16:38:01 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: Tony Hansen <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

> I almost hate to ask, but could we do anything with:
> 	MIME-Version: 2.0
> ?

Seems unnecessary to me, and in any case many believe this number can
never be incremented.

				Ned

Date: Sun, 2 Mar 2003 19:00:33 -0500
From: Keith Moore <[email-masked]>
To: [email-masked] (Claus =?ISO-8859-1?Q?F?=ärber)
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> > what should happen is that the sender gets a message back that says:
> > Your message could not be delivered because it was illegally formatted.
> > You should replace your mail reader with one that works correctly.
> > See http://foo.bar/mail-readers.html for a list of broken mail readers.
> 
> Noone would want to configure his server that way when it blocks  
> legitimate mail. And users of broken MUAs would still complain that  
> ``only this server is so picky'', even if they agree that it's right.

yup.  and like you say, it's a drawback to the robustness principle -
it encourages brokenness.

Keith

Date: Sun, 02 Mar 2003 18:24:25 -0500
From: Tony Hansen <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Sender: [email-masked]

I almost hate to ask, but could we do anything with:
	MIME-Version: 2.0
?

	Tony Hansen
	[email-masked]

[email-masked] wrote:
> 
>>[email-masked] wrote:
> 
> 
>>>the use of content-disposition filenames is seen by many of us as such
>>>an egregious offense.
> 
> 
>>A receiver using the suggested filename to determine the content type
>>is indeed egregious; only the content-type should be used for that.  A
>>receiver blindly using the suggested filename when saving to a file,
>>even if that filename is inconsistent with the content type according
>>to local naming rules, is also wrong.  But I see nothing wrong with a
>>receiver using the suggested filename for its stated purpose (RFC 2183
>>section 2.3): as a starting point for constructing a suitable default
>>filename.
> 
> 
> Neither do I.
> 
> 
>>HTTP had the right idea with Content-Encoding.  It's too bad that it's
>>too late for MIME to use that model.
> 
> 
> Yes, it is much too late, which makes this discussion largely pointless.
> 
> 				Ned

Date: 03 Mar 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Keith Moore <[email-masked]> schrieb/wrote:
> what should happen is that the sender gets a message back that says:
> Your message could not be delivered because it was illegally formatted.
> You should replace your mail reader with one that works correctly.
> See http://foo.bar/mail-readers.html for a list of broken mail readers.

Noone would want to configure his server that way when it blocks  
legitimate mail. And users of broken MUAs would still complain that  
``only this server is so picky'', even if they agree that it's right.

(That's one of the ill side effects of the robustness principle: If most  
servers accept broken messages you generate, you can get away with it.)

Claus
-- 
http://www.faerber.muc.de/

Date: Sun, 2 Mar 2003 15:04:30 -0800
Subject: Re: Unicode newsgroup name options
To: [email-masked]
From: "Eric Fischer" <[email-masked]>
Sender: [email-masked]

Keith Moore wrote:

> what should happen is that the sender gets a message back that says:
>
> Your message could not be delivered because it was illegally 
> formatted.  You should replace your mail reader with one that works 
> correctly.   See http://foo.bar/mail-readers.html for a list of broken 
> mail readers.

What ever happened to the "be liberal in what you accept, conservative 
in what you send" principle?  People who send misformatted messages are 
generally innocent victims of bad software, not malicious evildoers, and 
scolding them isn't going to do any good.

Eric

Date: Sun, 2 Mar 2003 16:44:17 -0500
From: Keith Moore <[email-masked]>
To: [email-masked] (Claus =?ISO-8859-1?Q?F?=ärber)
Cc: [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> ``I could not send mail to you, there was an error message.''
> ``What email client do you use?''
> ``Foo Mail 1.0 for the Bar Operating System.''
> ``Ah, I see. This mailer creates broken email; my mail server rejects
>   them because they are broken.'' (Optional: ``You have to set the
>   option foo to make it bejave correctly.'')
> ``But I can send email to anyone else!''
> 
> Note that this assumes that both users have a knowledge beyond that of
> the average user and far beyond that of a user who writes his text below
> the signature ``because you put the letterhead on the top, don't you?''

what should happen is that the sender gets a message back that says:  

Your message could not be delivered because it was illegally formatted. 
You should replace your mail reader with one that works correctly. 
See http://foo.bar/mail-readers.html for a list of broken mail readers.

of course, that still won't help users who think that if Microsoft makes it,
it must be fine.  then again, such people deserve to lose.

Date: 02 Mar 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Keith Moore <[email-masked]> schrieb/wrote:
> no, I'll be suggesting that they throw away mail that grossly violates
> the standards because trying to support too much garbage adds complexity to
> the mail system and in the long run decreases its reliability.

``I could not send mail to you, there was an error message.''
``What email client do you use?''
``Foo Mail 1.0 for the Bar Operating System.''
``Ah, I see. This mailer creates broken email; my mail server rejects
  them because they are broken.'' (Optional: ``You have to set the
  option foo to make it bejave correctly.'')
``But I can send email to anyone else!''

Note that this assumes that both users have a knowledge beyond that of
the average user and far beyond that of a user who writes his text below
the signature ``because you put the letterhead on the top, don't you?''

Claus
-- 
http://www.faerber.muc.de/

Date: Sun, 2 Mar 2003 13:10:35 -0800 (PST)
From: Benjamin Franz <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: Usefor Mailing List <[email-masked]>, Keith Moore <[email-masked]>, IMAP protocol mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

On Sun, 2 Mar 2003, Arnt Gulbrandsen wrote:

> Benjamin Franz writes:
> > You mean like the mail I just fished out of my box that starts like this?
> >
> >   To: [email-masked] Subject: Welcome to Wells Fargo Online®.
> >
> > I'm glad to know that it wasn't English. Or important.
> 
> Was it 2047-encoded?

No.

-- 
Jerry

"If the code and the comments disagree, then both are probably wrong."
                                        -- Norm Schryer, Bell Labs 

Date: Sun, 2 Mar 2003 10:09:19 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> > it's gotten to the point that anti-spam measures are causing more problems
> > than spam itself.
> 
> That's an unfair statement IMO. Those problems you talk about are 
> indirect effects of spam, and spam is ultimately to blame.

well, they certainly wouldn't exist but for the spam.  but I also think
they've gone overboard.  a lot of legitimate mail deliveries are broken
by the various spam filters and blacklists.

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: Unicode newsgroup name options
Cc: Usefor Mailing List <[email-masked]>, IMAP protocol mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>, Benjamin Franz <[email-masked]>
Date: Sun, 2 Mar 2003 15:15:08 +0100
Sender: [email-masked]

Keith Moore writes:
>>  FWIW, I used to get mail kind of like that from a company I did 
>>  business with. Biggish company, at least in the small Norwegian 
>>  pond. Spam Assassin thought that mail was spam. Then SA became 
>>  popular, and shortly later, that company mended its ways.
>
> then again, SA thinks that perfectly legitimate mail is spam too.

I'm not defending SA. Far be it from me. I'm simply observing that the 
pressure of anti-spam measures is forcing message senders to be more 
careful.

> I have no trouble blocking mail that grossly violates the standards - 
> because it's reasonable to expect people to follow the standards if 
> they want their mail to be readable anyway. I do have trouble with a 
> lot of SA's criteria.

AOL.

> it's gotten to the point that anti-spam measures are causing more problems
> than spam itself.

That's an unfair statement IMO. Those problems you talk about are 
indirect effects of spam, and spam is ultimately to blame.

--Arnt

Date: Sun, 2 Mar 2003 08:55:43 -0500
From: Keith Moore <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> FWIW, I used to get mail kind of like that from a company I did business 
> with. Biggish company, at least in the small Norwegian pond. Spam 
> Assassin thought that mail was spam. Then SA became popular, and 
> shortly later, that company mended its ways.

then again, SA thinks that perfectly legitimate mail is spam too.
I have no trouble blocking mail that grossly violates the standards -
because it's reasonable to expect people to follow the standards
if they want their mail to be readable anyway.  I do have trouble
with a lot of SA's criteria.

it's gotten to the point that anti-spam measures are causing more problems
than spam itself.

From: Arnt Gulbrandsen <[email-masked]>
To: Benjamin Franz <[email-masked]>
Subject: Re: Unicode newsgroup name options
Cc: Usefor Mailing List <[email-masked]>, Keith Moore <[email-masked]>, IMAP protocol mailing list <[email-masked]>, IETF-822 Mailing List <[email-masked]>
Date: Sun, 2 Mar 2003 14:57:08 +0100
Sender: [email-masked]

Benjamin Franz writes:
> You mean like the mail I just fished out of my box that starts like this?
>
>   To: [email-masked] Subject: Welcome to Wells Fargo Online®.
>
> I'm glad to know that it wasn't English. Or important.

Was it 2047-encoded?

FWIW, I used to get mail kind of like that from a company I did business 
with. Biggish company, at least in the small Norwegian pond. Spam 
Assassin thought that mail was spam. Then SA became popular, and 
shortly later, that company mended its ways.

>From this anecdote and others like it I infer that noone has to kowtow 
to standards violators. Spammers get everything wrong most of the time, 
and people hate spam so much that they _will_ accept some damage. To a 
large degree, the onus now is on message senders to avoid looking like 
spammers.

--Arnt

Date: Sat, 1 Mar 2003 23:11:39 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> > no, I'll be suggesting that they throw away mail that grossly violates 
> > the standards because trying to support too much garbage adds complexity to
> > the mail system and in the long run decreases its reliability.
> 
> According to that logic, I should change qmail to reject IDNA addresses.

feel free, Dan.  after all, you know better than anybody else what is good
for the Internet.

Date: Sat, 1 Mar 2003 21:56:49 -0500
From: Keith Moore <[email-masked]>
To: Benjamin Franz <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> You mean like the mail I just fished out of my box that starts like this?
> 
>   To: [email-masked]
>   Subject: Welcome to Wells Fargo Online®.
> 
> I'm glad to know that it wasn't English. Or important.

Hey, if Wells Fargo can't be bothered to follow the standards, I don't need to
do business with them.  It's like those web merchants that won't accept an
email address with a + in the local-part, even though it's perfectly legal.

Keith

p.s. to allow for the occasional glitch, my filter looks for five consecutive
non-ascii characters before declaring the message broken.

Date: 2 Mar 2003 02:58:17 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Keith ``Interoperability? What's that?'' Moore writes:
> no, I'll be suggesting that they throw away mail that grossly violates 
> the standards because trying to support too much garbage adds complexity to
> the mail system and in the long run decreases its reliability.

According to that logic, I should change qmail to reject IDNA addresses.
Let's look at the facts:

   * IDNA addresses grossly violate the full standards, which specify
     that addresses are 7-bit ASCII. IDNA is 7-bit but not ASCII.

   * This IDNA garbage adds a huge amount of unnecessary complexity to
     the mail system. UTF-8 achieves the same goal at much lower cost.

   * IDNA, in both the short run and the long run, decreases the
     reliability of the mail system; IDNA causes many failures that
     don't occur with ASCII addresses.

Meanwhile, I certainly don't have any legitimate IDNA mail in any of my
mailboxes; and I'm sure that blocking IDNA will stop some spam.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sat, 1 Mar 2003 18:51:45 -0800 (PST)
From: Benjamin Franz <[email-masked]>
To: Keith Moore <[email-masked]>
cc: [email-masked], <[email-masked]>, <[email-masked]>
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

On Fri, 28 Feb 2003, Keith Moore wrote:

> >   [ rejecting bytes 128-255 as an anti-spam technique ]
> > > zero false positives
> > 
> > Wrong. A huge number of non-spam messages use bytes 128-255.
> 
> I have yet to see a single one.  of course, since I only speak English,
> there's a very high probability that any message that I see that contains
> those bytes in a message header is spam.  at any rate, they're all in 
> violation of the standards, and as far as I'm concerned, it's reasonable to
> outright reject any message that is so flagrantly broken.

You mean like the mail I just fished out of my box that starts like this?

  To: [email-masked]
  Subject: Welcome to Wells Fargo Online®.

I'm glad to know that it wasn't English. Or important.

-- 
Benjamin Franz

"If the code and the comments disagree, then both are probably wrong."
                                        -- Norm Schryer, Bell Labs 

Date: Sat, 1 Mar 2003 18:19:43 -0500
From: Keith Moore <[email-masked]>
To: "D. J. Bernstein" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> Brilliant, Keith. Next you'll be suggesting that MTA authors throw away
> Chinese mail as an anti-spam technique with ``zero false positives.''

no, I'll be suggesting that they throw away mail that grossly violates 
the standards because trying to support too much garbage adds complexity to
the mail system and in the long run decreases its reliability.

Date: Sat, 1 Mar 2003 18:13:29 -0500
From: Keith Moore <[email-masked]>
To: Benjamin Franz <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

> You _most certainly_ have false postives if you block \200 - \377.

it's certainly the case that *I* have no false positives when filtering on
unencoded non-ascii characters in the message header.  any mail that I get
that's not in English and is not encoded correctly isn't something that I need
to read - assuming I would even be able to read it, which is unlikely.

that and there's such a thing as "too broken to deal with".   untagged,
unencoded 8bit text IMHO clearly qualifies.  I recognize that others' opinions
may vary, and how you configure your MTA is your own business.  But I don't
see any obligation for MTAs or UAs to deal with whatever random trash is
thrown at them.

Date: 01 Mar 2003 00:00:00 +0000
From: [email-masked] (=?ISO-8859-1?Q?Claus_F=E4rber?=)
To: [email-masked]
Subject: Re: gzip-8bit
Organization: "Old Europe" - and proud
Sender: [email-masked]

Marc Mutz <[email-masked]> schrieb/wrote:
> The next hop doesn't support 8BITMIME. No problem. This is an
> intelligent server that acts by the gateway model of rfc1652 and tries
> to recode the offending body parts. Whooops... bounce. :-(

> Am I missing something?

Yes, the proposed encapsulation.

The 8BITMIME server would only see an application/message-partial body  
labelled as 8bit. It can recode it with quoted-printable or base64  
(because that would be allowed for application/message-partial).

Only an EMIME MTA would see the enclosed EMIME entity directly. If it  
wants to send it to a non-EMIME MTA (such as a 8BITMIME-only MTA), it  
has to encapsulate it in the same way. This is very similar to message/ 
partial fragmentation (except that it uses application/message-partial  
and that there's only one part).

Claus
-- 
http://www.faerber.muc.de/

Date: Sat, 01 Mar 2003 14:54:20 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
Cc: [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > the use of content-disposition filenames is seen by many of us as such
> > an egregious offense.

> A receiver using the suggested filename to determine the content type
> is indeed egregious; only the content-type should be used for that.  A
> receiver blindly using the suggested filename when saving to a file,
> even if that filename is inconsistent with the content type according
> to local naming rules, is also wrong.  But I see nothing wrong with a
> receiver using the suggested filename for its stated purpose (RFC 2183
> section 2.3): as a starting point for constructing a suitable default
> filename.

Neither do I.

> HTTP had the right idea with Content-Encoding.  It's too bad that it's
> too late for MIME to use that model.

Yes, it is much too late, which makes this discussion largely pointless.

				Ned

Date: Sat, 1 Mar 2003 14:17:06 -0800 (PST)
From: Benjamin Franz <[email-masked]>
To: Keith Moore <[email-masked]>
cc: "D. J. Bernstein" <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

On Fri, 28 Feb 2003, Keith Moore wrote:

> > Rejecting bytes 128-255 as an anti-spam technique is idiotic. If any
> > noticeable percentage of the Internet started doing this, spammers would
> > simply stop using those bytes. (Most of them never even started.)
> 
> the number of spammers doing this is not a concern.  a significant percentage
> of spam messages are doing this.  and since the messages themselves are
> illegally formatted and unlikely to be correctly displayed, it seems perfectly
> reasonable to treat them as trash.  it's one of the few effective filtering
> techniques with zero false positives.

I've got dozens of non-spam emails in my inbox that have raw 8 bit text
embedded in them - both in headers and the body - right now. Some of them
were quite important to me to receive (things like e-ticket airline
itineraries from a _MAJOR_ popular online travel booking company, and
email from my bank). You _most certainly_ have false postives if you block
\200 - \377.

-- 
Benjamin Franz

"If the code and the comments disagree, then both are probably wrong."
                                        -- Norm Schryer, Bell Labs 

Date: Sat, 1 Mar 2003 21:56:08 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

[email-masked] wrote:

> the use of content-disposition filenames is seen by many of us as such
> an egregious offense.

A receiver using the suggested filename to determine the content type
is indeed egregious; only the content-type should be used for that.  A
receiver blindly using the suggested filename when saving to a file,
even if that filename is inconsistent with the content type according
to local naming rules, is also wrong.  But I see nothing wrong with a
receiver using the suggested filename for its stated purpose (RFC 2183
section 2.3): as a starting point for constructing a suitable default
filename.

> See RFC 2048 section 2.2.1. It really could not be clearer.

Thanks for the reference.  Actually, it could be clearer.  It says:

    Media types must function as an actual media format: Registration of
    things that are better thought of as a transfer encoding ... is not
    allowed.

The problem is that compression is neither a media format nor a transfer
encoding.  A transfer encoding is an encoding needed to allow the
message to conform to the transport requirements.  Compression is not
needed for that.  The purpose of compression is to save resources, not
only during transport but perhaps also before transport (on the original
sending host) and after transport (on the final receiving host).  It's
quite likely that the file being attached was already compressed before
the sending MUA ever saw it, so it's hardly a "transfer encoding".

HTTP had the right idea with Content-Encoding.  It's too bad that it's
too late for MIME to use that model.

AMC

Date: Sat, 01 Mar 2003 12:41:28 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
Cc: [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > We've been over this many times before.  Compression cannot be a
> > separate attribute because current agents assume that once the CTE
> > is removed they have the data identified by the content-type in hand
> > and no further processing is required.  Adding compression as, say, a
> > different header is therefore something that is guaranteed to cause
> > massive breakage.  Whereas adding compression through a new CTE is
> > something that should not cause problems with standards compliant
> > agents.

> Except for the 1000:1 expansion risk that could bring down transport
> agents that aren't sufficiently paranoid when altering the
> transfer-encoding.

And the solution is to call this out as a security consideration so transport
agents can be made to be sufficiently paranoid.

> And it's a bit gross to have to define N*M transfer
> encodings to support N escape mechanisms (quoted-printable, base64,
> escaped-8bit) and M compression schemes (gzip, deflate, bzip2, ...).

Gross perhaps, but hardly necessary.

> Here's a way to separate the compression from the escaping in a
> backward-compatible way.  I don't claim that this is the best way, but
> it's something to consider.

Sorry, it isn't worth considering. This just breaks a different rule, that of
media types actually describing the associated content. There is a vast amount
of infrastructure that depends on this being the case. (This is why the use of
content-disposition filenames is seen by many of as such an egregious offense.)

BTW, this particular rule is written down. See RFC 2048 section 2.2.1. It
really could not be clearer.

> Another approach would be to introduce structure into the
> transfer-encoding, and recommend that transport agents ignore all but
> the last component:

> Content-Type: application/postscript
> Content-Disposition: attachment; filename="foo.ps"
> Content-Transfer-Encoding: gzip|escaped-8bit

> Note that the semantics of these two approaches are different, as
> reflected in the suggested filename.

At the time MIME was defined I argued for such a structure. If memory serves
I was pretty much a lone voice in this; the opposition to it was overwhelming.

				Ned

Date: Sat, 1 Mar 2003 20:28:03 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: gzip-8bit
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

[email-masked] wrote:

> We've been over this many times before.  Compression cannot be a
> separate attribute because current agents assume that once the CTE
> is removed they have the data identified by the content-type in hand
> and no further processing is required.  Adding compression as, say, a
> different header is therefore something that is guaranteed to cause
> massive breakage.  Whereas adding compression through a new CTE is
> something that should not cause problems with standards compliant
> agents.

Except for the 1000:1 expansion risk that could bring down transport
agents that aren't sufficiently paranoid when altering the
transfer-encoding.  And it's a bit gross to have to define N*M transfer
encodings to support N escape mechanisms (quoted-printable, base64,
escaped-8bit) and M compression schemes (gzip, deflate, bzip2, ...).

Here's a way to separate the compression from the escaping in a
backward-compatible way.  I don't claim that this is the best way, but
it's something to consider.

Content-Type: encoded/gzip
Content-Decoded-Type: application/postscript
Content-Disposition: attachment; filename="foo.ps.gz"
Content-Transfer-Encoding: escaped-8bit

This would remove any temptation for transport agents to decompress the
body.

Another approach would be to introduce structure into the
transfer-encoding, and recommend that transport agents ignore all but
the last component:

Content-Type: application/postscript
Content-Disposition: attachment; filename="foo.ps"
Content-Transfer-Encoding: gzip|escaped-8bit

Note that the semantics of these two approaches are different, as
reflected in the suggested filename.

AMC

From: Marc Mutz <[email-masked]>
Organization: "Old Europe" - and proud
To: [email-masked]
Subject: Re: gzip-8bit
Date: Sat, 1 Mar 2003 20:37:08 +0100
Sender: [email-masked]

--Boundary-02=_5vQY+swR53VyjVj
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Description: signed data
Content-Disposition: inline

On Friday 28 February 2003 20:03, Bruce Lilly wrote:
<snip>
> A UA issue:
> Currently, with QP and Base64, binary content can simply
> be Base64 encoded (QP isn't terribly effective for binary
> content, nor is Base64 terribly useful for text which has
> a few 8-bit octets).  So the UA can pretty much do the
> right thing without user interaction or user
> sophistication.

base64 remains the safe bet even with deflate-* added...

>  Of course, a knowledgable user could
> apply compression before attaching.  Unfortunately in
> practice there exists a large variety of compression
> and/or packaging mechanisms (e.g. just the other day I
> recieved an attachment in "Stuffit" format, and I had
> to track down a decoder for that).
>
> With a larger choice for binary attachment encodings, things
> get a bit more complicated.  The UA can't necessarily
> determine whether the MTA will support 8BITMIME (the user
> may be off-line).

The the UA should record the capabilities of the SMTP server for that=20
account or default to a 7bit domain CTE.

> Yet a choice must be made among the
> available encodings. Negotiation along the lines
> of RFC 3297 might be useful.  Simply asking the user to
> choose probably isn't practical, as the vast majority of
> users have little technical sophistication.
>
> However, the UA could presumably choose between a straight
> base64 and deflate-base64 based on how well the binary
> content compresses.
<snip>

If a MUA wants to automatically determine the CTE to use (without always=20
using base64), it has to do character frequency counting (either by=20
hand or using prior knowledge based on the mime type). If you do that,=20
you can easily extend that to calculate the entropy-per-char of the=20
data and base your choice of using deflate-* or not on that, although=20
it's probably not worth it anyway, since deflate has a fairly low=20
overhead for already compressed data.

Marc

=2D-=20
Military justice is to justice what military music is to music.
                                                  -- Groucho Marx

--Boundary-02=_5vQY+swR53VyjVj
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_5vQY+swR53VyjVj--

Date: 1 Mar 2003 18:51:23 -0000
Mail-Followup-To: [email-masked], [email-masked], [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Re: Unicode newsgroup name options
Sender: [email-masked]

Keith Moore writes:
> since I only speak English, there's a very high probability that any
> message that I see that contains those bytes in a message header is spam.

Brilliant, Keith. Next you'll be suggesting that MTA authors throw away
Chinese mail as an anti-spam technique with ``zero false positives.''

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Sat, 01 Mar 2003 08:31:52 -0800 (PST)
From: [email-masked]
Subject: Re: gzip-8bit
To: Bruce Lilly <[email-masked]>
Cc: [email-masked], [email-masked]
Sender: [email-masked]

> [email-masked] wrote:

> > Very unlikely, actually. CTE has always been extensible and has always allowed
> > X- tokens. The rules for handling unknown CTEs are also well defined.

> Actually, there is a problem; the rule seems to be the following
> from RFC 2045, section 6.4:

>     Any entity with an unrecognized Content-Transfer-Encoding must be
>     treated as if it has a Content-Type of "application/octet-stream",
>     regardless of what the Content-Type header field actually says.

> That basically means that the content is treated as opaque. Obviously,
> one cannot decode an unknown encoding, so that part of the CTE is
> moot.  But the other part, viz. the content domain (7bit, 8bit, or
> binary) is unspecified.  So an MTA presented with a message with
> unknown CTE cannot tell (based on MIME header fields) whether or
> not the next stage of transfer requires 8BITMIME or binary transport
> or plain old 7bit transport.

First of all, there is normally supposed to be no need for a transport MTA to
need to know the domain of the message as a whole or of its various parts. Only
the range of the message as a whole is supposed to matter to the transport. And
the range is determined by the mechanism used to get the message to the MTA in
the first place. That could be conventional SMTP (7bit), 8bitMIME (8bit),
binary SMTP (binary), or some other mechanism that lies outside of the
standards. 

Of course the mechanism may not represent the true state of affairs, e.g.
message may be 8bit while being transferred over regular SMTP. It could even be
7bit while being transferred with 8bitMIME. Regardless, if an MTA elects not to
believe the transport labelling, it would be well advised to check the actual
range of the message rather than believing what the various CTE labels say.

> I suppose the MTA in question could peek inside the message body to
> determine the domain. Unless of course it's not all available (e.g.
> it's in a stream too large to be held in memory) -- oops.

Here you appear to be talking about the range of the message, not the domain.
But again, the range is something you're supposed to determine from the
transport, not by looking at the CTE labels of the parts.

And when it comes to upgrading or downgrading of an individual part, both the
domain and range of that part may be important. But the domain can only be
determined by decodiing the entire part, so the oops you refer to already
exists with the present set of CTEs.

As far as the range of an individual part goes, it should be known if the CTE
is known. if the CTE is unknown then you cannot perform the upgrade or
downgrade operation anyway. If the operation is an optional one you move
on. If it isn't optional then you have two choices: (1) Bounce the message or
(2) Check the actual range of the data (which of course can be done without
decoding) and see if it is OK to move forward without performing the downgrade
operation.
 
The bottom line is that viable options exist for the handling of unknown CTEs
even when dealing with a strema, limited memory, or whatever. They may not be
particularly pleasant, but I will again point out that we do in fact have
experience with what happens when CTEs that lie outside the original set are
used.

> If compression is to be considered part of the CTE, then it is
> certainly conceivable that some CTE may have binary domain, and
> lacking explicit information an assumption of binary domain is
> safe (it won't result in foisting incompatible content on the
> next hop).  OTOH if compression is considered a separate
> attribute, then there doesn't seem to be much point in a CTE with
> a binary domain other than "binary", and one could assume 8bit
> domain for an unknown CTE.

We've been over this many times before. Compression cannot be a separate
attribute because current agents assume that once the CTE is removed they have
the data identified by the content-type in hand and no further processing is
required. Adding compression as, say, a different header is therefore something
that is guaranteed to cause massive breakage. Whereas adding compression
through a new CTE is something that is should not cause problems with standards
compliant agents.

> The danger is that as both of the 2045-specified encodings (as
> opposed to the identity CTE values) have 7bit domains, there may
> be implementations that presume a 7bit domain for any unrecognized
> CTE (if I were a gambler, I'd bet on it).

I'm not much of a gambler, and even if I was this certainly isn't a bet I would
ever have made. But regardless, my sympathy is limited since there never was
any statement that additional CTEs would always be 7bit.

				Ned