Subject: Re: CFWS: beaten, down, kicked to death, and still being flogged (Was Re: Malformed header - what would you do?)
From: "Charles Lindsey" <[email-masked]>
To: [email-masked]
Date: Mon, 25 Jul 2005 09:50:00 GMT
Newsgroups: local.mime
List-Id: <ietf-822.imc.org>

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Fri July 22 2005 09:48, Charles Lindsey wrote:

>> Now of course we all know perfectly well that was not the intention, as
>> Appendix B makes clear.

>It is an accepted rule of logic that if one starts with a proposition and
>by following sound reasoning finds a contradiction, the proposition must be
>false.  That is a rule which is fundamental to a significant number of
>mathematical proofs, and more than three decades ago was routinely taught
>to students. <insert lament about the decline in educational rigor here>

All very true. I stated a proposition, and demonstrated that it followed
from the norative text of RFC 2822. If you claim that you can derive a
contradictionfrom the proposition, then please show us your derivation.

Note, however, that neither Appendix B nor my remark concerning "we all
know perfectly well" form part of the normative text of RFC 2822, hence
their disagreement with the proposition proves nothing, though they are
highly suggestive of the presence of a Bug.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2822 revised grammar
Date: Sun, 24 Jul 2005 19:08:33 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Sun July 24 2005 12:48, Frank Ellermann wrote:
> 
> Bruce Lilly wrote:

>  [limit 76]
> > Unfolding, refolding, and other modification needs to take
> > that into account.  It's not a "bug", it's merely a fact that
> > MIME is widely used and cannot be ignored.
> 
> Widely used _because_ it can be ignored, relays should not try
> to unfold or refold header fields, it's not their business.

List expanders sometimes insert " [list-name]" in the Subject field;
as that lengthens the line, the 76 char limit if there are any existing
encoded-words needs to be taken into account.  Likewise for UAs which
prepend noise to the Subject field (e.g "FWD: " etc.) when composing
responses.  It's not only a relay issue.
  
> If they do it anyway (= wannabe gateway) they can ignore the
> limit 76.

What, and convert a MIME-conforming message into junk so that the
recipient can't use it?

> BTW, I just stumbled over another obscure limit, a 
> boundary is limited to 70 characters.

Obscure?  It's in RFC 2046 and was in its predecessors.

> BTW, your <obs-utext> is _very_ different from 2822, you have
> only utext or NUL, 2822 has also %d127, bare CR, and bare LF.

%d127 is in NO-WS-CTL (in utext).  Lone CR and lone LF are
unconditionally prohibited in the message header.  obs-text permits
them (though it isn't used anywhere; there should probably be some
provision in "body" (for binary message content [*])).

>  encoded-word = "=?" charset "?" encoding "?" encoded-text "?="
>  charset = token    ; see section 3
>  encoding = token   ; see section 4
> [...]
>  encoded-text = 1*<Any printable ASCII character other than "?"
>                    or SPACE>

That permits '(', ')', and '\\', which won't work in comments.  Also
'.' and others prohibited in phrases.

> No "?".  Your <cchar> in <cew> allows "?", and that's wrong.
> It doesn't depend on the encoding in section 4.  Even if we'd
> invent =?us?X-BLURF?whatever?= now we cannot use a "?" in the
> <encoded-text>.
> 
> > It is absolutely related to the encoding
> 
> It's not, neither SPACE nor "?" is always REQUIRED.

Can't happen with B encoding.  SP[ACE] can't happen with Q encoding.
  
> > At minimum, a separate rule would have to be formulated.
> 
> No, cchar IS already the separate rule.  You just have to move
> "?" from cchar to ctext:
> 
>  cchar = %d33-39 / %d42-62 / ; Printable US-ASCII characters
>          %d64-91 / %93-126   ; excl. "(", ")", "?", or "\"
> 
>  ctext = NO-WS-CTL / cchar / "?" ; add "?" here

Maybe, but that obscures the meaning; cchar => characters allowed in
comments.
 
> The only other xref of <cchar> is <cew>, and that's precisely
> where you don't want the "?".  Both <uew> and <pew> are okay.
> 
> > There would be a temptation to have not one, but two rules
> > because the characters are different for Q and B encoding
> 
> NAK.  Restrictions depending on the encoding are only relevant
> for decoders, not for your purposes.  You're not interested to
> decode this crap, you only want to parse it.  But to parse it
> you need to know the 2047 "?" rule, it's essential.

Perhaps, but a separate rule would be the way to go.
---------
* perfectly legal for MIME with Content-Transfer-Encoding: binary, so
taking MIME into account should be part of "body", i.e. permitted for
generation.  If one puts on blinders and pretends MIME doesn't exist,
the tendency would be to leave it out (as is currently the case) or
put it in an obs- rule.  Another reason to take MIME into account when
updating 2822.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: 2822 revised grammar
Date:  Sun, 24 Jul 2005 18:48:20 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 114
Sender: [email-masked]

Bruce Lilly wrote:

>> In syntax it is, <word> is an "abbreviation" for
>> <atom> or <quoted-string>.

> No sale.

Fine, also true for your offer to treat the name <CFWS> as an
"abbreviation" different from other names like <word>, <blurf>,
or <sugar>.

> Among the symbols defined in RFC 822 and referenced in this
> memo are: 'addr-spec', 'atom', 'CHAR', 'comment', 'CTLs',
> 'ctext', 'linear-white-space', 'phrase', 'quoted-pair'.
> 'quoted-string', 'SPACE', and 'word'.

CHAR is bad => 822 says ASCII, 2234(bis) says ASCII minus NUL.
Fortunately 2045 uses only CHAR excl. CTL, WSP, or <tspecials>.

> the revised grammar attempts a solution by clearly defining
> the different types of encoded-words (in phrases, in
> comments, and in unstructured fields)

Yes, I've seen <cew>, <pew>, and <uew> in your text.

> It would still require careful rewording of a 2047 successor.
> Because of the interdependence, such issues need to be
> considered when the 2822 successor is produced.

Considerations are fine, only solving it in the same text could
be too much.  After reading draft-klensin-emailaddr-i18n-03.txt
I'm in my "grumpier than anybody claiming to be grumpy" mood:

John proposes some kind of "FUSSP" without the critical second
"S" for I18N.  At least he wants to get rid of <quoted-pair> -
if I understood it correctly, it's a plan for the 22nd century.

 [limit 76]
> Unfolding, refolding, and other modification needs to take
> that into account.  It's not a "bug", it's merely a fact that
> MIME is widely used and cannot be ignored.

Widely used _because_ it can be ignored, relays should not try
to unfold or refold header fields, it's not their business.

If they do it anyway (= wannabe gateway) they can ignore the
limit 76.  BTW, I just stumbled over another obscure limit, a
boundary is limited to 70 characters.  Too many magic numbers,
too many subsets of ASCII, too many RfCs, it's a royal PITA.

> It is unclear whether or not
>    Subject:=?us?q?foo_bar?=
> is legal

First guess: yes.  No %d127, no NUL, no bare CR. no bare LF =>
no problem with 822 vs. 2822.  Or 2821, why on earth has 2821
%d127 as an ordinary instead of a control character ?

BTW, your <obs-utext> is _very_ different from 2822, you have
only utext or NUL, 2822 has also %d127, bare CR, and bare LF.

> It's unclear whether
>    Subject:foo
>     =?us?q?bar_baz?=
> is legal, because the encoded word is separated from the
> text "foo" not by linear-white-space but by line folding.

My model of FWS is "the same as LWSP, but at most one CRLF".

Actually a similar argument as you used it for FWS wrt CFWS,
my justification is "different standards, different authors".

>> Yes, but no "?" is a _general_ rule for <encoded-text> in
>> chapter 2, it does not depend on the encoding.

> ?!?

 encoded-word = "=?" charset "?" encoding "?" encoded-text "?="
 charset = token    ; see section 3
 encoding = token   ; see section 4
[...]
 encoded-text = 1*<Any printable ASCII character other than "?"
                   or SPACE>

No "?".  Your <cchar> in <cew> allows "?", and that's wrong.
It doesn't depend on the encoding in section 4.  Even if we'd
invent =?us?X-BLURF?whatever?= now we cannot use a "?" in the
<encoded-text>.

> It is absolutely related to the encoding

It's not, neither SPACE nor "?" is always REQUIRED.

> At minimum, a separate rule would have to be formulated.

No, cchar IS already the separate rule.  You just have to move
"?" from cchar to ctext:

 cchar = %d33-39 / %d42-62 / ; Printable US-ASCII characters
         %d64-91 / %93-126   ; excl. "(", ")", "?", or "\"

 ctext = NO-WS-CTL / cchar / "?" ; add "?" here

The only other xref of <cchar> is <cew>, and that's precisely
where you don't want the "?".  Both <uew> and <pew> are okay.

> There would be a temptation to have not one, but two rules
> because the characters are different for Q and B encoding

NAK.  Restrictions depending on the encoding are only relevant
for decoders, not for your purposes.  You're not interested to
decode this crap, you only want to parse it.  But to parse it
you need to know the 2047 "?" rule, it's essential.  Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2822 revised grammar
Date: Sun, 24 Jul 2005 08:59:18 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Sun July 24 2005 02:11, Frank Ellermann wrote:
> 
> Bruce Lilly wrote:
> 
> >> word            =       atom / quoted-string
> > It's still not an abbreviation.
> 
> In syntax it is, <word> is an "abbreviation" for
> <atom> or <quoted-string>.

No sale.

> CFWS = *([FWS] comment) (([FWS] comment) / FWS)

> >> your idea: "Read <word> as <atom> or <quoted-string>".
> > I never said that.
> 
>  <[email-masked]>
> | If you read CFWS as "comments and/or folding
> | whitespace" there's no problem. 

That's in prose, and I didn't say "zero or more instances of
FWS followed by a comment and followed by either FWS and a comment
or FWS alone" or anything like that; I explained how the abbreviation
is used in prose, in a manner consistent with use throughout the
document including the Appendix B statement of the rule.
 
>  [revised grammar]
> > as mentioned, there are issues related to phrase that
> > affect encoded-words.
> 
> Sorry, you lost me there, is this about <specials> vs.
> <tspecials> and similar nits ?  Or in other words, is
> is possible to fix it in a 2231bis / 2047bis based on
> 2822, or is it something that has to be done together
> with 2822bis ?  Do you have an example ?

2047, currently based on 822, says:

   Among the symbols defined in RFC 822 and
   referenced in this memo are: 'addr-spec', 'atom', 'CHAR', 'comment',
   'CTLs', 'ctext', 'linear-white-space', 'phrase', 'quoted-pair'.
   'quoted-string', 'SPACE', and 'word'.  Successful implementation of
   this protocol extension requires careful attention to the RFC 822
   definitions of these terms. 

A revision taking 2822 or its successor into account would have
similar dependencies. As an example, 2047 also says:

   For example, an 'encoded-word' in a
   'phrase' preceding an address in a From header field may not contain
   any of the "specials" defined in RFC 822.

2822 defines phrase as:

word            =       atom / quoted-string

phrase          =       1*word / obs-phrase

quoted-string   =       [CFWS]
                        DQUOTE *([FWS] qcontent) [FWS] DQUOTE
                        [CFWS]

atom            =       [CFWS] 1*atext [CFWS]

That means that the restriction on encoded-words "in a 'phrase'", if
applied to 2822, would include encoded-words in comments, since the
comments (as part of CFWS) are *in* the phrase (unlike 822, where comments
were not specified in the phrase syntax, but could be freely inserted in
many places).  That is quite a difficult problem to solve; the revised
grammar attempts a solution by clearly defining the different types of
encoded-words (in phrases, in comments, and in unstructured fields) and
where they may appear.  It would still require careful rewording of a
2047 successor.  Because of the interdependence, such issues need to be
considered when the 2822 successor is produced.

> It's the most important feature of MIME that you can
> completely ignore it whenever it pleases you, without
> affecting the message format or transport (modulo some
> 8bit issues).  Have you found any bug where that won't
> work ?

RFC 2047:

   While there is no limit to the length of a multiple-line header
   field, each line of a header field that contains one or more
   'encoded-word's is limited to 76 characters.

Unfolding, refolding, and other modification needs to take that into
account.  It's not a "bug", it's merely a fact that MIME is widely used
and cannot be ignored.

> > The problem is that the 2822 definitions of phrase,
> > unstructured, and comment have to be consistent with
> > what 2047 uses.
> 
> Yes, they have to be consistent, e.g. if I know CFWS
> and treat it as "semantically invisible" I'd want the
> same result with or without knowing 2047 / 2231.

See above w.r.t. "in a phrase".
 
> For <unstructured> I've no idea how that could be ever
> a "problem", as you said.

RFC 2047 again:

(1) An 'encoded-word' may replace a 'text' token (as defined by RFC 822)
    in any Subject or Comments header field, any extension message
    header field, or any MIME body part field for which the field body
    is defined as '*text'.  An 'encoded-word' may also appear in any
    user-defined ("X-") message or body part header field.

    Ordinary ASCII text and 'encoded-word's may appear together in the
    same header field.  However, an 'encoded-word' that appears in a
    header field defined as '*text' MUST be separated from any adjacent
    'encoded-word' or 'text' by 'linear-white-space'.

That's the entire section 5 text regarding unstructured field use of
encoded-words.  Note that it refers to 'text' but says nothing about
specials.  It is unclear whether or not
   Subject:=?us?q?foo_bar?=
is legal (the encoded-word is adjacent to the colon, which isn't part
of the field body, and isn't field body 'text', although colon is a valid
character in the 'text' grammar rule).  With 2047 and 822, there's also
a question about line folding, which differs from "linear-white-space".
It's unclear whether
   Subject:foo
    =?us?q?bar_baz?=
is legal, because the encoded word is separated from the text "foo" not
by linear-white-space but by line folding.

These issues need to considered when the message format syntax is specified
so that the necessary encoded-word usage rules can be properly (clearly,
concisely, and unambiguously) specified without requiring yet another
revision to the message format specification.

> But since you insisted on 
> allowing the obs-phrase dots in USEFOR I fear that it
> could bite "us" (= USEFOR).  And if it does it's also
> a problem in 2822 or 2047.

Not a big problem. Dot is a special and is subject to the rules regarding
specials and encoded-words in a phrase.  The 2822 obs-phrase rule affects
an unquoted dot in a phrase.  Currently, dot is not allowed in a Q-encoded
encoded-word; that could conceivably be revised to allow it e.g. adjacent
to an encoded accented initial.  Otherwise it would be treated per 2047
and 2822 rules, e.g.
   From: Egbert =?iso-8859-1?q?_=C9?= . Egglesthwaite
would match2822 obs-phrase and 2047 rules; quoting the dot would be fully
conforming for generation.

> >> (=?us-ascii?Q?example?=?=)  Something's odd there.
>  
> > See RFC 2047 section 5, paragraphs labeled "(2)" (and
> > referring to the errata for 2047).
> 
> Okay, no "(", ")", "\", and separated by LWSP from ctext
> or any other encoded-word.
> 
> > Of course a literal '?' can't appear in the encoded-text
> > (2047 sect. 4.2 "(3)").  However, B and Q encodings
> > differ, and there may be private-use or future encodings
> > will still other characteristics
> 
> Yes, but no "?" is a _general_ rule for <encoded-text> in
> chapter 2, it does not depend on the encoding.

?!?  There is no way to have a question mark in B-encoded encoded-text
as that is not one of the 64 characters used in B encoding.  It is
absolutely related to the encoding, which is why special mention is made
for Q encoding but not for B encoding (where it cannot happen).

> And your 
> cchar allows "?" (%d63):
> 
>     cchar =  %d33-39 /       ; Printable US-ASCII
>              %d42-91 /       ;  characters not including "(",
>              %d93-126        ;  ")", or "\"
> 
> > While the ABNF could be tweaked somewhat, there is a limit
> > to what can be achieved with ABNF
> 
> Just exclude "?" in addition to "(", ")", "\".   Bye, Frank

No, because cchar is also used for comment content, and '?' is
perfectly legal in a comment.  At minimum, a separate rule would
have to be formulated.  There would be a temptation to have not
one, but two rules because the characters are different for Q and B
encoding, but then there is no provision for private-use or future
encodings.  So yes, it *could* be done, but there would have to be
some consensus about whether it's worthwhile going to that level of
detail for something already covered in the encoding rules and in
consideration of the liabilities w.r.t. private-use and future
encodings. 

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: 2822 revised grammar
Date:  Sun, 24 Jul 2005 08:11:42 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 103
Sender: [email-masked]

Bruce Lilly wrote:

>> word            =       atom / quoted-string
> It's still not an abbreviation.

In syntax it is, <word> is an "abbreviation" for
<atom> or <quoted-string>.  Otherwise - only if
you're in hardcore nitpicking mode - let's state
that CFWS is an acronym.

That's pretty irrelevant for the syntax, a few
days ago you proposed <blurf> or <usenet-from>.

Maybe it's "bruce lilly's ugly restricted from",
but it won't change the syntax of this <blurf>.

Similar the syntax of CFWS isn't affected by
any "C or FWS" or rather "C and/or FWS", it is:

CFWS = *([FWS] comment) (([FWS] comment) / FWS)

The name of a rule is only a courtesy for the
reader, we could replace all <CFWS> by <blurf>
without effect for the message format.

It can have effects for the semantics, because
it's not obvious what a <blurf> should be, but
<CFWS> is also not obvious, it has a paragraph
in prose with explanations and a motivation.

Maybe a name like <sugar>, <noise>, or <invis>
instead of <CFWS> would be also fine.  It would
at least spare us this discussion, where you
claimed that every FWS is a CFWS, and therefore
rules for CFWS also affect FWS outside of CFWS.

Good names like <host> or <id-domain> instead of
<blurf> or <id-right> are very important, but
they have no effect for the syntax.

>> your idea: "Read <word> as <atom> or <quoted-string>".
> I never said that.

 <[email-masked]>
| If you read CFWS as "comments and/or folding
| whitespace" there's no problem. 

 [revised grammar]
> as mentioned, there are issues related to phrase that
> affect encoded-words.

Sorry, you lost me there, is this about <specials> vs.
<tspecials> and similar nits ?  Or in other words, is
is possible to fix it in a 2231bis / 2047bis based on
2822, or is it something that has to be done together
with 2822bis ?  Do you have an example ?

It's the most important feature of MIME that you can
completely ignore it whenever it pleases you, without
affecting the message format or transport (modulo some
8bit issues).  Have you found any bug where that won't
work ?

> The problem is that the 2822 definitions of phrase,
> unstructured, and comment have to be consistent with
> what 2047 uses.

Yes, they have to be consistent, e.g. if I know CFWS
and treat it as "semantically invisible" I'd want the
same result with or without knowing 2047 / 2231.

For <unstructured> I've no idea how that could be ever
a "problem", as you said.  But since you insisted on
allowing the obs-phrase dots in USEFOR I fear that it
could bite "us" (= USEFOR).  And if it does it's also
a problem in 2822 or 2047.

>> (=?us-ascii?Q?example?=?=)  Something's odd there.
 
> See RFC 2047 section 5, paragraphs labeled "(2)" (and
> referring to the errata for 2047).

Okay, no "(", ")", "\", and separated by LWSP from ctext
or any other encoded-word.

> Of course a literal '?' can't appear in the encoded-text
> (2047 sect. 4.2 "(3)").  However, B and Q encodings
> differ, and there may be private-use or future encodings
> will still other characteristics

Yes, but no "?" is a _general_ rule for <encoded-text> in
chapter 2, it does not depend on the encoding.  And your
cchar allows "?" (%d63):

    cchar =  %d33-39 /       ; Printable US-ASCII
             %d42-91 /       ;  characters not including "(",
             %d93-126        ;  ")", or "\"

> While the ABNF could be tweaked somewhat, there is a limit
> to what can be achieved with ABNF

Just exclude "?" in addition to "(", ")", "\".   Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: 2822 revised grammar
Date: Sat, 23 Jul 2005 16:29:21 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Fri July 22 2005 22:27, Frank Ellermann wrote:
> 
> Bruce Lilly wrote:

> > Last time I checked, "word" wasn't an abbreviation.
> 
> Line 719 in 2822:
> 
> word            =       atom / quoted-string

It's still not an abbreviation.
 
> Using your idea: "Read <word> as <atom> or <quoted-string>".

I never said that.
 
> >> for 2822bis we better stick to the 2822 idea, no MIME.
> 
> > I guess that's a matter for community discussion and
> > consensus. Ignoring MIME doesn't seem rational
> 
> IBTD, it's a known way to limit the problem space, it worked
> for 2822.

No, as mentioned, there are issues related to phrase that affect
encoded-words.
 
> > there needs to be some sort of reconciliation between MIME
> [...]
> > and 2822
> 
> Sure, we also need a 2231bis etc.
> 
> > it really needs to be examined carefully.
> 
> Yes, starting with the 2822bis part would allow this.  If it
> turns out to be "not interesting enough" maybe add MIME later.

The problem is that the 2822 definitions of phrase, unstructured, and
comment have to be consistent with what 2047 uses.
 
> E.g. check <cew>, AFAIK you can't have "?=" in an encoded word:
> (=?us-ascii?Q?example?=?=)  Something's odd there.

See RFC 2047 section 5, paragraphs labeled "(2)" (and referring to
the errata for 2047).  Of course a literal '?' can't appear in the
encoded-text (2047 sect. 4.2 "(3)").  However, B and Q encodings
differ, and there may be private-use or future encodings will still
other characteristics; while '=' cannot appear in a Q-encoded
encoded-text sequence, it is not only legal but necessary for padding
in some B-encoded sequences.  The sequence in your example is a
valid atom, but is not an encoded-word because it does not contain
exactly 4 '?' characters (RFC 2047 sections 1 & 2).  While the ABNF
could be tweaked somewhat, there is a limit to what can be achieved
with ABNF while allowing enough flexibility to accommodate future and
private-use encodings. 

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: 2822 revised grammar
Date:  Sat, 23 Jul 2005 04:27:24 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 51
Sender: [email-masked]

Bruce Lilly wrote:

> I have no idea where you got that from; it's certainly
> not in 2822.

Of course not, it was an example where your idea that special
rules for CFWS automatically cover FWS, because it's a part
of CFWS, won't fly.

> Last time I checked, "word" wasn't an abbreviation.

Line 719 in 2822:

word            =       atom / quoted-string

Using your idea: "Read <word> as <atom> or <quoted-string>".

> There's at least one open source version of deroff that
> requires just a C compiler to build.

I won't check the state of my last surviving C compiler only
to "deroff" the flood of nroff texts I have to handle, i.e.
exactly two in the last decade, that includes your text ;-)

03:53   03:58 especially if my editor can do this with six
commands in less than 5 minutes.  But what I got won't pass
Bill's parser, and getting rid of the "ew" parts manually
would take some time.

>> for 2822bis we better stick to the 2822 idea, no MIME.

> I guess that's a matter for community discussion and
> consensus. Ignoring MIME doesn't seem rational

IBTD, it's a known way to limit the problem space, it worked
for 2822.

> there needs to be some sort of reconciliation between MIME
[...]
> and 2822

Sure, we also need a 2231bis etc.

> it really needs to be examined carefully.

Yes, starting with the 2822bis part would allow this.  If it
turns out to be "not interesting enough" maybe add MIME later.

E.g. check <cew>, AFAIK you can't have "?=" in an encoded word:
(=?us-ascii?Q?example?=?=)  Something's odd there.  Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: 2822 revised grammar (Was Re: Malformed header - what would you do?)
Date: Fri, 22 Jul 2005 20:15:57 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Fri July 22 2005 19:22, Frank Ellermann wrote:

> | However, where <word> occurs in this standard, it MUST NOT
> | be longer than 255 characters.

I have no idea where you got that from; it's certainly not in 2822.
 
> Does that affect any other <atom> outside of a <word> ?  I'm
> sure that it doesn't.
[...]
> > If you read CFWS as "comments and/or folding whitespace"
> > there's no problem.
> 
> This idea won't work for word = atom / quoted-string.

Last time I checked, "word" wasn't an abbreviation. 
 
>  [formatting]
> > Use abnff or deroff if you don't like it.
> 
> No nroff on my box, sorry.

Neither abnff nor deroff require nroff.  Abnff requires a C compiler
(also gperf, flex, and bison iff you want to modify the source).
There's at least one open source version of deroff that requires
just a C compiler to build.

> > The encoded-word rules are somewhat complex
> 
> Indeed, probably unnecessary for a 2822bis.  It's clear
> why you are interested to have it in a complete parser,
> but for 2822bis we better stick to the 2822 idea, no MIME.

I guess that's a matter for community discussion and consensus.
Ignoring MIME doesn't seem rational, and there needs to be some
sort of reconciliation between MIME (currently using 822 syntax
and differentiating user-defined fields from extension fields)
and 2822 (ABNF and no mention of user-defined fields).  There are
also issues related to 2822 definitions of word and phrase vs.
MIME (2047) rules regarding encoded-words "in" a phrase, especially
as that relates to encoded words and comments, which might be "in"
a phrase (depending on where CFWS appears in ABNF).  I think the
ABNF in the suggested revised grammar represents the intent, but
it really needs to be examined carefully.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Malformed header - what would you do?
Date:  Sat, 23 Jul 2005 01:22:56 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 61
Sender: [email-masked]

Bruce Lilly wrote:

 [replace [FWS] by *WSP where CRLF is forbidden]
> That alone would break the parse grammar, which needs to
> accommodate obs-FWS.

Dubious for USEFOR, because there we don't have any obs-*
with two clearly documented exceptions, both unrelated to
a leading ":" SP [FWS] or a trailing [FWS] CRLF.

We could introduce NFWS ("non-folding white space") if you
want to preserve the forbidden obs-FWS somehow:

     NFWS = *WSP / obs-FWS   ; replace 20 [FWS] by NFWS

OTOH I've already tested that it doesn't work in the case
of the leading ":" SP [FWS], it MUST be ":" SP *WSP, not
one CRLF, let alone many obs-FWS CRLF, exactly zero CRLF.

>> s/where CFWS occurs/where CFWS or FWS occurs/ in 3.2.3

> Not really, because every instance of FWS is necessarily
> CFWS (but not the converse):

Yes, and every instance of <atom> is a <word>.

| However, where <word> occurs in this standard, it MUST NOT
| be longer than 255 characters.

Does that affect any other <atom> outside of a <word> ?  I'm
sure that it doesn't.  Apparently Charles also reads it this
way.

One more experiment:  Every instance of <quoted-string> is
a <word>, same rule as above, does it say something about
any <quoted-string> elsewhere, e.g. in a <local-part> ?

Are all <quoted-string>s limited to 255 characters as soon
as this is the case for all <word>s ?

> If you read CFWS as "comments and/or folding whitespace"
> there's no problem.

This idea won't work for word = atom / quoted-string.

 [formatting]
> Use abnff or deroff if you don't like it.

No nroff on my box, sorry.  It would be easier to dicuss
your idea if you present it in human readable plain text,
with one alternative for each rule (the shortest without
any encoded uew / pew / ... rules).

> The encoded-word rules are somewhat complex

Indeed, probably unnecessary for a 2822bis.  It's clear
why you are interested to have it in a complete parser,
but for 2822bis we better stick to the 2822 idea, no MIME.

                          Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: CFWS: beaten, down, kicked to death, and still being flogged (Was Re: Malformed header - what would you do?)
Date: Fri, 22 Jul 2005 14:57:30 -0400
Cc: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

On Fri July 22 2005 09:48, Charles Lindsey wrote:

> The relevant sentence of 3.2.3 is:
> 
>    However, where CFWS occurs in this standard, it MUST NOT be inserted
>    in such a way that any line of a folded header field is made up
>    entirely of WSP characters and nothing else.
> 
> Observe very carefully what that does NOT say. It does NOT say "where CFWS
> occurs in some header field" (which might well be taken to include the
> particular case "where FWS occurs in some header field", because
> productions of FWS are evidently a subset of productions of CFWS.
> 
> What it DOES say is "where CFWS occurs IN THIS STANDARD" (emphasis added).
> "This standard" is evidently the document RFC 2822. So we must ask "Does
> CFWS occur in the ABNF of <unstructured>", to which the answer is
> evidently "No" (if you take 'CFWS' to mean the four characters C F W S in
> that order).
> 
> Even if you take 'CFWS' to mean "the productions of the ABNF rule 'CFWS'",
> the answer is still "No", because the ABNF of <unstuctured> includes only
> some of the productions of 'CFWS' - the subsetting is the wrong way round
> for the other interpretation in this case.
[...]
> Now of course we all know perfectly well that was not the intention, as
> Appendix B makes clear.

It is an accepted rule of logic that if one starts with a proposition and
by following sound reasoning finds a contradiction, the proposition must be
false.  That is a rule which is fundamental to a significant number of
mathematical proofs, and more than three decades ago was routinely taught
to students. <insert lament about the decline in educational rigor here>

If the proposition "CFWS means those literal characters in the document"
leads to a contradiction ("we all know perfectly well that was not the
intent"), then clearly the proposition is false.  Ditto for "the
productions of the ABNF rule 'CFWS'".

Both propositions are also rather obviously false as it is clearly neither
the literal characters nor an ABNF production which are being inserted in a
field.

The proposition "CFWS means comments, line folding, and/or whitespace" does
not lead to such a contradiction.  That proposition is also consistent with
the above and other usage in 2822, e.g. sect. 4.4 "CFWS were allowed
between..." (that would be a mismatch in number with "were" if "CFWS"
referred to a single literal string or an ABNF production).  Such usage is
referred to as an "abbreviation" and is not uncommon in technical writing;
repeating "comments, line folding, and/or whitespace" at every occurrence
would be as tedious as spelling out Transmission Control Protocol instead of
using the well-known abbreviation TCP.

Granted, the text could be worded better, and when there's a draft successor
we can all pick that sentence to death.

To: [email-masked]
Xref: clerew local.mime:4862
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Malformed header - what would you do?
Date: Fri, 22 Jul 2005 13:48:17 GMT
Lines: 102
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Thu July 21 2005 18:20, Frank Ellermann wrote:
>> 
>> Bruce Lilly wrote:

[Snipped lots of irrelevant arguments where Bruce is destroying Frank's
Red Herrings, or vice versa.]

And proceeding to the _only_ point that has merit, and needs to be taken
seriously.

>> > A conforming generator is still bound by the normative
>> > provisions of section 3, including 3.2.3.
>> 
>> That talks only about "CFWS"..."MUST NOT", but the problem is
>> the FWS in <unstructured>.  If you'd want to fix it in prose:
>> s/where CFWS occurs/where CFWS or FWS occurs/ in 3.2.3

>Not really, because every instance of FWS is necessarily CFWS
>(but not the converse):

No, you cannot wriggle out of it like that. Since the point is important,
we need to consider it very carefully.

Proposition to Consider:

That the following example (in which SP is represented by "_", so it is
clear exactly where SP is present)

    Subject:_Foo_CRLF
    _CRLF
    _Bar_CRLF

is lawful to generate in RFC 2822.

Frank maintains the proposition is True. I think he is right. Bruce claims
it is False.

Evidently, the Proposition reduces to showing that

    _Foo_CRLF
    _CRLF
    _Bar_CRLF

is a legally generateable <unstructured>, where

    unstructured    =       *([FWS] utext) [FWS]

It is evidently legal to generate according to that ABNF, so we need to
examine whether or not it is forbidden by 3.2.3.

The relevant sentence of 3.2.3 is:

   However, where CFWS occurs in this standard, it MUST NOT be inserted
   in such a way that any line of a folded header field is made up
   entirely of WSP characters and nothing else.

Observe very carefully what that does NOT say. It does NOT say "where CFWS
occurs in some header field" (which might well be taken to include the
particular case "where FWS occurs in some header field", because
productions of FWS are evidently a subset of productions of CFWS.

What it DOES say is "where CFWS occurs IN THIS STANDARD" (emphasis added).
"This standard" is evidently the document RFC 2822. So we must ask "Does
CFWS occur in the ABNF of <unstructured>", to which the answer is
evidently "No" (if you take 'CFWS' to mean the four characters C F W S in
that order).

Even if you take 'CFWS' to mean "the productions of the ABNF rule 'CFWS'",
the answer is still "No", because the ABNF of <unstuctured> includes only
some of the productions of 'CFWS' - the subsetting is the wrong way round
for the other interpretation in this case.

Hence the ABNF rule for <unstructured> is NOT one of the places where
"CFWS occurs in this standard", and hence the 3.2.3. exception does not
apply to it.

Q.E.D.

Now of course we all know perfectly well that was not the intention, as
Appendix B makes clear. It is a BUG. Nevertheless, that is what RFC 2822
says, and we have to live with it. Clearly, an Erratum sent to the Drafts
Editor would be in order.

Fortunately, the corresponding text in USEFOR does not contain the Bug
(the text in USEFOR is different from that in RFC 2822 because it is
trying to make a somewhat wider prohibition).

[snipped discussion of the remainder of Frank's Red Herrings.]

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Malformed header - what would you do?
Date: Thu, 21 Jul 2005 21:40:36 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Thu July 21 2005 18:20, Frank Ellermann wrote:
> 
> Bruce Lilly wrote:

> > unstructured    =       *([FWS] utext) [FWS]
> 
> > The trailing [FWS] in the definition of unstructured provides
> > for trailing whitespace on such a single-line unstructured
> > field.
> 
> Sure, we're talking about _replacing_ this one [FWS] here (= in
> a future 2822bis) and all similar USEFOR [FWS] by *WSP, because
> [1*WSP] is the same as *WSP.

That alone would break the parse grammar, which needs to accommodate
obs-FWS.
 
> The same exercise for the obs-FWS = 1*WSP *(CRLF 1*WSP) case
> has the same result:  *WSP CRLF is what we really want if we
> don't like any CRLF 1*WSP CRLF at the end of an <unstructured>.

But that *is* what is desired for obs-FWS (see prose in 4.2).
 
> > This was discussed more than a year ago:
> > http://article.gmane.org/gmane.ietf.rfc822/9919
> 
> Very vaguely I recall this, but I probably didn't look at the
> details of your <unstructured> at this time.  What you have is:
> 
> | unstructured    =       [[FWS] *(1*utext FWS) 1*utext] *WSP
> | unstructured    =       [1*utext] *(FWS 1*utext) *WSP
> 
> (skipping the two "uew" incarnations adressing encoded words)
> 
> Obviously we agree on ... *WSP replacing the trailing [FWS].
> And you guarantee that any FWS is followed by 1*utext, good, I
> missed that part.
> 
> > do you believe there is a problem with the "end" rule and its
> > use in the revised grammar proposed?
> 
> All I can say without digging deep into your idea:  I know that
> it's in theory possible to fix "CFWS-separated references" into
> "FWS-separated references allowing comments".  So it should be
> possible to get rid of all unwanted cases of CRLF 1*WSP CRLF
> 
> And apparently you've avoided the [CFWS] [CFWS] problem in your
> version of the references, by msg-id = [CFWS] stuff  instead of
> msg-id = [CFWS] stuff [CFWS]  moving the critical right [CFWS]
> to the end of <references> or <message-id> in the form of <end>
> 
> And end = [[FWS] comment] *WSP avoids the CRLF 1*WSP CRLF trap.
> 
> Hm, I miss the case of more than one <comment> near the <end>,
> how about end = *( [FWS] comment ) *WSP ?

Good point; that will be in the next revision.

> > A conforming generator is still bound by the normative
> > provisions of section 3, including 3.2.3.
> 
> That talks only about "CFWS"..."MUST NOT", but the problem is
> the FWS in <unstructured>.  If you'd want to fix it in prose:
> s/where CFWS occurs/where CFWS or FWS occurs/ in 3.2.3

Not really, because every instance of FWS is necessarily CFWS
(but not the converse):

CFWS            =       *([FWS] comment) (([FWS] comment) / FWS)

(zero repetitions of the first group, and taking the last alternative
of the second group yields FWS. Q.E.D.)

If you read CFWS as "comments and/or folding whitespace" there's
no problem.
 
> > And in fact is what the section 3.2.3 restriction does.
> 
> Sorry, that's what it _should_ do, but it's only a near miss.

No, because FWS is subsumed by CFWS (see above).
 
> > That's why a revised grammar was produced and discussed here.
> 
> Fine, how about a more readable version without nroff, and for
> starters without the "uew" etc. encoded words ?

The nroff facilitates checking and formatting (abnff), and permits
comments which are not part of the ABNF (including ABNF comments).
Use abnff or deroff if you don't like it.  The encoded-word grammar
is a practical necessity for a message parser.  You can of course ignore
it if you don't care about MIME conformance (egrep -v "ew|WSWS|WSCF"),
as non-MIME alternatives are provided.  The encoded-word rules are
somewhat complex; having gone through the trouble to figure out the ABNF,
I'm not going to throw that work away.  Whether it is in the 2822
successor or in a subsequent MIME revision ("updates WXYZ", where WXYZ
is the 2822 successor) is academic at this point.

> > Note especially the second line of the "To:" field.  It
> > starts with two space characters.  (Note that "__" represent
> > blank spaces.)
> 
> Sure, what do you think why I mentioned it ?  But "To:" has no
> [FWS] problem, it's covered by the "CFWS"..."MUST NOT".  Only
> <unstructured> has this problem and there's no example with a
> "Subject:" and "__" in the appendix.

To    : Mary Smith
__
          <[email-masked]>

contains folding whitespace but no comments.  It's covered by 3.2.3
because FWS is CFWS.

> Besides you'd shoot me if I'd claim that an example is in any
> way "normative" without explicitly saying so.

I abhor unnecessary violence.
 
> > the 3.2.3 rule applies to all fields.
> 
> To all fields with CFWS, e,g. "To:", but not "X-SPAM-foobar:".

You're going out on a limb because you don't know what the syntax
definition of X-Spam-Foobar is.  Anyway, a field with FWS has CFWS
(but not necessarily comments).

> > If RFC 2822 is used as a normative reference (possibly with
> > implementation notes where appropriate), there is no problem,
> > as the 2822 rules, including 3.2.3, apply.
> 
> Argh... yes, for header fields ending with [CFWS] CRLF.  But we
> have about ten NetNews header fields ending with [FWS] CRLF.

So add a note affirming that 2822 3.2.3 applies.  Or don't.  The
2822 successor will probably be published before the USEFOR documents
are in a state where there's any hope of passing IESG muster, so the
revised grammar can be used, eliminating any chance of ambiguity.
 
> Unlike 2822 USEFOR has at least a prose MUST covering [FWS].
> But saying [FWS] when the only allowed case is *WSP is bogus.

You're forgetting the parse grammar again.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Malformed header - what would you do?
Date:  Fri, 22 Jul 2005 00:20:28 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 140
Sender: [email-masked]

Bruce Lilly wrote:

> it is perfectly valid for a single-line unstructured field to
> end in one or more whitespace characters, which is what the
> 1*WSP in the definition of FWS allows.

> unstructured    =       *([FWS] utext) [FWS]

> The trailing [FWS] in the definition of unstructured provides
> for trailing whitespace on such a single-line unstructured
> field.

Sure, we're talking about _replacing_ this one [FWS] here (= in
a future 2822bis) and all similar USEFOR [FWS] by *WSP, because
[1*WSP] is the same as *WSP.

  FWS = ([*WSP CRLF] 1*WSP) /   ; Folding white space
        obs-FWS

In [FWS] CRLF we don't want the *WSP CRLF 1*WSP CRLF case, in
other words we don't want the [*WSP CRLF] part of the FWS, that
leaves us with 1*WSP instead of FWS, putting it all together it
is [1*WSP] CRLF, or in canonical form *WSP CRLF.

The same exercise for the obs-FWS = 1*WSP *(CRLF 1*WSP) case
has the same result:  *WSP CRLF is what we really want if we
don't like any CRLF 1*WSP CRLF at the end of an <unstructured>.

Including the X-SPAM-example or similar USEFOR header fields.

> This was discussed more than a year ago:
> http://article.gmane.org/gmane.ietf.rfc822/9919

Very vaguely I recall this, but I probably didn't look at the
details of your <unstructured> at this time.  What you have is:

| unstructured    =       [[FWS] *(1*utext FWS) 1*utext] *WSP
| unstructured    =       [1*utext] *(FWS 1*utext) *WSP

(skipping the two "uew" incarnations adressing encoded words)

Obviously we agree on ... *WSP replacing the trailing [FWS].
And you guarantee that any FWS is followed by 1*utext, good, I
missed that part.

> do you believe there is a problem with the "end" rule and its
> use in the revised grammar proposed?

All I can say without digging deep into your idea:  I know that
it's in theory possible to fix "CFWS-separated references" into
"FWS-separated references allowing comments".  So it should be
possible to get rid of all unwanted cases of CRLF 1*WSP CRLF

And apparently you've avoided the [CFWS] [CFWS] problem in your
version of the references, by msg-id = [CFWS] stuff  instead of
msg-id = [CFWS] stuff [CFWS]  moving the critical right [CFWS]
to the end of <references> or <message-id> in the form of <end>

And end = [[FWS] comment] *WSP avoids the CRLF 1*WSP CRLF trap.

Hm, I miss the case of more than one <comment> near the <end>,
how about end = *( [FWS] comment ) *WSP ?  Example:

Message-ID: <[email-masked]> (one) (two)

> A conforming generator is still bound by the normative
> provisions of section 3, including 3.2.3.

That talks only about "CFWS"..."MUST NOT", but the problem is
the FWS in <unstructured>.  If you'd want to fix it in prose:
s/where CFWS occurs/where CFWS or FWS occurs/ in 3.2.3

Actually we're not worried about completely empty comments, so
s/where CFWS occurs/where FWS occurs (directly or indirectly)/
in 3.2.3 would also fix it.

> And in fact is what the section 3.2.3 restriction does.

Sorry, that's what it _should_ do, but it's only a near miss.

> FWS includes obs-FWS and is necessary for parsing.

Is that something your grammar does ?  Stupid question, yes,
you have a separate <obs-unstructured> which can end with an
<obs-FWS>.  Nice.

> That's why a revised grammar was produced and discussed here.

Fine, how about a more readable version without nroff, and for
starters without the "uew" etc. encoded words ?

> Note especially the second line of the "To:" field.  It
> starts with two space characters.  (Note that "__" represent
> blank spaces.)

Sure, what do you think why I mentioned it ?  But "To:" has no
[FWS] problem, it's covered by the "CFWS"..."MUST NOT".  Only
<unstructured> has this problem and there's no example with a
"Subject:" and "__" in the appendix.

Besides you'd shoot me if I'd claim that an example is in any
way "normative" without explicitly saying so.

> the 3.2.3 rule applies to all fields.

To all fields with CFWS, e,g. "To:", but not "X-SPAM-foobar:".

> If RFC 2822 is used as a normative reference (possibly with
> implementation notes where appropriate), there is no problem,
> as the 2822 rules, including 3.2.3, apply.

Argh... yes, for header fields ending with [CFWS] CRLF.  But we
have about ten NetNews header fields ending with [FWS] CRLF.

And we have the same ten starting with "name:" SP [FWS], which
should be be "name:" SP *WSP to avoid a CRLF before the first
no-WSP character of the header field body.

Unlike 2822 USEFOR has at least a prose MUST covering [FWS].
But saying [FWS] when the only allowed case is *WSP is bogus.

> we know what the intent was, as it has been discussed; at
> minimum: http://www.imc.org/ietf-usefor/2003/May/0052.html

I didn't read this list in May 2003.  Or any other IETF list.

The 2822-author talks about <mailbox-list> => structured =>
CFWS => covered by the MUST NOT in 3.2.3, nobody doubted that.

>> Now that it's finally clear that the next step for RfC 2822
>> will be historic

> ???

Okay, that was a bit harsh for _one_ s/[FWS]/*WSP/.  OTOH you
showed that it would kill the trailing obs-FWS, which is wrong.
A minimal erratum could be to fix the 3.2.3 prose (see above).

                        Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Malformed header - what would you do?
Date: Thu, 21 Jul 2005 10:32:36 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Thu July 21 2005 05:02, Frank Ellermann wrote:
> 
> Bruce Lilly wrote:
> 
> >> Where do you see this in 2822 ?
> 
> >| 2.2. Header Fields
> [...]
> >| A field body may be composed of any US-ASCII characters,
> >| except for CR and LF.  However, a field body may contain
> >| CRLF when used in header "folding" and  "unfolding" as
> >| described in section 2.2.3.  All field bodies MUST conform
> >| to the syntax described in sections 3 and 4
> 
> That doesn't talk about the trailing [FWS] in <unstructured>.

Why do you think it should? (hint: the FWS rule also includes
obs-FWS for parsing)

FWS             =       ([*WSP CRLF] 1*WSP) /   ; Folding white space
                        obs-FWS

Note also that it is perfectly valid for a single-line unstructured
field to end in one or more whitespace characters, which is what the
1*WSP in the definition of FWS allows.

unstructured    =       *([FWS] utext) [FWS]

The trailing [FWS] in the definition of unstructured provides for
trailing whitespace on such a single-line unstructured field.

This was discussed more than a year ago:
http://article.gmane.org/gmane.ietf.rfc822/9919
(oddly absent from the IMC web-based archive)
Do you believe that there is something new to be said w.r.t. revising
the 2822 ABNF?  In particular, do you believe there is a problem with
the "end" rule and its use in the revised grammar proposed?
 
> It's also not straight forward with the CR and LF as found in:
> 
> | obs-qp          =       "\" (%d0-127)
> | obs-text        =       *LF *CR *(obs-char *LF *CR)

2822 obs- rules are for parsing; the discussion here is about a
field which was recently generated.
 
> >| 3.6.8. Optional fields
> [...]
> >| They MUST conform to the syntax of an optional-field.
> >| This is a field name, made up of the printable US-ASCII
> >| characters except SP and colon, followed by a colon,
> >| followed by any text which conforms to unstructured.
> 
> That allows the trailing [FWS] in the X-SPAM header field (or
> whatever its name was) reported by the OP.

For parsing, of course, since 822 and its predecessors did not have
a rule against whitespace-only continuation lines (in turn because a
certain company's "programmers" hadn't been around to botch parsing).
A conforming generator is still bound by the normative provisions of
section 3, including 3.2.3.
 
> >| Appendix B:
> >| 13. Folding continuation lines cannot contain only white
> >| space.*
> 
> That's a list of _claimed_ differences from STD 11:

And in fact is what the section 3.2.3 restriction does.
 
> | Items marked with an asterisk (*) below are items which
> | appear in section 4 of this document and therefore can no
> | longer be generated.
> 
> Therefore point 13 is talking about obs-FWS, not trailing FWS:

Sort of.  It says that while whitespace-only continuation lines
used to be allowed, and conforming parsers must still handle them
properly, they "can no longer be generated" by conforming
generators.
 
> | 4.2. Obsolete folding white space
> |
> | In the obsolete syntax, any amount of folding white space MAY
> | be inserted where the obs-FWS rule is allowed.  This creates
> | the possibility of having two consecutive "folds" in a line,
> | and therefore the possibility that a line which makes up a
> | folded header field could be composed entirely of white
> | space.
> |
> |   obs-FWS         =       1*WSP *(CRLF 1*WSP)
> 
> The trailing [FWS] in <unstructured> is erroneous, in RfC 2822

No, FWS includes obs-FWS and is necessary for parsing.  Now I agree
that it's not an ideal way to indicate parse vs. generate grammars,
and that some things which could be specified in ABNF aren't.  That's
why a revised grammar was produced and discussed here.

> and in its numerous USEFOR children, starting with the USEFOR
> incarnation of <unstructured>.  Apparently RFC 2822 didn't want
> it this way, but it forgot to say so explicitly.

It sounds like you;re confusing the parse grammar and the generate
grammar.  Section 3 (including 3.2.3) applies to generation.

> And we're _copying_ this bug to USEFOR, I can't believe it. :-(

There's no bug (at least not in that area).
 
> > The Appendix B text clearly identifies the problem which is
> > the subject of the current discussion.
> 
> It addressses only obs-FWS as indicated by the (*).

The reason it's labeled obs- is precisely because it is no longer
permitted for generation (but is required to be handled by parsers).

> Example 
> A.6.3 illustrates the [CFWS] rule, the trailing [FWS] somehow
> escaped in the wild - hard to find it, obs-FWS hides it almost
> everywhere,

   Note especially the second line of the "To:" field.  It starts with
   two space characters.  (Note that "__" represent blank spaces.)

> it only affects <unstructured> in RfC 2822.

No, the 3.2.3 rule applies to all fields.

> But in  
> USEFOR it hits about ten header fields ending with [FWS] CRLF.

If RFC 2822 is used as a normative reference (possibly with implementation
notes where appropriate), there is no problem, as the 2822 rules, including
3.2.3, apply.  Of course if instead of citing 2822 as a normative reference
and using it as such, the approach of rewriting syntax is taken, then the
issue needs to be addressed anew.  That is precisely why the first paragraph
of section 1.6 of the usefor-usefor draft points out the problems that arise
when cite-by-reference is not used.
 
> > The 2822 grammar as written does not correspond to the prose
> > specifying no continuation lines containing only whitespace.
> 
> The prose is misleading.  We are only guessing the intention
> based on an unrelated example, an unrelated point in the list
> of differences, and a related MUST NOT valid only for the CFWS.

No, we know what the intent was, as it has been discussed; at minimum:
http://www.imc.org/ietf-usefor/2003/May/0052.html
 
> No big issue for RfC 2822, it only affects X-header fields and
> other unstructured header fields, but for USEFOR it's far more
> serious.

No.  Believe it or not, the To field is not unstructured and does not begin
with "X-".  That *is* the A.6.3 example.
 
> >> so far nobody supported a s/[FWS]/*WSP/ cleanup in USEFOR.
> 
> > USEFOR has no jurisdiction over RFC 2822.
> 
> Now that it's finally clear that the next step for RfC 2822 
> will be historic

???

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Malformed header - what would you do?
Date:  Thu, 21 Jul 2005 11:02:15 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 87
Sender: [email-masked]

Bruce Lilly wrote:

>> Where do you see this in 2822 ?

>| 2.2. Header Fields
[...]
>| A field body may be composed of any US-ASCII characters,
>| except for CR and LF.  However, a field body may contain
>| CRLF when used in header "folding" and  "unfolding" as
>| described in section 2.2.3.  All field bodies MUST conform
>| to the syntax described in sections 3 and 4

That doesn't talk about the trailing [FWS] in <unstructured>.
It's also not straight forward with the CR and LF as found in:

| obs-qp          =       "\" (%d0-127)
| obs-text        =       *LF *CR *(obs-char *LF *CR)

>| 3.6.8. Optional fields
[...]
>| They MUST conform to the syntax of an optional-field.
>| This is a field name, made up of the printable US-ASCII
>| characters except SP and colon, followed by a colon,
>| followed by any text which conforms to unstructured.

That allows the trailing [FWS] in the X-SPAM header field (or
whatever its name was) reported by the OP.

>| Appendix B:
>| 13. Folding continuation lines cannot contain only white
>| space.*

That's a list of _claimed_ differences from STD 11:

| Items marked with an asterisk (*) below are items which
| appear in section 4 of this document and therefore can no
| longer be generated.

Therefore point 13 is talking about obs-FWS, not trailing FWS:

| 4.2. Obsolete folding white space
|
| In the obsolete syntax, any amount of folding white space MAY
| be inserted where the obs-FWS rule is allowed.  This creates
| the possibility of having two consecutive "folds" in a line,
| and therefore the possibility that a line which makes up a
| folded header field could be composed entirely of white
| space.
|
|   obs-FWS         =       1*WSP *(CRLF 1*WSP)

The trailing [FWS] in <unstructured> is erroneous, in RfC 2822
and in its numerous USEFOR children, starting with the USEFOR
incarnation of <unstructured>.  Apparently RFC 2822 didn't want
it this way, but it forgot to say so explicitly.

And we're _copying_ this bug to USEFOR, I can't believe it. :-(

> The Appendix B text clearly identifies the problem which is
> the subject of the current discussion.

It addressses only obs-FWS as indicated by the (*).  Example
A.6.3 illustrates the [CFWS] rule, the trailing [FWS] somehow
escaped in the wild - hard to find it, obs-FWS hides it almost
everywhere, it only affects <unstructured> in RfC 2822.  But in
USEFOR it hits about ten header fields ending with [FWS] CRLF.

> The 2822 grammar as written does not correspond to the prose
> specifying no continuation lines containing only whitespace.

The prose is misleading.  We are only guessing the intention
based on an unrelated example, an unrelated point in the list
of differences, and a related MUST NOT valid only for the CFWS.

No big issue for RfC 2822, it only affects X-header fields and
other unstructured header fields, but for USEFOR it's far more
serious.

>> so far nobody supported a s/[FWS]/*WSP/ cleanup in USEFOR.

> USEFOR has no jurisdiction over RFC 2822.

Now that it's finally clear that the next step for RfC 2822 
will be historic we can freely fix all bugs we find in USEFOR.

                        Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Malformed header - what would you do?
Date: Wed, 20 Jul 2005 08:37:22 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Tue July 19 2005 20:04, Frank Ellermann wrote:
> 
> Bruce Lilly wrote:
> 
> > extension/user-defined (unlike 822, 2822 does not officially
> > distinguish the two distinct types) fields must conform to
> > the unstructured field restrictions on syntax, and that would
> > include the restriction on whitespace-only header lines.
> 
> Where do you see this in 2822 ?

2.2. Header Fields

   Header fields are lines composed of a field name, followed by a colon
   (":"), followed by a field body, and terminated by CRLF.  A field
   name MUST be composed of printable US-ASCII characters (i.e.,
   characters that have values between 33 and 126, inclusive), except
   colon.  A field body may be composed of any US-ASCII characters,
   except for CR and LF.  However, a field body may contain CRLF when
   used in header "folding" and  "unfolding" as described in section
   2.2.3.  All field bodies MUST conform to the syntax described in
   sections 3 and 4 of this standard.
[...]
3.6.8. Optional fields

   Fields may appear in messages that are otherwise unspecified in this
   standard.  They MUST conform to the syntax of an optional-field.
   This is a field name, made up of the printable US-ASCII characters
   except SP and colon, followed by a colon, followed by any text which
   conforms to unstructured.

   The field names of any optional-field MUST NOT be identical to any
   field name specified elsewhere in this standard.

optional-field  =       field-name ":" unstructured CRLF

field-name      =       1*ftext

ftext           =       %d33-57 /               ; Any character except
                        %d59-126                ;  controls, SP, and
                                                ;  ":".

   For the purposes of this standard, any optional field is
   uninterpreted.

And in Appendix B:

   13. Folding continuation lines cannot contain only white space.*

The Appendix B text clearly identifies the problem which is the subject
of the current discussion.

Note that CFWS is:
CFWS            =       *([FWS] comment) (([FWS] comment) / FWS)
and need not contain a comment; it can consist of FWS alone.

> I found...  
> 
> | unstructured    =       *([FWS] utext) [FWS]
> 
> ...and that allows a trailing FWS.  It should be *WSP to avoid
> a whitespace-only header line.  The "MUST NOT" for CFWS isn't
> good enough, <unstructured> has no CFWS.

The 2822 grammar as written does not correspond to the prose specifying
no continuation lines containing only whitespace.  The "simplified"
ABNF discussed here more than a year ago does prohibit whitespace-only
continuation lines.

> >>> Is the software adding the 'X-SPAM-STATUS' line wrong
>  
> > Does the software claim conformance with RFC 2822?  If so, yes.
> 
> I'm not convinced.  But so far nobody supported a s/[FWS]/*WSP/
> cleanup in USEFOR.

USEFOR has no jurisdiction over RFC 2822.

To: [email-masked]
Xref: clerew local.mime:4853
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Malformed header - what would you do?
Date: Wed, 20 Jul 2005 10:58:22 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Bruce Lilly wrote:

>> extension/user-defined (unlike 822, 2822 does not officially
>> distinguish the two distinct types) fields must conform to
>> the unstructured field restrictions on syntax, and that would
>> include the restriction on whitespace-only header lines.

>Where do you see this in 2822 ?  I found... 

>| unstructured    =       *([FWS] utext) [FWS]

I think Frank has a point here. In Section 3.2.3 of RFC 2822, it says:

   There are several places in this standard where comments and FWS may
   be freely inserted.  To accommodate that syntax, an additional token
   for "CFWS" is defined for places where comments and/or FWS can occur.
   However, where CFWS occurs in this standard, it MUST NOT be inserted
   in such a way that any line of a folded header field is made up
   entirely of WSP characters and nothing else.

But the prohibition against having WSP-only header lines only seems to
apply "where CFWS occurs in this standard", and not "where FWS occurs".

So it seems that you are allowed to have an empty (WSP-only) line in the
middle of a Subject header field.

That seems like a bug in RFC 2822 to me.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 20 Jul 2005 08:19:04 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

I wrote:

> You might want to forbid blocks from straddling line breaks, so that
> a non-validating decoder can eliminate a compare-and-branch operation
> from the processing of every byte.

That's not as easy as I thought, because the last block has no choice
but to have a line break appear before the byte counter reaches 84.

AMC

Date: Wed, 20 Jul 2005 07:23:11 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> > I think we must have all had similar ideas in mind.
> 
> FYI, I based the numbers above on:
> o output consists of an addition code (1 octet) followed by a block of
>   modified output octets (each octet is input value plus addition code,
>   modulo 256).

Well then, not so similar after all.  I'd never heard of nor thought of
that scheme.

>   For N=3, M=84, expansion is 1.39%
>   Expansion is independent of input octet sequence.

For typical (roughly random) input, the expansion of the
variable-length-block scheme is half that, and the worst case is only
slightly greater than 1.39%, at 14/972 = 1.44%.  (The 1.63% worst-case
quoted in the source code includes two filler bytes per line for
protecting leading double-hyphen and trailing whitespace, and would not
be a fair comparison with the 1.39% figure.)

>   (of course the decoder needs to keep track of the data count up to
>   the buffer length and needs to save and apply the offset code to
>   each octet)

For a non-validating decoder for the variable-length-block scheme, all
the original bytes that weren't 0, 10, or 13 can be copied blindly
without being inspected.  For typical (roughly random) input that's
98.2% of the decoder's input bytes, most of them in runs of dozens of
consecutive bytes, so this scheme might be faster or consume less power,
especially if it can take advantage of block-move instructions.

The addition-code scheme has the advantage of being conceptually
simpler and easier to code.  You might want to forbid blocks from
straddling line breaks, so that a non-validating decoder can eliminate a
compare-and-branch operation from the processing of every byte.

AMC

Date: Wed, 20 Jul 2005 06:23:48 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> > A long block is a header byte k in 1..9 followed
> > by 80k data bytes.
> 
> Do you mean 80 * k?

Yes.

> 80k bytes of data would exceed the 998 octet line length limit.

:)  I'll clarify that in the next version.

AMC

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Date: Tue, 19 Jul 2005 22:15:20 -0400
Sender: [email-masked]

On Sun July 17 2005 05:22, Adam M. Costello wrote:
> 
> Bruce Lilly <[email-masked]> wrote:
> 
> > On Fri July 1 2005 18:33, [email-masked] wrote:
> >
> > > It is relatively easy to design a scheme that limits the
> > > overhead to 1-2% no matter what the input.
> > 
> > Maybe, depending on the constraints.  The minimum constraints on the
> > output are:
> > o CRLF only for line endings, no lone CR or lone LF
> > o no NUL
> > o line length <= 998 octets
> > (for that is the definition of 8bit).  The input of course is an
> > unconstrained sequence of octets.
> > 
> > About 1.4% expansion should be possible with only those constraints,
> > a fairly simple algorithm (faster decode than encode), and moderate
> > encoder memory requirements (an 84 octet input buffer).
> 
> I think we must have all had similar ideas in mind.  I finally got
> around to specifying (informally) and implementing (as a demo) my idea,
> see below.

FYI, I based the numbers above on:
o elimination of N=3 octets (0x00, 0x0A, 0x0D) from the output sequence
  leaves 253=(256-N) valid output codes
o output consists of an addition code (1 octet) followed by a block of
  modified output octets (each octet is input value plus addition code,
  modulo 256).  The addition code is selected for a block such that the
  taboo output codes are not produced.  Obviously, addition codes cannot
  assume taboo values (in this case, 0, 10, and 13 (decimal))
o a buffer (block) length of M=(256-N)/N is the maximum length that is
  guaranteed to work for N and sets a lower bound on the expansion factor.
  At N=3, M=84
o expansion is (M+1)/M*1000/998, which is broken down as M+1 output
  octets per M input octets and CRLF insertion after 998 output octets
  (decoder ignores CR, LF). For N=3, M=84, expansion is 1.39% (n=4,
  M=62 yields expansion of 1.82%, larger N with this method yields >2%
  expansion).  Expansion is independent of input octet sequence.
o to simplify decoding, block length is fixed (not variable) and requires
  no buffering (of course the decoder needs to keep track of the data
  count up to the buffer length and needs to save and apply the offset
  code to each octet)

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Malformed header - what would you do?
Date:  Wed, 20 Jul 2005 02:04:20 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 23
Sender: [email-masked]

Bruce Lilly wrote:

> extension/user-defined (unlike 822, 2822 does not officially
> distinguish the two distinct types) fields must conform to
> the unstructured field restrictions on syntax, and that would
> include the restriction on whitespace-only header lines.

Where do you see this in 2822 ?  I found... 

| unstructured    =       *([FWS] utext) [FWS]

...and that allows a trailing FWS.  It should be *WSP to avoid
a whitespace-only header line.  The "MUST NOT" for CFWS isn't
good enough, <unstructured> has no CFWS.

>>> Is the software adding the 'X-SPAM-STATUS' line wrong
 
> Does the software claim conformance with RFC 2822?  If so, yes.

I'm not convinced.  But so far nobody supported a s/[FWS]/*WSP/
cleanup in USEFOR.
                    Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: IETF RFC-822 list <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Date: Tue, 19 Jul 2005 19:12:49 -0400
Sender: [email-masked]

On Sun July 17 2005 05:22, Adam M. Costello wrote:

> A long block is a header byte k in 1..9 followed
> by 80k data bytes.

Do you mean 80 * k?  80k bytes of data would exceed the 998 octet
line length limit.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Malformed header - what would you do?
Date: Tue, 19 Jul 2005 18:54:51 -0400
Cc: Frank Ellermann <[email-masked]>
Sender: [email-masked]

On Fri July 15 2005 20:00, Frank Ellermann wrote:
> 
> Paul Smith wrote:

> > Is the software adding the 'X-SPAM-STATUS' line wrong
> 
> Not really, 2822 says about CFWS:
> 
> | However, where CFWS occurs in this standard, it MUST NOT
> | be inserted in such a way that any line of a folded header
> | field is made up entirely of WSP characters and nothing
> | else.
> 
> That's only about 2822-header fields, and not about the
> obscure X-SPAM-STATUS.

2822 covers the Internet Message Format, and specifically
notes that extension/user-defined (unlike 822, 2822 does not
officially distinguish the two distinct types) fields must
conform to the unstructured field restrictions on syntax, and
that would include the restriction on whitespace-only header
lines.

So, returning to the original question:
> > Is the software adding the 'X-SPAM-STATUS' line wrong

Does the software claim conformance with RFC 2822?  If so, yes.

Date: Sun, 17 Jul 2005 09:22:56 +0000
From: "Adam M. Costello" <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Bruce Lilly <[email-masked]> wrote:

> On Fri July 1 2005 18:33, [email-masked] wrote:
>
> > It is relatively easy to design a scheme that limits the
> > overhead to 1-2% no matter what the input.
> 
> Maybe, depending on the constraints.  The minimum constraints on the
> output are:
> o CRLF only for line endings, no lone CR or lone LF
> o no NUL
> o line length <= 998 octets
> (for that is the definition of 8bit).  The input of course is an
> unconstrained sequence of octets.
> 
> About 1.4% expansion should be possible with only those constraints,
> a fairly simple algorithm (faster decode than encode), and moderate
> encoder memory requirements (an 84 octet input buffer).

I think we must have all had similar ideas in mind.  I finally got
around to specifying (informally) and implementing (as a demo) my idea,
see below.

AMC

/*
base253.c 0.0.0 (2005-Jul-17-Sun)
Adam M. Costello
http://www.nicemice.net/amc/

Base253 is a proposed MIME content-transfer-encoding that transforms an
arbitrary sequence of bytes (octets) into 8bit data, which is defined in
RFC-2045 as lines ending with the two bytes CR LF (13 10) containing at
most 998 other bytes, none of which are NUL (0), CR (13), or LF (10).

Base253 is designed to be compact (for both typical input and worst-case
input), simple to encode, even simpler to decode, and simple to explain
and understand.

This implementation expands the data by a theoretical maximum ratio of
935/920 = 1.63% (if I have correctly imagined the worst-case input), but
in tests on JPEG, MP3, and gzip files the actual expansion was 0.65% to
0.67%.

This implementation is a demo intended to illustrate the algorithms and
enable measurements of the output.  A practical implementation would do
more error checking and would probably have a different interface.

Specification:

In the encoded stream, each line is a sequence of long blocks, short
blocks, and filler bytes, in any order, followed by the line break.
Decoders ignore filler bytes and line breaks, and view an encoded
stream as a single sequence of blocks.  A filler byte is byte 15, which
can appear in the middle of a line, but is intended to be inserted
at the end of lines that would otherwise end with whitespace (which
some MTAs might strip off), and at the beginning of lines that would
otherwise begin with two hyphens (which might accidentally match MIME
part boundaries).  A long block is a header byte k in 1..9 followed
by 80k data bytes.  A short block is a header byte 16+80i+n (where i
is in 0..2, and n is in 0..79) followed by n data bytes.  Every block
(long or short) except the very first block has an additional data byte
implicitly prepended to its data if the previous block was a short
block.  The implicit byte is 0, 10, or 13, depending on whether the i
value of the previous block was 0, 1, or 2, respectively.  The meaning
of the encoded stream is the sequence of all data bytes (both explicit
and implicit).

Rationale:

A block cannot span a line break.  If it could, then either it would
have to include the line break as literal data (in which case the
common practice of storing messages using local line-break conventions
would corrupt the data or require decoders to convert back to the wire
representation) or else the line breaks would have to be ignored even
inside blocks (in which case decoders would have to inspect data bytes,
whereas non-validating decoders of the format described above can
blindly copy data bytes without inspecting them).

The bytes 11, 12, and 14 are not used except as data bytes.  The range
of n or k could be increased by using these bytes values (and either
range could be further increased at the expense of the other).  The
ideal allocation might be 0..81 for n and 1..6 for k (plus 1 filler byte
and the 3 prohibited bytes), but any such changes would make the ranges
non-contiguous, which would complicate the encoding.  The improvement in
compactness would be extremely slight, so it doesn't seem worth it.

An unintended fortuitous consequence of the formula 16+80i+n is that the
constants 16 and 80 are both divisible by 16, which makes it easier to
debug the output by looking at a hex dump.

This is ANSI C (C89) code.

*/

#include <assert.h>
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* non-validating non-error-checking decoder */

static void decode(FILE *in, FILE *out)
{
  static const int banned[3] = {0,10,13};
  int previ, byte, nplus80i, i, n;
  unsigned char buf[80*9];

  /*

  The decoder doesn't actually need any lookahead at all, but it's
  simpler and more efficient to use a buffer to copy whole blocks rather
  than to copy a byte at a time.

  previ >= 0 means banned[previ] will be inserted before the next block
  (if there is one); previ < 0 means nothing will be inserted.

  n >= 0 means the current block contains n bytes; n < 0 means there is 
  no current block, just a line break, filler byte, or invalid byte.    

  Since this is a non-validating decoder, we are free to treat invalid
  bytes any way we like.  It's convenient to treat bytes 11, 12, 14,
  lone 10, and lone 13 like filler bytes, and to treat byte 0 like the
  header of a zero-length long block.

  */

  for (previ = -1;;) {
    byte = getc(in);
    if (byte == EOF) break;
    nplus80i = byte - 16;

    if (nplus80i >= 0) { /* short block */
      i = nplus80i / 80;
      n = nplus80i % 80;
    }
    else if (byte <= 9) { /* long block */
      i = -1;
      n = 80 * byte;
    }
    else continue; /* ignore filler or line break */

    if (previ >= 0) putc(banned[previ], out);
    fread(buf,n,1,in);
    fwrite(buf,n,1,out);
    previ = i;
  }
}

/* non-error-checking encoder */

static void encode(FILE *in, FILE *out)
{
  enum { linemax = 997 };
    /* maximum number of bytes in a line, excluding the line   */
    /* ending, which can be up to three bytes (filler, CR, LF) */

  int roomleft, m, byte, limbo, lastout, i, n, header;

  unsigned char buf[80*9];
    /* 80*9 bytes is the largest lookahead that is useful.  An    */
    /* encoder could use as little as 80 bytes of lookahead and   */
    /* still produce valid streams that are not quite as compact. */

  roomleft = linemax;  /* bytes allowed to be added to the current line */
  m = 0;               /* bytes in buf[] */
  byte = getc(in);     /* latest input byte */
  limbo = 0;           /* boolean: an implicit byte is waiting to hitch */
                       /* a ride on the next block */
  lastout = 10;        /* most recent byte written, initially LF to */
                       /* indicate the beginning of a line */

  for (;;) {
    /* Each iteration of this loop writes one block, */
    /* a header byte followed by the first n bytes   */
    /* of buf[] (header and n will be computed).     */

    /* First fill the buffer with data bytes (that is, */
    /* bytes other than the three forbidden ones):     */

    while (byte != EOF && byte != 0 && byte != 10 && byte != 13 && m < 80*9) {
      buf[m++] = byte;
      byte = getc(in);
    }

    if (m == 0 && byte == EOF && !limbo) break;  /* all done */

    if (m < 80) {  /* short block */
      if (byte == EOF) i = limbo = 0;
      else {
        if (byte == 0) i = 0;
        else if (byte == 10) i = 1;
        else assert(byte == 13), i = 2;
        /* Now that byte goes into limbo, and we read the next byte. */
        limbo = 1;
        byte = getc(in);
      }

      n = m;
      header = 16 + 80*i + n;
    }
    else {  /* long block */
      limbo = 0;
      n = m - m % 80;

      if (1+n > roomleft) {  /* block won't fit on current line */
        if (roomleft <= 357) {
          /* End the current line, increasing roomleft to more than n. */
          /* Insert a filler byte if the line would end in whitespace. */
          if (isspace(lastout)) putc(15,out);
          putc(13,out);
          putc(10,out);
          lastout = 10;
          roomleft = linemax;
        }
        else {
          /* Split the block, decreasing n to less than roomleft.    */
          /* The threshold of 357 is the point where the overhead of */
          /* splitting the block becomes outweighed by the better    */
          /* amortization of the trailing CRLF over a longer line.   */

          n = roomleft - 1;
          n = n - n % 80;
        }
      }

      header = n / 80;
      assert (header >= 1 && header <= 9);

      /* Insert a filler byte at the start of */
      /* a line that begins with two hyphens.  */

      if (lastout == 10 && header == 45 && buf[0] == 45) {
        /* It's possible that m is 0, in which case buf[0] would */
        /* be garbage, but in that case header would be 16 or 96 */
        /* or 176, not 45, so the && protects us.                */

        putc(15,out);
        --roomleft;
      }
    }

    /* Write the block. */

    putc(header,out);
    fwrite(buf,n,1,out);
    lastout = n == 0 ? header : buf[n-1];
    roomleft -= 1+n;

    /* Slide the buffer.  A circular buffer might be more */
    /* efficient, but sliding is simpler to implement.    */

    m -= n;
    if (m > 0) memmove(buf, buf + n, m);
  }

  /* End the last line if we're in the middle of it. */

  if (lastout != 10) {
    if (isspace(lastout)) putc(15,out);
    putc(13,out);
    putc(10,out);
  }
}

int main(int argc, char **argv)
{
  if ( argc != 2 ||
       (strcmp(argv[1], "-e") != 0 && strcmp(argv[1], "-d") != 0) ) {
    fprintf(stderr, "usage:\n%s -e\n%s -d\n", argv[0], argv[0]);
    return EXIT_FAILURE;
  }

  if (argv[1][1] == 'e') encode(stdin,stdout);
  else assert(argv[1][1] == 'd'), decode(stdin,stdout);
  return EXIT_SUCCESS;
}

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Malformed header - what would you do?
Date:  Sat, 16 Jul 2005 02:00:25 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 48
Sender: [email-masked]

Paul Smith wrote:
 
> on the second "14/7/2005 11:27:42.567" timestamped line,
> there's a tab character (and nothing else) on the line.
 
> Our software treats that line as a continuation of the
> 'X-SPAM-STATUS' line, but Outlook Express treats it as
> the end of the message header.

OE does it wrong, the mail header ends with ...<CRLF>CRLF>

> our software is adding something to the end of the
> message header, and Outlook Express (and possibly other
> email clients) are seeing our extra header as being
> added to the start of the message text rather than at
> the end of the header.

You could silently replace ...<CRLF><HT><CRLF> by ...<CRLF>
after you talked to your legal department.  It could have
all kinds of funny effects like breaking header signatures.

Another idea, because you already discussed with your legal
department that you manipulate the mail at all, insert your
stuff before the offending header field.

> Part of me says that we should make our software send a
> 25kV surge down the line to fry the computer which was
> adding the superfluous tab character,

That's another idea, or how about frying Redmond ? <gd&r>

> Is our software wrong?

No.

> Is the software adding the 'X-SPAM-STATUS' line wrong

Not really, 2822 says about CFWS:

| However, where CFWS occurs in this standard, it MUST NOT
| be inserted in such a way that any line of a folded header
| field is made up entirely of WSP characters and nothing
| else.

That's only about 2822-header fields, and not about the
obscure X-SPAM-STATUS.
                       Bye, Frank

Date: Fri, 15 Jul 2005 09:37:44 +0100
To: IETF RFC-822 list <[email-masked]>, [email-masked]
From: Paul Smith <[email-masked]>
Subject: Re: Malformed header - what would you do?
Sender: [email-masked]

At 08:43 15/07/2005, Adam M. Costello wrote:

>Paul Smith <[email-masked]> wrote:
>
>Your software is correct, and the Microsoft software is incorrect.

That was my feeling ;)

> > Is the software adding the 'X-SPAM-STATUS' line wrong (ie is it
> > allowed to have FWS at the end of a line and put a CRLF before that
> > final FWS)?
>
>That software is wrong according to RFC-2822, which is only four years
>old.  It forbids the creation of messages containing whitespace-only
>lines in the header, but still requires all parsers to handle them
>properly, because they were valid under RFC-822.

Ah, I've just found that in section 3.2.3 - I'd been studying section 2.2.x 
madly looking for something saying that, but couldn't find it (otherwise, 
I'd probably not have asked the question here)

> > What would you do?  Is our software wrong?
>
>Your software is not wrong to simply add a header field.  One thing
>you might consider is to first repair the header by removing all
>whitespace-only lines, or just whitespace-only lines at the end of the
>header, if that's easier.  In general, mucking with somebody else's
>headers is a no-no and is dangerous, but this would just be a form of
>refolding (which is supposed to be safe) for the purpose of making the
>message more conformant to RFC-2822, so it's probably justifiable (in my
>opinion).

Yep - now I've found that bit in section 3.2.3 of RFC 2822, I'll probably 
strip out any whitespace only header lines (with an option to leave them 
in, in case it causes problems for someone for some odd reason..)

Thanks for the pointers.

Paul                            VPOP3 - Internet Email Server/Gateway
[email-masked]                      http://www.pscs.co.uk/

Date: Fri, 15 Jul 2005 07:43:08 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Malformed header - what would you do?
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Paul Smith <[email-masked]> wrote:
 
> Our software treats that line as a continuation of the 'X-SPAM-STATUS' 
> line, but Outlook Express treats it as the end of the message header.

Your software is correct, and the Microsoft software is incorrect.

RFC-822 (now 23 years old) makes it perfectly clear that a field always
ends with a CRLF, and the header is separated from the body by a CRLF,
implying that there is a completely empty line between the header and
body.

     field       =  field-name ":" [ field-body ] CRLF
     message     =  fields *( CRLF *text )       ; Everything after
                                                 ;  first null line
                                                 ;  is message body

The spec is not completely rigorous, because there is no rule that says
that "fields" is a sequence of "field", but the null-line separator is
clarified in prose:

    A message consists of header fields and, optionally, a body.  The
    body is simply a sequence of lines containing ASCII characters.  It
    is separated from the headers by a null line (i.e., a line with
    nothing preceding the CRLF).

> Part of me says that we should make our software send a 25kV surge    
> down the line to fry the computer which was adding the superfluous    
> tab character,                                                        
>
> Is the software adding the 'X-SPAM-STATUS' line wrong (ie is it
> allowed to have FWS at the end of a line and put a CRLF before that
> final FWS)?

That software is wrong according to RFC-2822, which is only four years
old.  It forbids the creation of messages containing whitespace-only
lines in the header, but still requires all parsers to handle them
properly, because they were valid under RFC-822.

> What would you do?  Is our software wrong?

Your software is not wrong to simply add a header field.  One thing
you might consider is to first repair the header by removing all
whitespace-only lines, or just whitespace-only lines at the end of the
header, if that's easier.  In general, mucking with somebody else's
headers is a no-no and is dangerous, but this would just be a form of
refolding (which is supposed to be safe) for the purpose of making the
message more conformant to RFC-2822, so it's probably justifiable (in my
opinion).

AMC

Date: Thu, 14 Jul 2005 15:47:58 +0100
To: [email-masked]
From: Paul Smith <[email-masked]>
Subject: Malformed header - what would you do?
Sender: [email-masked]

I've just seen a case where our software falls over slightly, and I'm not 
sure what it 'should' do..

Here's a section from our server log:

14/7/2005 11:27:42.473 - <X-OriginalArrivalTime: 14 Jul 2005 10:26:27.0139 
(UTC) FILETIME=[7DF44130:01C5885E]
14/7/2005 11:27:42.473 - <X-SPAM-STATUS: No,SpamAssassinReport=Spam 
detection software, running on the system "MAILSERVER2", has scanned
14/7/2005 11:27:42.489 - <      this message. For more information contact 
the administrator of that system
14/7/2005 11:27:42.489 - <      Content analysis details:   (0.7 points, 
5.0 required)
14/7/2005 11:27:42.504 - <       pts rule name              description
14/7/2005 11:27:42.504 - <      ---- ---------------------- 
--------------------------------------------------
14/7/2005 11:27:42.520 - <       0.0 HTML_TAG_EXISTS_TBODY  BODY: HTML has 
"tbody" tag
14/7/2005 11:27:42.520 - <       0.1 HTML_TAG_BALANCE_HTML  BODY: HTML has 
unbalanced "html" tags
14/7/2005 11:27:42.536 - <      -0.7 BAYES_10               BODY: Bayesian 
spam probability is 10 to 20%
14/7/2005 11:27:42.536 - <                                  [score: 0.1614]
14/7/2005 11:27:42.551 - <       1.2 HTML_20_30             BODY: Message 
is 20% to 30% HTML
14/7/2005 11:27:42.551 - <       0.1 HTML_MESSAGE           BODY: HTML 
included in message
14/7/2005 11:27:42.567 - <       0.0 UPPERCASE_25_50        message body is 
25-50% uppercase
14/7/2005 11:27:42.567 - <
14/7/2005 11:27:42.582 - <
14/7/2005 11:27:42.614 - <
14/7/2005 11:27:42.629 - <Start of the message text

What you can't see is that on the second "14/7/2005 11:27:42.567" 
timestamped line, there's a tab character (and nothing else) on the line.

Our software treats that line as a continuation of the 'X-SPAM-STATUS' 
line, but Outlook Express treats it as the end of the message header. It 
potentially might not be a problem - except that our software is adding 
something to the end of the message header, and Outlook Express (and 
possibly other email clients) are seeing our extra header as being added to 
the start of the message text rather than at the end of the header.

Part of me says that we should make our software send a 25kV surge down the 
line to fry the computer which was adding the superfluous tab character, 
but I can't find anywhere in RFC 2822 which allows for this behaviour ;)

What would you do? Is our software wrong? Is the software adding the 
'X-SPAM-STATUS' line wrong (ie is it allowed to have FWS at the end of a 
line and put a CRLF before that final FWS)?

Paul                            VPOP3 - Internet Email Server/Gateway
[email-masked]                      http://www.pscs.co.uk/

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: New internet-Drafts for an extensible Inernet Message Format
Date: Tue, 12 Jul 2005 12:23:45 -0400
Cc: [email-masked]
Sender: [email-masked]

On Tue July 12 2005 12:12, Bruce Lilly wrote:
> Four related new drafts have been submitted[...]
> The drafts are draft-lilly-extensible-internet-message-format-p01-00
> and -p02-00, -p03-00, and -p04-00.

P.S. in addition to the official ASCII text versions, other formats
(PostScript and PDF, both for ANSI A and ISO A4 formats, and HTML)
are available at http://users.erols.com/blilly/Internet-Drafts

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: New internet-Drafts for an extensible Inernet Message Format
Date: Tue, 12 Jul 2005 12:12:15 -0400
Cc: [email-masked]
Sender: [email-masked]

Four related new drafts have been submitted discussing an extensible
Internet Message Format to address a number of existing issues:
o clutter in the header of the current (2822) format as received,
  which has led to implementations suppressing important content
  as well as clutter
o ability to protect (sign and/or encrypt) the end-to-end content
  including the originator-supplied header fields
o extensibility for further internationalization of content and/or
  for experimentation with new message formats
o ability to reliably detect multiple list expansions, including
  the order of list expansions
o avoiding a problem with current message/partial fragmentation
  and reassembly whereby the originator's Date field is altered
  and/or the partial message header Date fields do not accurately
  convey the RFC 2822-specified semantics
all while maintaining backward compatibility and reliability.

The drafts are draft-lilly-extensible-internet-message-format-p01-00
and -p02-00, -p03-00, and -p04-00.

It is expected that there may be additional related parts (repositories
for UAs, MSAs, filters, message stores, etc. to record non-originator
content).

-p02 is intended for Standards Track, which is not clearly indicated
due to a typo; the others indicate their respective intended status
in the page one heading per ID-Checklist suggestions.

-p01 and -p04 contain MIME media type registration templates and
-p03 contains some registration guidelines for related media type
registrations.  -p01 is an introduction.

All four drafts are in a very early stage of development, rushed
somewhat to meet the pre-IETF publication deadline.

General discussion should probably take place on the ietf-822
list (where replies are suggested by the Reply-To field of this
message), whereas media registration specific discussions should
probably take place on the ietf-types list.  The author of the drafts
reads both lists, so cross-posting to both lists should be reserved
for issues likely to be of interest to both lists.

The author is expected to be incommunicado from tomorrow through
July 19, so don't expect immediate responses.

Date: Fri, 08 Jul 2005 14:24:11 -0400
From: John C Klensin <[email-masked]>
To: John Leslie <[email-masked]>, Tony Hansen <[email-masked]>
cc: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: OPES and Email
Sender: [email-masked]

--On Friday, 08 July, 2005 13:37 -0400 John Leslie
<[email-masked]> wrote:

> Tony Hansen <[email-masked]> wrote:
>> ... 
>> OPES SMTP Use Cases (draft-ietf-opes-smtp-use-cases-02.txt).
>> ...
>>    The Open Pluggable Edge Services (OPES) framework is
>>    application agnostic.  Application specific adaptations
>>    extend that framework. This document describes OPES SMTP
>>    use cases and deployment scenarios in preparation for SMTP
>>    adaptation with OPES.
> 
>    We should be very careful here. "Open Pluggable Edge
> Services" suggests we're talking only about "edge services";
> and IMHO we should be sure to explain any situations where the
> OPES WG might have a different opinion than others as to what
> an "edge service" is.
> 
> ] In SMTP the OPES processor may be any agent participating in
> SMTP ] exhanges, including MSA, MTA, MDA, and MUA.  This
> document focues on ] use cases in which the OPES processor is
> a mail transfer agent (MTA).

Note that, while most of us know, more or less, what all of
those nice abbreviations mean, there is no consensus definition
for an MDA (the 2821 definition of a "delivery MTA" may or may
not be the same thing) nor, IMO, does was the submission
terminology of RFC 2476 or 2476bis (aka "originating server" in
2821) intended to  adequately delimit the "MSA" term.

>    I'm afraid we start out with different opinions. :^(
> 
>    MTAs have a number of functions; some of which we consider
> to be "edge" functions, and some which we don't. (YMMV.)
>...

Tony,

Let me add one impression to John's useful explanation...

In terms of the structure and language of 2821 (and 2821bis,
which I just sent off for posting (!)), the only MTAs (SMTP
clients or servers) that are even vaguely "edge" services or
systems are:

	* The originator (aka the submission server or MSA)
	
	* The destination server
	
	* A gateway

The definitions of the first and second in 2821/ 2821bis are
deliberately a little fuzzy, but most of us think we know one
when we see one and would probably be in 95% agreement on any
given case.  The discriminating definitions for a gateways is
also a bit fuzzy, but there was an effort during the DRUMS
period to keep that definition firm enough to make it hard for
someone to say "I want to do what would otherwise be considered
Bad Things, hence I will claim to be a gateway".

All of the other cases are relays (with some funny edge cases
around what a non-primary MX can do or not do), and actions
committed by them, especially some of the modifications
suggested in the OPES draft, would, from a 2821 perspective,
constitute mucking around in the middle of the mail environment.
Such actions are prohibited to the extent to which 2821 can
prohibit anything.  

You know all of this.  The question for me is that, given that
the OPES doc pushes really close to, or past, the boundary in
which its recommendations and structure require violating the
provisions of 2821, how would you like people to respond in a
way that would be most constructive?  The answer obviously
interacts with several of the discussion threads on the
ietf-smtp list during the last few weeks, including such
questions as the legitimacy or appropriateness of mail relays
doing virus removal.

best,
     john

Date: Fri, 8 Jul 2005 13:37:35 -0400
From: John Leslie <[email-masked]>
To: Tony Hansen <[email-masked]>
Cc: ietf-822 <[email-masked]>, [email-masked]
Subject: Re: OPES and Email
Sender: [email-masked]

Tony Hansen <[email-masked]> wrote:
>... 
> OPES SMTP Use Cases (draft-ietf-opes-smtp-use-cases-02.txt).
>...
>    The Open Pluggable Edge Services (OPES) framework is application
>    agnostic.  Application specific adaptations extend that framework.
>    This document describes OPES SMTP use cases and deployment scenarios
>    in preparation for SMTP adaptation with OPES.

   We should be very careful here. "Open Pluggable Edge Services" suggests
we're talking only about "edge services"; and IMHO we should be sure to
explain any situations where the OPES WG might have a different opinion
than others as to what an "edge service" is.

] In SMTP the OPES processor may be any agent participating in SMTP
] exhanges, including MSA, MTA, MDA, and MUA.  This document focues on
] use cases in which the OPES processor is a mail transfer agent (MTA).

   I'm afraid we start out with different opinions. :^(

   MTAs have a number of functions; some of which we consider to be
"edge" functions, and some which we don't. (YMMV.)

] This work focuses on SMTP based services that want to modify command
] values and those that want to block commands by defining an error
] response that the MTA should send in response to the response it
] received.  OPES MTA will be involved in SMTP command modification and
] command satisfaction, analog to request modification and request
] satisfaction from HTTP [9].

   I'm afraid I don't understand why this describes an "edge service".

]  +-------+  +---------+      +---------+      +--------+  +-------+
]  |mail  M|  |M mail  M| SMTP |M mail  M| SMTP |M mail M|  |M mail |
]  |clnt  U|--|S srvr  T|------|T gway  T|------|T srvr D|--|U clnt |
]  |      A|  |A       A|      |A       A|      |A      A|  |A      |
]  +-------+  +---------+      +---------+      +--------+  +-------+
]                  |                |                |
]                  | OCP            | OCP            | OCP
]                  |                |                |
]             +----------+     +----------+     +----------+
]             |  callout |     |  callout |     |  callout |
]             |  server  |     |  server  |     |  server  |
]             +----------+     +----------+     +----------+
] 
]                        Figure 2: OPES SMTP Flow

   This describes three situations. I think most of us would agree
that the middle (gateway) function is an edge function. I'm less sure
about the other two. (In fact, I'm fresh out of any "the" answers,
here.)

   There's also the very common situation:
" 
"                -------+      +---------+      +------
"                mail  M| SMTP |M mail  M| SMTP |M mail
"                srvr  T|------|T xfr   T|------|T srvr
"                      A|      |A       A|      |A     
"                -------+      +---------+      +------

wherein the middle box is considered "an intermediate MTA", and does
no gateway functions. It's not sufficiently clear, IMHO, whether OPES
intends for it to be appropriate for it to link to an OCP. (I think
most of us would say it shouldn't.)

   There are, of course, cases where outgoing email is filtered on a
separate machine from the MSA which initially receives it. I honestly
don't know whether most of us would consider that an "edge" process.

   So, bottom line, I don't have good answers, but I'm pretty sure we
should clarify whether there's agreement as to what SMTP-related
functions deserve to interact with "edge services".

--
John Leslie <[email-masked]>

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: Tony Hansen <[email-masked]>
Subject: Re: OPES and Email
Date: Fri, 8 Jul 2005 10:12:38 -0400
Cc: ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

On Thu July 7 2005 14:58, Tony Hansen wrote:
> 
> Heads up, email experts!
> 
> I don't know if you saw the recent I-D documents that came out, entitled
> OPES SMTP Use Cases (draft-ietf-opes-smtp-use-cases-02.txt).
[...]
> I'm co-chair of the OPES group, and am aware of its potential
> ramifications to the email world. So, I'd appreciate it if you could
> review the use-cases document. Discussion about the document is
> occurring on the [email-masked] mailing list. (If you'd rather
> not join that list, I can channel your comments.)

I saw and downloaded it, but hadn't had time to review it.  I have now
skimmed over it; here are some initial comments not covering issues
raised by others (e.g. unchecked spelling errors), based on a cursory
scan of the draft:

1. Check against I-D checklist and I-D Guidelines (e.g. via idnits):
   there is no IANA Considerations section -- one is required per
   I-D Guidelines.  Has any thought been given to IANA considerations?
   [A debate has been raging on the IETF discussion list for close to
   a month, however the current IESG guidelines require an IANA
   Considerations section].

2. Numeric references are deprecated because they lead to RFC Editor
   busy-work (renumbering).  See the rfc-interest mailing list archive.
   They also make review and use unnecessarily difficult; I know what
   "[RFC2821]" references, but "[8]" isn't helpful.

3. Examples (some already noted by others) should be checked; a valid
   and carefully formulated example can illustrate a specification, but
   a poorly formulated or invalid example simply causes confusion.  A
   message or message fragment should be checked for validity -- there
   are validating parsers available for authors' use (see
   http://tools.ietf.org (tools inventory and wiki)).  The draft contains:
      From: [email-masked]
      To: [email-masked]
      Subject: Test
      Hi, this is a test!
   An empty line is required between the message header and the body text
   (see RFC 2822, which would be suitable as a reference...).  A valid
   message (RFC 2822 again) MUST have a Date field in the message header,
   and that is missing from the example.

4. Use of standard terminology aids clarity (RFC 1958).  FYI 18 provides
   some general terminology; FYI 36 for security.  RFC 2822 is also
   apropos for terminology.   In particular, note the standard meaning of
   "header" as defined in FYI 18 and RFC 2822.

5. Figure 2 implies that there is a direct connection between an MDA and
   MUA.  An MDA typically stores messages in a message store of some sort,
   which is accessed by an MUA using some protocol (typically one of
   direct file access, a POP version, or an IMAP version).  It is unclear
   from the draft whether or not there is any intention to address those
   other core mail protocols, whereas the draft does mention completely
   unrelated protocols such as HTTP -- that is at best rather odd.

6. Speaking of MUAs, the draft mentions filtering at MTAs, which is a
   dubious practice (as briefly touched upon by some other reviewers;
   basically it violates the end-to-end architectural principle (RFC 1958
   and others))).  Filtering may also take place from an MUA, which is
   well situated to consider filtering characteristics appropriate for a
   particular mailbox user (unlike some random MSA or MTA).  However, this
   use is not mentioned by the draft, which *seems* reasonable as a
   recipient MUA doesn't use SMTP, however as it is a primary means of
   filtering, it probably warrants a mention in the Introduction (oh dear,
   now I have to choose the order of segues).

7. Speaking of SMTP, some mention should probably be made regarding
   SMTP variants, viz. whether or not the draft is to be considered
   applicable to submission and/or LMTP.

8. Speaking of the Introduction, it would be nice to use general
   terminology until OPES-specific jargon is first defined.
   unfortunately, the Introduction starts with obscure jargon (e.g.
   "callout"), which coupled with the obtuse references (e.g. "[3]"),
   non-standard use of terminology, and lack of definitions of specific
   jargon in section 2, makes understanding the draft quite difficult.

9. Careful review of the text is warranted; how exactly can an error
   message cause removal of an attachment ("The attachment is removed
   by an error message")?

10. I know what OPES is, but what is "OEPS"?

11. Some consistency, particularly in the generality of approach, with
   existing practices documents would be nice.  In particular, draft
   sect. 4.9 is specific to one peculiar access mechanism, whereas e.g.
   RFC 1344 suggests a more general MIME mechanism.  Speaking of RFC
   1344, please pay particular attention to the implications of
   message/partial and message/external-body for filters (which act as
   a type of gateway) [much of the rest of 1344 has been superseded].
   In light of RFC 1344 and several years after 
   http://www.kb.cert.org/vuls/id/836088 a draft regarding the topic of
   content/virus scanners that doesn't address the security issues is
   bordering on unconscionable.

12. Speaking of security, the draft Security Considerations section
   needs some careful thought.  It is woefully inadequate (see above,
   BCP 72 and FYI 18).

13. The recommended organization (instructions2authors) is to place
   Appendices before References, and that implies that the references
   sections should be unnumbered.

14. Categorization of references needs to be carefully reviewed.
   Considering that the topic is SMTP and therefore requires some
   understanding of the details of that protocol (seems to be missing
   on the part of the authors, based on numerous errors in the
   examples, as noted by others), RFC 2821 should be normative rather
   than merely informative, for example.

15. Please don't encourage "Add a mark to the subject of the email";
   see RFC 4096.  That is a layering violation; Subject, To, From,
   Cc, Reply-To, Date, etc. are originator-specified fields (RFC 2822)
   and modification by others w/o the express permission of the
   author(s) is nothing less than forgery.  Modification of message
   content also invalidates security mechanisms such as digital
   signatures (RFC 1847).  Because of those considerations, such
   practices should be thoroughly and firmly prohibited rather than
   encouraged.

16. Examples of client-server dialogs use "S:" and "R:" labels, which
   would be fine if the draft were the only document using such dialog
   examples.  It is not, and the typical use would be "S:" for server
   and "C:" for client.  Unfortunately, the draft choice associates "S:"
   with the opposite part of the dialog from the typical cases, and that
   can lead to confusion.

17. Formatting could be improved dramatically; e.g. page 4 is mostly
   wasted empty space.

18. (general comment on subject matter)  Frankly, filtering can be
   accomplished in any number of ways; there are methods which have
   been in use in practice for quite some time (e.g. the simple and
   old method of passing information to a program and examining the
   exit code) and there is at least one Standards Track method
   specifically for email filtering (Sieve).  It is unclear precisely
   what benefit would justify the overhead of a heavy-weight (layers of
   new protocols) method such as seems to be the case for OPES (I haven't
   yet retrieved all of the numbered references for review, but that's
   the impression I get).   A method for filtering email that doesn't
   integrate with Sieve, and doesn't even mention it is simply mind-
   boggling.  See also item 6 above, and consider the possibility that
   this work might be abandoned.  I realize that authors don't like to
   hear that, but I think this work is going to be difficult to salvage
   considering the architectural and layering issues as well as lack
   of integration with existing mail-related standards track
   specifications.

19. What about internationalization considerations?  How can some
   random MTA in Omaha reasonably filter content being sent from a
   Croatian emigre to relatives in the old country, for example?

I think most of the other items that I noticed have already been
mentioned.  The document needs quite a bit of work, some requiring
careful thought (layering issues, protocol issues, security issues,
integration with Sieve and other mail-related Standards Track
specifications, relationship to the Internet Architecture and other
approved RFCs), some mostly mechanical (proofreading, spelling and
grammar checks, validation of examples, chacks against I-D/RFC 
guidelines, instructions, and checklists).  It's going to need at least
another version before it's ready for thorough review, unless of course
it is simply dropped in light of the architectural/layering/integration
issues noted above.

Date: Thu, 07 Jul 2005 22:51:59 -0400
From: Tony Hansen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: OPES and Email
Sender: [email-masked]

An edge service is a service plugged into the side of an existing
service. For example, real time call outs from within HTTP servers or
SMTP servers to virus scanners or spam blockers are examples of edge
services.

An open pluggable edge service is an edge service that meets certain
criteria that allows them to be swapped in and out. For example, you
could swap out company X's virus scanner and replace it with company
Y's. Or cascade a series of virus scanners from multiple sources.

The OPES framework has defined such a set of criteria for edge
services. In addition, the OPES framework includes a profile for how
these services could be plugged into an HTTP server.

The RFCs 3238, 3752, 3835-3838 and 4037 all discuss OPES. In particular,
RFC 3752 covers a variety of use cases that are fairly generic. The
internet draft OPES SMTP Use Cases expands on the core set with use
cases that could be considered more specific to email.

I hope this helps.

	Tony Hansen
	[email-masked]

Frank Ellermann wrote:
> Tony Hansen wrote:
> 
>> I don't know if you saw the recent I-D documents that came out, 
>> entitled OPES SMTP Use Cases
>> draft-ietf-opes-smtp-use-cases-02.txt).
> 
> Yes, I saw the announcement, and then decided that "somebody" will 
> probably explain what OPES is, I've not the faintest idea.
> 
>> The Open Pluggable Edge Services (OPES) framework is application 
>> agnostic.
> 
> Yes, and I'm OPES agnostic... ;-)  So what is it ?  Bye, Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: OPES and Email
Date:  Fri, 08 Jul 2005 03:54:06 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 14
Cc: [email-masked]
Sender: [email-masked]

Tony Hansen wrote:
 
> I don't know if you saw the recent I-D documents that came
> out, entitled OPES SMTP Use Cases
> (draft-ietf-opes-smtp-use-cases-02.txt).

Yes, I saw the announcement, and then decided that "somebody"
will probably explain what OPES is, I've not the faintest idea.

> The Open Pluggable Edge Services (OPES) framework is
> application agnostic.

Yes, and I'm OPES agnostic... ;-)  So what is it ?  Bye, Frank

Date: Thu, 07 Jul 2005 16:26:06 -0700
To: ietf-822 <[email-masked]>
From: SM <[email-masked]>
Subject: Re: OPES and Email
Cc: Tony Hansen <[email-masked]>
Sender: [email-masked]

At 14:00 07-07-2005, Matti Aarnio wrote:

>   S: 220 mail.example.com Sample ESMTP MAIL Service, Version: 1.2 
> ready at Thu, 20 Jan 2005 11:24:40+0100
>   C: HELO ThatsMe
>
>corrected:  HELO thatsme.example.org
>
>   S: 250 mail.example.com Hello [192.168.0.138]

Shouldn't the HELO use [192.168.0.138] instead of thatsme.example.org 
as per RFC 2821 Section 4.1.1.1?

Regards,
-sm 

Date: Thu, 07 Jul 2005 17:08:27 -0400
From: Tony Hansen <[email-masked]>
CC: ietf-822 <[email-masked]>
Subject: Re: OPES and Email
Sender: [email-masked]

Matti Aarnio wrote:
> In my first quick scan (oh my.. 13 000 plus IDs in IETF archive!
> mere 1932 IDs after cleaning all expired/completed/etc...)
> I spotted example SMTP session that violates a bunch of protocol
> syntax details - and we have just had a discussion about the myriad
> ways that people violate them.. ....

Yes, those are already being fixed.

> About the idea of "put attachments in web-server" -- one thing that I
> hate profoundly is MS Exchange's habit of converting incoming
> TEXT/PLAIN email into TEXT/HTML in some obscure situations. Sender
> didn't send HTML, and recipient (= me) doesn't want HTML. Why that
> beast must do such stupidity ?
> 
> This idea of converting message so that the attachment can be 
> downloaded from elsewere (other than over IMAP) sounds like its
> writers are thinking that everybody will move to webmail and HTML.
> 
> All in all, as a user I am very much opposed to ideas where my emails
> encounter any other conversions, than those of transfer encoding.

Thanks for your comments and taking the time to review the document.

	Tony Hansen
	[email-masked]

Date: 	Fri, 8 Jul 2005 00:00:59 +0300
From: Matti Aarnio <[email-masked]>
To: Tony Hansen <[email-masked]>
Cc: ietf-822 <[email-masked]>
Subject: Re: OPES and Email
Sender: [email-masked]

On Thu, Jul 07, 2005 at 02:58:31PM -0400, Tony Hansen wrote:
> Heads up, email experts!
> 
> I don't know if you saw the recent I-D documents that came out, entitled
> OPES SMTP Use Cases (draft-ietf-opes-smtp-use-cases-02.txt).
...
> I'm co-chair of the OPES group, and am aware of its potential
> ramifications to the email world. So, I'd appreciate it if you could
> review the use-cases document. Discussion about the document is
> occurring on the [email-masked] mailing list. (If you'd rather
> not join that list, I can channel your comments.)

Hello Tony,

In my first quick scan (oh my.. 13 000 plus IDs in IETF archive!
mere 1932 IDs after cleaning all expired/completed/etc...)
I spotted example SMTP session that violates a bunch of protocol
syntax details - and we have just had a discussion about the myriad
ways that people violate them..

  S: 220 mail.example.com Sample ESMTP MAIL Service, Version: 1.2 ready at Thu, 20 Jan 2005 11:24:40+0100
  C: HELO ThatsMe

corrected:  HELO thatsme.example.org

  S: 250 mail.example.com Hello [192.168.0.138]
  C: MAIL FROM: [email-masked]

corrected:  MAIL FROM:<[email-masked]>

  S: 250 2.1.0 [email-masked]....Sender OK
  C: RCPT TO: [email-masked]

corrected:  RCPT TO:<[email-masked]>

  S: 250 2.1.5 [email-masked]

e.g. must have angle-brackets, must not have spaces in places
where they are not specified as allowed per RFC 2821 strict syntax.

About the idea of "put attachments in web-server" -- one thing
that I hate profoundly is MS Exchange's habit of converting
incoming TEXT/PLAIN email into TEXT/HTML in some obscure situations.
Sender didn't  send HTML, and recipient (= me) doesn't want HTML.
Why that beast must do such stupidity ? 

This idea of converting message so that the attachment can be
downloaded from elsewere (other than over IMAP) sounds like
its writers are thinking that everybody will move to webmail
and HTML.

All in all, as a user I am very much opposed to ideas where my
emails encounter any other conversions, than those of transfer
encoding.

> Thanks!
> 
> 	Tony Hansen
> 	[email-masked]
-- 
/Matti Aarnio	<[email-masked]>

Date: Thu, 07 Jul 2005 14:58:31 -0400
From: Tony Hansen <[email-masked]>
To: ietf-822 <[email-masked]>, [email-masked]
Subject: OPES and Email
Sender: [email-masked]

Heads up, email experts!

I don't know if you saw the recent I-D documents that came out, entitled
OPES SMTP Use Cases (draft-ietf-opes-smtp-use-cases-02.txt).

As the abstract says:

   The Open Pluggable Edge Services (OPES) framework is application
   agnostic.  Application specific adaptations extend that framework.
   This document describes OPES SMTP use cases and deployment scenarios
   in preparation for SMTP adaptation with OPES.

The intent is to provide access from within email to the OPES filters
being used with HTTP. (Virus scanners are an example.)

I'm co-chair of the OPES group, and am aware of its potential
ramifications to the email world. So, I'd appreciate it if you could
review the use-cases document. Discussion about the document is
occurring on the [email-masked] mailing list. (If you'd rather
not join that list, I can channel your comments.)

Thanks!

	Tony Hansen
	[email-masked]

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Date: Fri, 1 Jul 2005 23:02:29 -0400
Cc: [email-masked]
Sender: [email-masked]

On Fri July 1 2005 18:33, [email-masked] wrote:
> It is relatively easy to design a scheme that limits the
> overhead to 1-2% no matter what the input.

Maybe, depending on the constraints.  The minimum constraints on the
output are:
o CRLF only for line endings, no lone CR or lone LF
o no NUL
o line length <= 998 octets
(for that is the definition of 8bit).  The input of course is an
unconstrained sequence of octets.

About 1.4% expansion should be possible with only those constraints,
a fairly simple algorithm (faster decode than encode), and moderate
encoder memory requirements (an 84 octet input buffer).

Adding constraints makes things more difficult.  For example,
constraining line length to 76 octets as is the case with
the other CTEs necessitates an expansion by 78/76 which is >2%
overhead exclusive of any other considerations.

Staying within 2% expansion while avoiding lone CR, lone LF, and NUL
implies a line length of about 250 octets.

Other constraints include amount of memory an encoder or decoder
might require, restrictions on leading or trailing whitespace,
avoiding other troublesome output sequences, whether or not a
stream can be encoded on the fly, and complexity of encoder and/or
decoder.

From: [email-masked]
Cc: ietf-822 <[email-masked]>, [email-masked]
To: Bruce Lilly <[email-masked]>
Date: Fri, 01 Jul 2005 15:33:45 -0700 (PDT)
Subject: Re: gzip/deflate compression/encoding
Sender: [email-masked]

> On Thu June 30 2005 15:28, [email-masked] wrote:

> > Doesn't follow. A binary-to-8bit encoding has enough characters available that
> > the overhead can be limited to at most 1-2%.

> for random data.

> The yEnc method will yield a 2x expansion for a bitmap image of
> an elephant in a fog (every pixel has a value of 0xD6).  That's
> considerably worse than a 37% increase.

No, for any data. It is relatively easy to design a scheme that limits the
overhead to 1-2% no matter what the input. The question is which is worth more,
some measure of yEnc compatibility or having this guarantee. I'm persoally
somewhat ambivalent on this and would appreciate hearing other opinions.

				Ned

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Date: Fri, 1 Jul 2005 18:03:50 -0400
Cc: [email-masked]
Sender: [email-masked]

On Thu June 30 2005 15:28, [email-masked] wrote:

> Doesn't follow. A binary-to-8bit encoding has enough characters available that
> the overhead can be limited to at most 1-2%.

for random data.

The yEnc method will yield a 2x expansion for a bitmap image of
an elephant in a fog (every pixel has a value of 0xD6).  That's
considerably worse than a 37% increase.

From: [email-masked]
Cc: [email-masked]
To: Charles Lindsey <[email-masked]>
Date: Fri, 01 Jul 2005 14:24:32 -0700 (PDT)
Subject: Re: gzip/deflate compression/encoding
Sender: [email-masked]

> In <[email-masked]> [email-masked] writes:

> >Repeating for the Nth time: Media types are supposed to be used to identify
> >types of media, not general compression schemes or other stuff. Section 2.2.1
> >of RFC 2048 is very clear about this, and this requirement remains unchanged in
> >section 4.1 of draft-freed-media-type-reg-04.txt. As such, any attempt to
> >register such a media type will be immediately rejected.

> >If you want to play around with general compression schemes and MIME in a
> >standards-compliant way, nothing prevents you from using x-gzip or whatever
> >as a CTE.

> I don't want to play around with x-gzip. I am trying to see what options
> are available within the present MIME setup.

There's only one: Define one or more new CTEs.

> The problem appears to be that there might be a largish number of
> compression algorithms one might wish to use,

"Might wish" != "really need"
 
> combined with a variety of ways of encoding them into 8bit.

There is certainly no reason for there to be more than one of these.

> That would seem to require a rather large
> number of new CTEs to be registered.

Doesn't follow.

> One would like to be able to write

>    Content-Transfer-Encoding: base64; compression=gzip

> but that is incompatible with the present syntax, and it is doubtful it
> could be introduced as an extension without upping the MIME-Version
> number.

Correct.

> Essentially, we have three bits of information to convey:

> The CTE
> The compression algorithm
> The media-type of the uncompressed object.

> I tried to squash two of those into the Content-Type. If that is not to be
> allowed, then please suggest another method. A new Content-Compression
> header seems the only other alternative.

That's also a nonstarter for reasons previously given.

The only option is to define one or more new CTEs. Period.

				Ned

To: [email-masked]
Xref: clerew local.mime:4825
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Date: Fri, 1 Jul 2005 11:06:10 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>Repeating for the Nth time: Media types are supposed to be used to identify
>types of media, not general compression schemes or other stuff. Section 2.2.1
>of RFC 2048 is very clear about this, and this requirement remains unchanged in
>section 4.1 of draft-freed-media-type-reg-04.txt. As such, any attempt to
>register such a media type will be immediately rejected.

>If you want to play around with general compression schemes and MIME in a
>standards-compliant way, nothing prevents you from using x-gzip or whatever
>as a CTE.

I don't want to play around with x-gzip. I am trying to see what options
are available within the present MIME setup.

The problem appears to be that there might be a largish number of
compression algorithms one might wish to use, combined with a variety of
ways of encoding them into 8bit. That would seem to require a rather large
number of new CTEs to be registered.

One would like to be able to write

   Content-Transfer-Encoding: base64; compression=gzip

but that is incompatible with the present syntax, and it is doubtful it
could be introduced as an extension without upping the MIME-Version
number.

Essentially, we have three bits of information to convey:

The CTE
The compression algorithm
The media-type of the uncompressed object.

I tried to squash two of those into the Content-Type. If that is not to be
allowed, then please suggest another method. A new Content-Compression
header seems the only other alternative.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:4827
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: gzip/deflate compression/encoding
Date: Fri, 1 Jul 2005 16:34:45 GMT
Lines: 29
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>> Are the usual audio, image, etc. formats truly 8bit clean (i.e. are they
>> guaranteed not to contain NUL or naked CR or LF)?

>Of course they aren't.

Yes, that's what I thought.

>> If not, then you are
>> back to the 37+% expansion of base64.

>Doesn't follow. A binary-to-8bit encoding has enough characters available that
>the overhead can be limited to at most 1-2%.

Sure, but there is no currently recognized/standardized encoding with that
property (except for yEnc, which I don't think is a serious candidate
yet).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5