Subject: Re: [ietf-822] [Editorial Errata Reported] RFC5322 (3400)

From: Pete Resnick <[email-masked]>
Date: Tue, 27 Nov 2012 17:02:00 -0600

[Bcc'ed to [email-masked]; please discuss over on [email-masked].]

Folks,

The following erratum was posted for 5322. I'm inclined to reject it 
since this discussion actually took place during DRUMS (17-18 March 1998 
in a thread with a subject of "Small Clarification to msg-fmt-04" if you 
are inclined to look) and the consensus outcome as far as I could tell 
(as document editor) was that messages without a final CRLF were SMTP's 
problem. However, 5321 (and 2821) 4.1.1.4 says:

    The mail data are terminated by a line containing only a period, that
    is, the character sequence "<CRLF>.<CRLF>", where the first <CRLF> is
    actually the terminator of the previous line (see Section 4.5.2).
    This is the end of mail data indication.  The first <CRLF> of this
    terminating sequence is also the <CRLF> that ends the final line of
    the data (message text) or, if there was no mail data, ends the DATA
    command itself (the "no mail data" case does not conform to this
    specification since it would require that neither the trace header
    fields required by this specification nor the message header section
    required by RFC 5322 [4] be transmitted).  An extra <CRLF> MUST NOT
    be added, as that would cause an empty line to be added to the
    message.  The only exception to this rule would arise if the message
    body were passed to the originating SMTP-sender with a final "line"
    that did not end in <CRLF>; in that case, the originating SMTP system
    MUST either reject the message as invalid or add <CRLF> in order to
    have the receiving SMTP server recognize the "end of data" condition.

Allowing the originating SMTP system to reject the message as invalid 
seems in conflict with 5322 on this point. So my rejecting this erratum 
will simply end us up with an erratum against 5321.

I'm inclined to hear opinions.

pr

> The following errata report has been submitted for RFC5322,
> "Internet Message Format".
>
> --------------------------------------
> You may review the report below and at:
> http://www.rfc-editor.org/errata_search.php?rfc=5322&eid=3400
>
> --------------------------------------
> Type: Editorial
> Reported by: Christoph Anton Mitterer<[email-masked]>
>
> Section: 3.5.
>
> Original Text
> -------------
>     message         =   (fields / obs-fields)
>                         [CRLF body]
>
>     body            =   (*(*998text CRLF) *998text) / obs-body
>
>
> Corrected Text
> --------------
>     message         =   (fields / obs-fields)
>                         [CRLF body]
>
>     body            =   (*(*998text CRLF) *998text) / obs-body
>
> It is RECOMMENDED that message bodies are terminated by CRLF, though this is in principle not necessary (this does not apply to messages consisting only of a header section, as header fields are always CRLF terminated).
>
> Note however, that when transporting messages via SMTP the last lines of message bodies MUST be terminated by CRLF as specified int RFC 5321, section 4.1.1.4.
>
> Notes
> -----
> Hi folks.
>
> AFAIU, the definition of body allows message bodies (not header sections) that end without CRLF.
>
> RFC5321 section 4.1.1.4. however states: "The mail data are terminated by a line containing only a period, that is, the character sequence "<CRLF>.<CRLF>", where the first<CRLF>  is actually the terminator of the previous line".
>
> So SMTP forbids, what this RFC allows.
> I guess the SMTP RFC can't be changed here and it makes no particular sense to restrict RFC5322 on the other hand.
>
> My suggestion was to add this clarification.
>
> Perhaps a similar one should be added to RFC5321, telling that Internet Messages themselves wouldn't need the last CRLF.
>
> Instructions:
> -------------
> This errata is currently posted as "Reported". If necessary, please
> use "Reply All" to discuss whether it should be verified or
> rejected. When a decision is reached, the verifying party (IESG)
> can log in to change the status and edit the report, if necessary.
>
> --------------------------------------
> RFC5322 (draft-resnick-2822upd-06)
> --------------------------------------
> Title               : Internet Message Format
> Publication Date    : October 2008
> Author(s)           : P. Resnick, Ed.
> Category            : DRAFT STANDARD
> Source              : IETF - NON WORKING GROUP
> Area                : N/A
> Stream              : IETF
> Verifying Party     : IESG
>    

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Keith Moore <[email-masked]>
Date: Wed, 28 Nov 2012 02:00:58 -0500

IMO:

1. A message can be legal without a CRLF at the end.   However,

2. If the message doesn't end with CRLF, there's no way to transfer it 
using SMTP /DATA/ without modifying the message.   (note the emphasis on 
DATA)

3.  Similarly, any message received via SMTP DATA already has a CRLF at 
the end, so there's no ambiguity concerning the last line of a message 
when dealing with any message received using SMTP DATA.  The receiver 
has to assume that the terminating CRLF (before the '.CRLF') was part of 
the original message.

4. A message not ending in CRLF could be transmitted via other means, 
e.g. using SMTP BDAT if the MTA supports it, or any other suitable 
message transfer protocol.   More generally, use of Internet Message 
Format is not limited to SMTP.

However, if the receiving MTA then tried to relay the message to another 
MTA that didn't support BDAT, or to relay or deliver it to any 
environment (e.g. mbox format) that expected messages to consist 
entirely of terminated lines of text, results are more-or-less 
undefined.   The message could be modified to cause it to end in a 
terminated line (with or without changing the content represented by the 
message), or it could be bounced.

Note that there are corner cases where simply appending CRLF could do 
harm, e.g. a MIME message consisting of a single quoted-printable body 
part and a filename specified via the Content-Disposition header field - 
e.g. some sort of application-specific data that needed to be preserved 
end-to-end. In such a case it would make more sense to re-encode the 
message body in base64 than to simply append a CRLF.   But this seems 
out-of-scope for 5321.   If anything needs an erratum explaining this, 
it's RFC 3030.

Conclusions:

1. It seems reasonable to add an erratum to 5322 recommending that 
messages, as originated, end in CRLF, unless it is known that the 
message can be transmitted from sender-to-recipient without loss of full 
binary transparency.

(Note that there are lots of things that are perfectly valid according 
to 5322 which are unlikely to work well in practice, and which would 
probably also be recommended against if anyone took the trouble to try 
to identify them and file errata for them.)

2. It's reasonable, during mail submission only, for a submission server 
or MTA to EITHER adapt a message that doesn't end in CRLF so that it 
does, OR bounce the message, according to local policy.   But as far as 
SMTP is concerned, this is only relevant if BDAT is used to submit the 
message.   So it's really not in scope for RFC 5321.

3. When relaying mail (past the first hop), the issue comes up only when 
either BDAT or a nonstandard protocol or protocol extension are used.    
So again, it's not in scope for 5321.

4. It might be worthwhile to examine RFC 3030 for errata along these lines.

Keith

On 11/27/2012 06:02 PM, Pete Resnick wrote:
> since this discussion actually took place during DRUMS (17-18 March 
> 1998 in a thread with a subject of "Small Clarification to msg-fmt-04" 
> if you are inclined to look) and the consensus outcome as far as I 
> could tell (as document editor) was that messages without a final CRLF 
> were SMTP's problem. However, 5321 (and 2821) 4.1.1.4 says:
>
> seems in conflict with 5322 on this point. So my rejecting this 
> erratum will simply end us up with an erratum against 5321.
>
>> It is RECOMMENDED that message bodies are terminated by CRLF, though 
>> this is in principle not necessary (this does not apply to messages 
>> consisting only of a header section, as header fields are always CRLF 
>> terminated).
>>
>> Note however, that when transporting messages via SMTP the last lines 
>> of message bodies MUST be terminated by CRLF as specified int RFC 
>> 5321, section 4.1.1.4.
>>
>> Notes
>> -----
>> Hi folks.
>>
>> RFC5321 section 4.1.1.4. however states: "The mail data are 
>> terminated by a line containing only a period, that is, the character 
>> sequence "<CRLF>.<CRLF>", where the first<CRLF>  is actually the 
>> terminator of the previous line".
>>
>> I guess the SMTP RFC can't be changed here and it makes no particular 
>> sense to restrict RFC5322 on the other hand.
>>
>> Perhaps a similar one should be added to RFC5321, telling that 
>> Internet Messages themselves wouldn't need the last CRLF.
>>
>> Instructions:
>> -------------

---

From: Tony Finch <[email-masked]>
Date: Wed, 28 Nov 2012 10:50:20 +0000

Keith Moore <[email-masked]> wrote:

I pretty much agree with Keith.

Although I think the clarification is correct, I don't think the defect is
bad enough to cause interoperability failure, which implies it would be OK
to reject the erratum. If you decide to accept the erratum it should
probably mention that textual parts without final line terminators should
be encoded with MIME.

A couple of tangential points:

MIME describes CRLFs as line separators rather than line terminators which
is consistent with 5322 (and necessary for correctly representing the
encapsulated data) but unhelpful for resolving this niggle. RFC 3030
(BINARYMIME / BDAT) assumes lines in textual bodies are terminated rather
than separated which seems to be stricter than MIME specifies, but it
does avoid Keith's worries.

> message body in base64 than to simply append a CRLF.

Or you could append =<CR><LF> but I expect that would not be
bug-compatible.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first.
Rough, becoming slight or moderate. Showers, rain at first. Moderate or good,
occasionally poor at first.

---

From: John C Klensin <[email-masked]>
Date: Wed, 28 Nov 2012 04:17:50 -0500

--On Tuesday, November 27, 2012 17:02 -0600 Pete Resnick
<[email-masked]> wrote:

> The following erratum was posted for 5322. I'm inclined to
> reject it since this discussion actually took place during
> DRUMS (17-18 March 1998 in a thread with a subject of "Small
> Clarification to msg-fmt-04" if you are inclined to look) and
> the consensus outcome as far as I could tell (as document
> editor) was that messages without a final CRLF were SMTP's
> Allowing the originating SMTP system to reject the message as
> invalid seems in conflict with 5322 on this point. So my
> rejecting this erratum will simply end us up with an erratum
> against 5321.
> 

Pete,

Historically, there was no requirement that SMTP transport only
Msg-Fmt (822/ 2822/ 3822) messages.  There is no such
requirement in 821 other than, implicitly, that whatever is
being transported not be screwed up by SMTP relay addition of
trace fields.  Insisting that only 822-compliant messages be
transported would have combined with the now-deprecated IM-ish
SEND (and possibly SAML and SOML) command(s) would have led to a
UI silly state.    There was a bit of a contradiction there
because an issue with trace fields and what the origin and
delivery SMTP servers were required to add, but that was
clarified in 2821 at the same time that SEND and friends were
deprecated (an additional reason for requiring that Msg-Fmt
messages be transmitted was to make MIME and its body part
description headers work).

Conversely and much more important, 822 and its successors have
never required that message bodies be transported over SMTP.  We
don't see non-SMTP transport as often as we did a few decades
ago, but it is still very much with it.  That is the reason why
5322 needs its own requirement for CRLF-terminated lines even
though "normal" SMTP can transmit nothing else.

So my impression is that 5322 is ok.  I personally don't like
it, but the current description is, as you point out, a design
decision, not a textual error.  There would be an error if 5322
got messed up with a headers-only message (empty content) that
reached end of data without a terminating CRLF (with or without
the trailing blank line), but I don't believe it does.

My impression is that 5321 is ok too because the requirement it
imposes doesn't contradict 5322 in any way.  First, while that
requirement of SMTP may not be required by 5322, nothing
prevents SMTP from imposing it.  Again, these are design
decisions made during the DRUMS period, not errors, and errata
are not an appropriate way to reopen those design decisions.   A
prohibition on SMTP rejecting a message that was valid under
5322 would also be pointless: SMTP servers are permitted to
reject messages for any reason at all and a prohibition on
rejecting for this reason would, at best, result in a
less-specific response message when an SMTP server did that.

I don't see the proposed clarification text as a real problem,
but don't see it as necessary either.   If you reject the
present erratum proposal and another one is generated against
5321, I will certainly recommend rejecting the latter as well
(or at least putting it into "hold for document revision".
However, I note that each "hold for document revision"
placeholder, especially the subset that involve earlier design
decisions, will bring us closer to the need for a WG to revise
5321 rather than allowing that to be an independent effort).

best,
    john

---

From: Ned Freed <[email-masked]>
Date: Wed, 28 Nov 2012 07:41:48 -0800 (PST)

> though "normal" SMTP can transmit nothing else.

In one sense we do see it: Binary SMTP (RFC 3030) doesn't have any such
restriction.

I'll also note that MIME explicitly allows text parts inside of a multipart
that don't end in CRLF. In practice the CRLF often ends up getting added
and the use case for such parts is no longer relevant, but the fact remains
they are allowed.

> the trailing blank line), but I don't believe it does.

Agreed on all points.

> less-specific response message when an SMTP server did that.

Agreed as well. And as for whether or not these issues should be reopened,
if this is causing actual operational problems somewhere I for one am not
aware of it.

If we're going to worry about text handling issues in email, there are far
more pressing problems than this.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 28 Nov 2012 07:54:30 -0800 (PST)

> does avoid Keith's worries.

Sorry, that's incorrect. The statements RFC 3030 makes are in the context of
converting to and from local storage formats. The intention is clearly to say
that line separators/terminator/delimiters/whatever have to be represented by a
canonical CRLF sequence.

At no time does RFC 3030 say that all lines in a text part have to be
terminated with a CRLF, only that when lines are terminated the terminator
has to be a CRLF. And as I pointed out previously: MIME explicitly allows
text parts that don't end in CRLF, and encoding is *not* required to
represent such parts inside of a multipart.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Wed, 28 Nov 2012 16:27:06 +0000

> that line separators/terminator/delimiters/whatever have to be represented by a

The problem is that "separators/terminator/delimiters/whatever" is not
sufficiently precise when we are concerned about the last line in the
file. If CRLFs are separators then the terminal one is optional; if they
are terminators then it is not.

Tony.

---

From: Ned Freed <[email-masked]>
Date: Wed, 28 Nov 2012 10:04:22 -0800 (PST)

> are terminators then it is not.

That's correct but really beside the point. The text in RFC 3030 doesn't
explcicitly say "all lines MUST be terminated". Rather, it's saying that when
terminators (it actually uses delimiter in another place) are used they have to
be CRLFs.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Wed, 28 Nov 2012 18:18:35 +0000

Ned Freed <[email-masked]> wrote:
>
> The text in RFC 3030 doesn't explcicitly say "all lines MUST be
> terminated". Rather, it's saying that when terminators (it actually uses
> delimiter in another place) are used they have to be CRLFs.

That isn't how I read it:

  "In particular, it is essential that text be canonically encoded with
   each line properly terminated with <CR><LF>."

  "In particular, text/* MUST be sent with <CR><LF> terminated lines."

I agree with your interpretation of the intent of RFC 3030 but the way it
expresses that intent is not quite correct.

Tony.

---

From: Ned Freed <[email-masked]>
Date: Wed, 28 Nov 2012 11:34:43 -0800 (PST)

>    each line properly terminated with <CR><LF>."

The word "each" gives some weight to your argument. But not a lot - it would
have to be "every" to actually impose the restriction.

>   "In particular, text/* MUST be sent with <CR><LF> terminated lines."

This says nothing about unterminated lines.

> expresses that intent is not quite correct.

Well, the bigger problem is that the entire style of the RFC 3030 prose is
obsolete. We no longer mess around with discussions of local storage formats
and how to convert to and from them. We now focus on what's on the wire and
leave the implementation details to implementors.

MIME has some of this too, but not as bad.

In hindsight we should not have included this sort of language, but it seemed
like a good idea at the time.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Wed, 28 Nov 2012 14:07:59 -0800 (PST)

> to identify them and file errata for them.)

It's not an unreasonable change but it it does not, as I understand the current
rules, rise to the level of calling for an erratum.

> 4. It might be worthwhile to examine RFC 3030 for errata along these lines.

It might be possible to construct an errata there, but I doubt it. I think
a revision is really needed to fix this. But that raises the question of
whether or not it's worth it, which I also doubt.

				Ned