Subject: Re: [ietf-dkim] canonicalized null body and dkim

From: Eric Allman <[email-masked]>
Date: Sun, 07 Jan 2007 10:49:50 -0800

I have (finally) managed to slog my way through all the messages on 
this topic.  Let me start out by saying that I don't see the 
ambiguity in the current text:

        If there is no trailing CRLF on the message, a CRLF is added.
        It makes no other changes to the message body. In more formal
        terms, the "simple" body canonicalization algorithm converts
        "0*CRLF" at the end of the body to a single "CRLF".

So if the message ends without a CRLF (which should only be possible 
using CHUNKING) one gets added.  In particular, this is important 
because if a message is sent using CHUNKING through one relay and 
DATA through another, the CRLF will have to be added to get the 
<CRLF>.<CRLF>.

Based on the subject of the original message in this thread, the 
issue is actually what happens if the <CRLF><CRLF> separating the 
header from the body is missing.  Even that is (I think) well 
defined.  The first <CRLF> terminates the final header field and is 
required by 2822 --- that is, if it is missing then the message is 
malformed.  There is no reason that we should define this.  The 
second is the blank line that separates the header from the body, but 
it is not part of either.  If it is missing, then there is no body, 
equivalent to a body length of zero.  That clearly has no trailing 
CRLF, so one is added.  See the examples at the end of this message.

In an attempt to clarify this, I added the following at the end of 
3.4.3:

        Note that a completely empty or missing body is canonicalized
        as a single "CRLF"; that is, the canonicalized length will be
        2 octets.

The l= tag doesn't come into it, since the body is canonicalized 
before l= applies.  There does seem to be some confusion about this, 
so I made a one word change to section 3.7:

        In hash step 1, the signer/verifier MUST hash the message
        body, canonicalized using the body canonicalization algorithm
        specified in the "c=" tag and then truncated to the length
        specified in the "l=" tag.

Specifically, I added the word "then" to try to clarify that you 
canonicalize first, then truncate.

There was some discussion of back compatibility with DK, but this 
seems irrelevant given the addition of bh=, since that's a new, 
separate hash.

eric

Examples using DATA:

    INPUT                               OUTPUT (BODY MARKED)

    Field: foobar<CRLF>                 Field: foobar<CRLF>
    .<CRLF>                             ----
                                        <CRLF>
                                        ----

    Field: foobar<CRLF>                 Field: foobar<CRLF>
    <CRLF>                              ----
    .<CRLF>                             <CRLF>
                                        ----
    Field: foobar<CRLF>                 Field: foobar<CRLF>
    <CRLF>                              ----
    <CRLF>                              <CRLF>
    .<CRLF>                             ----

    Field: foobar<CRLF>                 Field: foobar<CRLF>
    <CRLF>                              ----
    .<CRLF>                             <CRLF>
                                        ----
    Field: foobar<CRLF>                 Field: foobar<CRLF>
    <CRLF>                              ----
    <CRLF>                              <CRLF>
    <CRLF>                              ----
    <CRLF>
    <CRLF>
    .<CRLF>
                                        ----
    Field: foobar<CRLF>                 Field: foobar<CRLF>
    <CRLF>                              ----
    baz<CRLF>                           baz<CRLF>
    <CRLF>                              ----
    <CRLF>
    .<CRLF>

Examples using BDAT (CHUNKING):

    INPUT                               OUTPUT (BODY MARKED)

    Field: foobar                       <error: undefined>

    Field: foobar<CRLF>                 Field: foobar<CRLF>
                                        ----
                                        <CRLF>
                                        ----

    Field: foobar<CRLF>                 Field: foobar<CRLF>
    <CRLF>                              ----
                                        <CRLF>
                                        ----

    Field: foobar<CRLF>                 Field: foobar<CRLF>
    <CRLF>                              ----
    baz                                 baz<CRLF>
                                        ----

_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 08 Jan 2007 11:51:13 -0000

On Sun, 07 Jan 2007 18:49:50 -0000, Eric Allman <[email-masked]>  
wrote:

> I have (finally) managed to slog my way through all the messages on this  
> topic.  Let me start out by saying that I don't see the ambiguity in the  
> current text:
>
> because if a message is sent using CHUNKING through one relay and DATA  
> through another, the CRLF will have to be added to get the <CRLF>.<CRLF>.

Indeed there is no ambiguity in that, but that is because you have only  
quoted half the text. The full text is:

    The "simple" body canonicalization algorithm ignores all empty lines
    at the end of the message body.  An empty line is a line of zero
    length after removal of the line terminator.  If there is no trailing
    CRLF on the message, a CRLF is added.  It makes no other changes to
    the message body.  In more formal terms, the "simple" body
    canonicalization algorithm converts "0*CRLF" at the end of the body
    to a single "CRLF".

Observe carefully that the text some times tells you to consider the  
"message", and somtimes the "message body" (which I take to mean exactly  
the <body>, if any, defined by RFC 2822).

Consider the example, in DATA format:

    Field: foobar<CRLF>.
    <CRLF>
    <CRLF>
    .<CRLF>

The ".<CRLF>" is evidently not a part of either the message or of the  
message body. The "message body" consists of "<CRLF>". Let us apply the  
sentences of 3.4.3 one by one.

    The "simple" body canonicalization algorithm ignores all empty lines
    at the end of the message body.

To see what an "empty line" is we need one more sentence;

    An empty line is a line of zero
    length after removal of the line terminator.

So the line "<CRLF>" (which is the whole of the message body) IS an empty  
line.
That empty line is at the end of the message body, so we ignore it. That  
leaves the message

    Field: foobar<CRLF>.
    <CRLF>

Take the next sentence:

    If there is no trailing
    CRLF on the message, a CRLF is added.

There IS a trailing CRLF on the message (NB, it does not say "message  
body" there), so we add nothing. We still have:

    Field: foobar<CRLF>.
    <CRLF>

Take the next sentence:

    It makes no other changes to
    the message body.

So we are finished. We have a message with an empty <body>, so that empty  
<body> is what we hash.

Take the next sentence:

    In more formal terms, the "simple" body
    canonicalization algorithm converts "0*CRLF" at the end of the body
    to a single "CRLF".

That is supposed to produce the same result, so start over with the  
original message:

    Field: foobar<CRLF>.
    <CRLF>
    <CRLF>
    .<CRLF>

of which we have already seen that "<CRLF>" is the body.

Indeed if contains ""0*CRLF" at its end (in fact, it contains 1*CRLF), so  
we convert it to "<CRLF>", and that is what we hash.

Therefore, the description in the first four sentences produces a  
different result to the supposedly identifal description in the fifth  
sentence.

Q.E.D.

Now it appears that some implementations have followed one interpretation  
and some the other, so something needs to be fixed. My suggested wording  
is:

    The "simple" body canonicalization removes empty lines from the end of  
the
    body until either the last line is non-empty, or no lines remain. An  
empty
    line is a line of zero length after removal of any terminating CRLF. If
    the body is not now empty and the last line is not already terminated by
    CRLF, a CRLF is added to it.

       INFORMATIVE NOTE: Following [RFC 2822}, the CRLF which separates the
       header fields from the body is NOT part of the body, and therefore is
       never presented to the signing or verification algorithm. In the case
       of a pure binary message (such as one with a  
Content-Transfer-Encoding
       of 'binary') the concept of "lines" may not be meaningful.  
Nevertheless,
       wherever the pair of octets that represent CRLF happens to occur,  
that
       is to be considered as the end of a "line" for the purposes of this
       canonicalization algorithm.

That follows what I consider to be both the spirit and the letter of the  
first four sentences, at the expense of ignoring and renmoving the fifth  
sentence.

In particular, it leads to the easily remembered invariant:

    "After canonicalization, there will NEVER be an empty line at the end
     of what remains to be hashed."

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Mon, 08 Jan 2007 07:37:43 -0500

> Consider the example, in DATA format:

Isn't this mal-formed message?  No need to go any futher.

---
HLS

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 08 Jan 2007 18:50:33 +0100

> Isn't this mal-formed message?  No need to go any futher.

Guessing:  Charles actually wants this without the trailing dot:

| Field: foobar<CRLF>
| <CRLF>
| <CRLF>
| .<CRLF>
 
Frank

---

From: Tony Hansen <[email-masked]>
Date: Mon, 08 Jan 2007 13:54:37 -0500

1) Ignore the "." at the end of the header Field: foobar<CRLF>
2) Loop back and re-read the rest of the message

	Tony Hansen
	[email-masked]