Subject: [ietf-dkim] canonicalized null body and dkim

From: Tony Hansen <[email-masked]>
Date: Thu, 14 Dec 2006 19:20:20 -0500

Mike Thomas and I have been having a conversation about how empty
message bodies should be canonicalized.

Go and reread section 3.4.3. There is text in there that was added to
handle the case of a message body that has been transmitted using
CHUNKING and doesn't have a CRLF at the end of its last line. It
specifies that a missing CRLF must be added to the last line before
passing it to the canonicalizer.

Prior to adding this text, I think that an empty message body would be
canonicalized into an empty input into the canonicalizer. However, a
careful reading of the text in section 3.4.3 indicates that it *would*
have the side affect of adding a CRLF to an empty body as well.

Personally, I think this is an incorrect side affect, and an empty body
should have an empty canonicalization. Several implementations already
canonicalize an empty body into an empty string.

I know we've hit and passed LC, but I think this issue needs a
clarification, one way or another way, before we go to RFC status.

I'd like to see either one of these two statements added to the end of
section 3.4.3:

   a)	As a limiting case, an empty message body would be canonicalized
into a single CRLF pair.

   b)	An empty message body should be canonicalized into an empty string.

	Tony Hansen
	[email-masked]
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 15 Dec 2006 13:17:29 -0000

>    b)	An empty message body should be canonicalized into an empty string.
But isn't even the simple canonicalization then supposed to discard blank  
lines at the end of the message, thus rendering those two the same?

Or what am I missing?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Tony Hansen <[email-masked]>
Date: Fri, 15 Dec 2006 10:30:43 -0500

You're missing this statement:

   "In more formal terms, the "simple" body canonicalization algorithm
    converts "0*CRLF" at the end of the body to a single "CRLF"."

This states *how*

	Tony Hansen
	[email-masked]

Charles Lindsey wrote:
> But isn't even the simple canonicalization then supposed to discard
> blank lines at the end of the message, thus rendering those two the same?
> 
> Or what am I missing?
> 
> --Charles H. Lindsey ---------At Home, doing my own thing------------------------

---

From: Tony Hansen <[email-masked]>
Date: Fri, 15 Dec 2006 10:32:35 -0500

(Sorry for the truncated message previously. I hit "send" accidentally.)

This states *how* the blank lines at the end of the message are to be
discarded.

	Tony Hansen
	[email-masked]

---

From: Michael Thomas <[email-masked]>
Date: Fri, 15 Dec 2006 07:58:39 -0800

> canonicalize an empty body into an empty string.

Note that l=0 should still mean that the final potentially virtual CRLF 
would
not be added to the hash. I'm not proposing that as a solution, just an
observation.

The thing that I've seen is that at the very least sendmail will strip a 
body
with a single CRLF to a null body in its output stage; I would not be
surprised to hear that other MTA's do similar transformations. For something
either like a milter or something simply upstream of those mungers, the
net effect is that the way the current spec is written, they will still pass
verification. If we change that, they will not. That to me is a good reason
to keep it the way it currently reads, but add an implementation note to
the effect.

Just to be clear: this is a fairly common edge situation -- all one has to
do to reproduce this is to type a subject line in Thunderbird with no
body and it will produce a message body with a single CRLF. Other
MUA's may well make the same mistake, and at Cisco we're seeing
about 5k or so of those kinds of messages a day.

       Mike

---

From: Eric Allman <[email-masked]>
Date: Fri, 15 Dec 2006 09:24:36 -0800

--On December 15, 2006 7:58:39 AM -0800 Michael Thomas 
<[email-masked]> wrote:

> The thing that I've seen is that at the very least sendmail will
> strip a body with a single CRLF to a null body in its output stage;
> I would not be surprised to hear that other MTA's do similar
> transformations.

Um...  to the best of my knowledge, sendmail will never strip any 
data, including trailing CRLFs.  It will /add/ a CRLF under certain 
circumstances, which include when mailing to a local v7 mailer that 
requires a double CRLF between messages.

However, Tony is completely right about the ambiguity, except that 
it's actually larger than he says --- it appears the draft doesn't 
clearly specify whether you canonicalize then truncate, or truncate 
then canonicalize.  The only one that actually makes sense from an 
interoperability sense is to canonicalize and then truncate.  By this 
reading, l=0 would not include a trailing CRLF.

Is my logic logical?

eric

---

From: Hector Santos <[email-masked]>
Date: Fri, 15 Dec 2006 13:37:21 -0500

Eric Allman wrote:

> Um...  to the best of my knowledge, sendmail will never strip any data, 
> including trailing CRLFs.  It will /add/ a CRLF under certain 
> However, Tony is completely right about the ambiguity, except that it's 
> actually larger than he says --- it appears the draft doesn't clearly 
> specify whether you canonicalize then truncate, or truncate then 
> canonicalize.  The only one that actually makes sense from an 

I would this so.

One of the reasons I wrote the STRIP cl4n I-D was to normalize 
everything without <CR><LF> interoperabiliy known or unknown issues.

http://tools.ietf.org/wg/dkim/draft-santos-dkim-strip-00.txt

The STRIP c14n method is similar to the RELAXED c14n with the added step 
of removing the trailing white spaces in the hashing engine feed.

Anywho, I think it is vital that DKIM-BASE design issues with no 
documented ambiguities are squared away.  DKIM is already too complex 
(and costly) for adoption considerations.  Technical issues need to 
resolved.

---
HLS

---

From: Arvel Hathcock <[email-masked]>
Date: Mon, 18 Dec 2006 10:45:35 -0600

> Go and reread section 3.4.3 [snip] It specifies that a missing CRLF
 > must be added to the last line before passing it to the canonicalizer.

Right.  This is how our implementation is doing it because the current 
spec says to do it that way (at least for simple).  We're also doing it 
for relaxed which might be a mistake?  The spec doesn't say to do this 
for relaxed (at least not in 3.4.4).

-- 
Arvel

---

From: Tony Hansen <[email-masked]>
Date: Mon, 18 Dec 2006 11:50:49 -0500

Yes, but what if all that's in your body are empty lines? What then? Do
you wind up with nothing? Or a single CRLF?

	Tony Hansen
	[email-masked]

---

From: Michael Thomas <[email-masked]>
Date: Mon, 18 Dec 2006 09:34:18 -0800

Eric Allman wrote:
>
>
> --On December 15, 2006 7:58:39 AM -0800 Michael Thomas <[email-masked]> 
> wrote:
>

An update: just proving that none of this stuff can be simple... I 
traced between
my home sendmail and cisco's ironports, and you are right: sendmail does 
keep
that CRLF in the body. However, something between our DKIM verifier and
one of the inbound Ironport chopped that CRLF off. In reality it looks like:

mtcc.com(sendmail) --> inbound.cisco.com(ironport) --> core 
(sun/sendmail) --> dkim (linux/sendmail)

somewhere beyond the first hop, the last CRLF got chopped.
also:

mail.com(sendmail) -> deliver (procmail local)

is chopping off the CRLF as well.

So it may be that local delivery *and* something in Cisco's core is 
doing this. Which
kind of points the finger at the Ironportm but who knows. I don't have 
access to the core
sendmail configs, but the dkim sendmail configs don't leap out to me 
that they might be doing
something odd.

In any case, this weirdness says that regardless of intent the current 
canonicalization as
written is actually more robust, and we should keep it that way.

       Mike

---

From: Arvel Hathcock <[email-masked]>
Date: Mon, 18 Dec 2006 11:39:33 -0600

Let me do some additional fact checking to be sure I give the correct 
answer.  I'll report back soon.

-- 
Arvel

---

From: Arvel Hathcock <[email-masked]>
Date: Mon, 18 Dec 2006 20:07:54 -0600

Ok, I'm very sorry for the confusion on this topic.  We did a code 
review and found that what I said before was wrong.

In our implementation, an empty body leads to hashing of nothing (no 
CRLF).  I think this goes against 3.4.3.

I'm going to wait for the final verdict on what I should be doing and 
then make a correction to our code based on this groups collective wisdom.

-- 
Arvel

Arvel Hathcock wrote:
 >> Yes, but what if all that's in your body are empty lines? What then?
 >> Do you wind up with nothing? Or a single CRLF?