Subject: Re: [ietf-dkim] canonicalized null body and dkim

From: Eric Allman <[email-masked]>
Date: Mon, 08 Jan 2007 09:27:49 -0800

--On January 8, 2007 11:51:13 AM +0000 Charles Lindsey 
<[email-masked]> wrote:

> On Sun, 07 Jan 2007 18:49:50 -0000, Eric Allman
> <[email-masked]>  wrote:
>
>> I have (finally) managed to slog my way through all the messages
>> on this   topic.  Let me start out by saying that I don't see the
>> ambiguity in the   current text:
>>
>>         If there is no trailing CRLF on the message, a CRLF is
>>         added. It makes no other changes to the message body. In
>>         more formal terms, the "simple" body canonicalization
>>         algorithm converts "0*CRLF" at the end of the body to a
>>         single "CRLF".
>>
> ...
>
> Indeed there is no ambiguity in that, but that is because you have
> only  quoted half the text. The full text is:
>
>     The "simple" body canonicalization algorithm ignores all empty
> lines
>     at the end of the message body.  An empty line is a line of zero
>     length after removal of the line terminator.  If there is no
> trailing
>     CRLF on the message, a CRLF is added.  It makes no other
> changes to
>     the message body.  In more formal terms, the "simple" body
>     canonicalization algorithm converts "0*CRLF" at the end of the
> body
>     to a single "CRLF".
>
> Observe carefully that the text some times tells you to consider
> the  "message", and somtimes the "message body" (which I take to
> mean exactly  the <body>, if any, defined by RFC 2822).

OK, I'll change the single instance of "message" to "message body".

> Consider the example, in DATA format:
>
>     Field: foobar<CRLF>.
>     <CRLF>
>     <CRLF>
>     .<CRLF>
>
> The ".<CRLF>" is evidently not a part of either the message or of
> the  message body. The "message body" consists of "<CRLF>". Let us
> apply the  sentences of 3.4.3 one by one.

As Hector points out, this is not a valid 2822 message to begin with, 
so sending it would be undefined.  At that point how it gets 
canonicalized is the least of your problems.

I can't come up with a scenario that starts with a valid message that 
has the problem you describe.  None the less, changing "message" to 
"message body" will improve consistency, and it would make 
interpretation of this degenerate case consistent.

eric 
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 09 Jan 2007 11:23:44 -0000

> OK, I'll change the single instance of "message" to "message body".

Well that certainly fixes the case I gave, but then the question arises as  
to whether it has fixed it in the "right" direction. I still, cannot see  
why one would want to canonicalize a genuinely empty <body> intpo "CRLF",  
nor do some others on this list. I gather that some implementations do not  
do so either, and neither did the original DK, and many on the list  
thought they were standardizing the DK behaviour.

Moreover, there remains another case that is ambiguous. Consider:

      Field: foobar<CRLF>
      .<CRLF>

That is a valid RFC 2822 message with NO <body> at all (which is NOT the  
same thing as an empty <body>). Let us apply your revised wording.

    The "simple" body canonicalization algorithm ignores all empty lines
    at the end of the message body.

There is no body, so no action is needed.

    An empty line is a line of zero
    length after removal of the line terminator.

Not needed.

    If there is no trailing
    CRLF on the message BODY, a CRLF is added.

That is your modified wording. There is no <body> so no action occurs.  
(But it would have been the same with your original wording.)

    It makes no other changes to
    the message body.

There is no <body>, so there is nothing to not make any changes to. But we  
are still finished.

So what do we pass to the canonicalization? It doesn't say, but the only  
reasonable intpretation would be to pass <empty>. So it appears that an  
absent body canonicalizes differently to an empty body.

Now let us compare that with the final sentence:

    In more formal terms, the "simple" body
    canonicalization algorithm converts "0*CRLF" at the end of the body
    to a single "CRLF".

Since there is no <body>, there is nothing to do, so it indeed agrees with  
the first 4 sentences in this case.

But we still have the bizarre situation that an absent body is treated  
differently from an empty body. Can you please confirm that this was your  
intention?

> As Hector points out, this is not a valid 2822 message to begin with, ...

Sure, there was a typo, as various people spotted.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Tue, 09 Jan 2007 07:12:05 -0500

> same thing as an empty <body>). Let us apply your revised wording.

Unless there is some other different notation I am unaware of, it is 
*not* a valid RFC 2822 message.  I believe you meant:

       Field: foobar<CRLF>
       <CRLF>
       .<CRLF>

Why is this important?  Well, many reasons, but one good reason is that 
there are lot of code that search for the first "\r\n\r\n" to find the 
header/body separator, i.e,  strstr(buffer, "\r\n\r\n");

So we can't continue trying to make sense of mal-formed messages.
> There is no body, so no action is needed.

But there is a body.  It has a dot. Most likely an useless body, but 
body nonetheless.   Who knows? Maybe a "single dot" message means 
something to some one other there.  Maybe it means "Come Home, its late 
and stop by the store and pick up some mike."

>    length after removal of the line terminator.

I think in this regard, being "specific is terrific".

> differently from an empty body. Can you please confirm that this was 
> your intention?

I disagree it is "absent".  It isn't. There is a dot.  It could of been 
a Q or a X, or Z or slash.  Lets not begin making judgments on the 
quality of message bodies.

As far as I am concern, we must resolve the deterministic design, not 
the abstract design.  The technique question for me was if the SIMPLE 
cl4n must end with <CRLF> bytes.  I see the following:

    L=0, no hashing is done,
    L=1, is not possible or is it?
    L=2, means we have a NULL body according to Erik, or is it?
    L=3, 1 byte message, or is it?

In other words, it is not technically impossible from a signing stand 
point to perform a SIMPLE cl4n with a large body w/o <crlf>, and the 
signer only hashes L=X bytes where X is less than the canonicalized text 
size.

In other words, you can have 5000 bytes in the message body and still 
say l=2.

According to Eric message, he cleared up this technique question for me.

During verification, you only need to canonicalized up to l=X size, you 
don't need to worry about going any further in the feed.

---
HLS

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 9 Jan 2007 08:04:31 -0800

At 11:23 AM +0000 1/9/07, Charles Lindsey wrote:
>I still, cannot see why one would want to canonicalize a genuinely 
>empty <body> intpo "CRLF", nor do some others on this list. I gather 
>that some implementations do not do so either, and neither did the 
>original DK, and many on the list thought they were standardizing 
>the DK behaviour.

It doesn't matter which direction a canonicalization goes, as long as 
it is completely clear what to do.

Is this horse dead yet? With Eric's proposed change, are there any 
open issues? Tony posted a somewhat-cryptic note yesterday; is that 
resolved?

--Paul Hoffman, Director
--Domain Assurance Council
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 09 Jan 2007 17:13:40 +0100

Hector Santos wrote:

> Unless there is some other different notation I am unaware of,
> it is *not* a valid RFC 2822 message.

Of course it is, the body is optional, see 2822 ch. 3.5:

| message = (fields / obs-fields) [CRLF body]

> there are lot of code that search for the first "\r\n\r\n" to
> find the header/body separator, i.e,  strstr(buffer, "\r\n\r\n")

If they don't find this there's no body, that's ugly but allowed.
It's the way how spam probes end up as "empty" mails, the spammer
sends C DATA
      S 3xx end with dot
      C .
      S 2xx got mail for valid address

The user sees the Return-Path, a timestamp line, an artificial
2822-From derived from the Return-Path, and an artificial Date
(typically the same as in the timestamp).  And maybe also an
artificial Message-ID, plus an empty Bcc.

> But there is a body.  It has a dot.

No, Charles transcript is the DATA, a real dot would appear as
<CRLF>..<CRLF> in this style.  A single dot is the end of DATA.

I don't care about the canonicalization of spam probes or other
mails without body, but the issue is no nonsense.

Frank

---

From: Hector Santos <[email-masked]>
Date: Tue, 09 Jan 2007 17:08:25 -0500

Frank Ellermann wrote:
> sends
>     C DATA

But that means no headers too, a totally blank message which depending 
on how anal the SERVER is, may or may not accept or it will generate the 
a new header on its own based on the 2821 envelope, plus a message-id.

But in either case, it is not a DKIM message and if its a user 
submission, again, depending on the server option, it may not be 
accepted or the headers auto generated with a NULL or empty body. No 
problem here DKIM wise.

> mails without body, but the issue is no nonsense.

I'm not grasping the problem.  I think the issue is being overblown. The 
only question for me was how read or set a l=X for the SIMPLE c14n. Eric 
cleared that up for me.

---
HLS

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 06:30:05 +0100

Hector Santos wrote:

> I'm not grasping the problem.

Apparently the canonicalization of "no body" (no CRLF CRLF) is
different from "empty body" (only CRLF CRLF).  It's odd enough
to mention it in the spec. (maybe as note or example).
 
> I think the issue is being overblown.

If your implementation treats "no body" like "empty body" and
other implementations don't we've to agree on one way to get
this right.  Or rather Eric has to describe it unambiguously.

Frank

---

From: Hector Santos <[email-masked]>
Date: Wed, 10 Jan 2007 10:52:28 -0500

Charles Lindsey wrote:

> The example of interest is
>    C DATA
>    S 3xx end with dot
>    C Subject: lots of headers here, but I an just showing one<CRLF>
>    C .<CRLF>
>    S 2xx valid message received
>
> That is a valid messge with an empty body. Stupid but valid and
> therefore DKIM-base should tell us how to sign it.

So your notation was the SMTP client transmission.  Forgive me if I
missed this.

Nevertheless, the storage is whats important, and based on our server
experiences, it could be one of the following depending on the server
idiosyncrasies:

     headers:<crlf><crlf><crlf><eof>
     headers:<crlf><crlf><eof>
     headers:<crlf><eof>   (Probably less likely)

I don't see the problem with SIMPLE c14n hashing any one of these.

> Hashing the headers
> is no problem; what to hash for the body?

In terms of the SIMPLE cl4n, they all yield a "<CRLF>" for hashing.
Isn't this the key point?

> The alternatives are to hash <empty> which, with sha-256, gives
>    47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU
> or to hash <CRLF>, which gives
>    frcCV1k9oG9oKj3dpUqdJg1PxRT2RSN/XKdLCPjaYaY

But the first one isn't following SIMPLE c14n rules. Right?  That would
be a

> There are two questions:
>
> 1. Should absence of a body and an empty body result in the same thing
> being hashed? I would have thought the answer to that would be a
> resounding YES.

Sure, so whats the problem either way?   I believe your question is
whether the originality of a single byte message is important?

> 2. Should the thing hashed in those two cases be <empty> or <CRLF>?

 From my understanding, for SIMPLE c14n, they both have a <crlf> FEED to
into the hashing engine.

> My preference would be <empty> (in both cases). I gather that
> is what DK does, and what some implementations of DKIM do.

Then the DKIM implementations are not following the SIMPLE c14n rule.
No problem. Its all ALPHA ware, so they can fix it before production.

> So, if <empty> is to be hashed, you can only write l=0.
> If <CRLF> is to be hashed, you can write l=0, l=1 or l=2.

Which was part of my questions before about all this.  I don't think l=1
is valid for a SIMPLE c14n algorithm.

The question is really about how important it is in preserving the
originality of the body data for a SINGLE <CRLF> byte message or lack
there of from a SMTP CLIENT transmission.

Regardless of how it is stored by the receiver, from a SIMPLE c14n
normalization standpoint,I don't see the issue as it is written and Eric
cleared up.  The message body when canonicalized, all return <CRLF>

Maybe it would be clearer to have a dedicated section in the specs that
defines two new terms

     DKIM MESSAGE
     DKIM NULL MESSAGE

For semantic reasons, NULL or EMPTY is the same in terms of how the
message c14n.   The difference with DKIM MESSAGE and 2822 MESSAGE is
that the DKIM MESSAGE is canonicalized.    I think that a DKIM NULL
MESSAGE will have a constant hash (using your base64 output) of:

	frcCV1k9oG9oKj3dpUqdJg1PxRT2RSN/XKdLCPjaYaY

The section can describe that when L=2, then is exactly what the SIMPLE
hash should be.
	
Also, in 3.4.3, the sentence

    It makes no other changes to the message body.

Should be changed to:

    It makes no other changes to the HASHING algorithm feed.

In any case, I think it is overblown, because each SERVER will handle
the SMTP-DATA empty message in their own ways, and in my opinion, the
SIMPLE c14n rule does handle each one the same.

 From my standpoint, I want to optimized the verification overhead.  So
just consider the logic depending on L=X is:

   L=0     no hashing of DKIM MESSAGE
   L=1     indeterminate
   L=2     DKIM NULL MESSAGE

---
HLS

---

From: Hector Santos <[email-masked]>
Date: Wed, 10 Jan 2007 12:12:46 -0500

> this right.  Or rather Eric has to describe it unambiguously.

Doesn't both translate to a <CRLF> for the SIMPLE c14n?

 From a signing standpoint, the signer implementation can decide for 
itself if it wants to sign a DKIM NULL MESSAGE (one that canonicalizes 
to <CRLF> only).  So here, you will have either l=0 or l=2.

 From a verification standpoint, the verifier has the design option to 
look at the l=X value to check for any DKIM NULL MESSAGE consideration.

    l=0   no hashing, bh= should be blank or not available
    l=1   SHOULD NOT HAPPEN. invalid verification
    l=2   DKIM NULL MESSAGE (see note 1)

Note 1:

The question here is whether we *SHOULD* allow a DKIM NON-NULL MESSAGE 
to have a l=2 value.  It is kinda ridiculous and it is the worst case 
scenario for the Truncation Threat that is already documented in the 
threats document.  Example, even is a body canonicalizes to:

      12345678<CR><LF>
      12345678<CR><LF>
      12345678<CR><LF>
      12345678<CR><LF>
      <CR><LF>

is a l=2 acceptable which means only the first 2 bytes are feed into the 
hashing engine?

Maybe we can't do anything about this and it is what it is.

In any case, algorithm wise, I think it is clear at this point, with 
Eric's additional changes.

---
HLS

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 21:03:43 +0100

> Doesn't both translate to a <CRLF> for the SIMPLE c14n?

I'd hope that they do this, a canonicalization should eliminate
this rather obscure difference.  But if Charles reads the spec.
in a way where that's apparently not the case, then maybe Eric
can find some better wording for this detail.  In AUTH48, this
is editorial and completely unrelated to Lisa's old DISCUSS.

Frank

---

From: Eric Allman <[email-masked]>
Date: Wed, 10 Jan 2007 12:11:55 -0800

I've changed the text to read "If there is no body or no trailing 
CRLF on the message body, a CRLF is added" in order to avoid any 
confusion.

eric

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 10 Jan 2007 21:33:50 +0100

> CRLF on the message body, a CRLF is added" in order to avoid any

Thanks.

---

From: Hector Santos <[email-masked]>
Date: Wed, 10 Jan 2007 15:34:27 -0500

Eric Allman wrote:

> I've changed the text to read "If there is no body or no trailing CRLF 
> on the message body, a CRLF is added" in order to avoid any confusion.

Eric, this is good. I will comment that the only confusion is there is 
an indirect implication that the final two bytes of the FEED must have a 
<CRLF>.

Which is not true, right?

Because a) A signer can canonicalized the message, including adding a 
<CRLF> if necessary, that yields size X, but B) the signer decides to 
hash only L=N  bytes where N < X bytes.

That was the only "scratch head" thought I had about all this - The 
reason to add a <CRLF> if a signer was not going to hash the entire body.

So I guess, if anything, a statement, sentence or comment that states 
"the need to add a <CRLF> is only necessary if the entire body is going 
to be hashed."

---
HLS

---

From: Eric Allman <[email-masked]>
Date: Wed, 10 Jan 2007 12:55:33 -0800

--On January 10, 2007 3:34:27 PM -0500 Hector Santos 
<[email-masked]> wrote:

> Eric, this is good. I will comment that the only confusion is there
> is an indirect implication that the final two bytes of the FEED
> must have a <CRLF>.
>

I'm not sure what you mean by the "FEED" here.  If you mean the data 
transmitted using either DATA or BDAT you can have a message without 
a trailing CRLF using BDAT.

> Because a) A signer can canonicalized the message, including adding
> a <CRLF> if necessary, that yields size X, but B) the signer
> decides to hash only L=N  bytes where N < X bytes.
>

It's true that if the signer uses l= then you don't actually have to 
add the trailing CRLF --- in fact, it doesn't have to canonicalize 
anything at all after it has sent N bytes (where l=N).  We could say 
that, but it would be an implementation detail that probably doesn't 
belong in the spec, and certainly not in normative language.

> So I guess, if anything, a statement, sentence or comment that
> states "the need to add a <CRLF> is only necessary if the entire
> body is going to be hashed."

See above.

eric

---

From: Tony Hansen <[email-masked]>
Date: Thu, 11 Jan 2007 09:42:19 -0500

Deciding to skip adding those two bytes for the case of N < X-2 is
purely an optimization.* Your implementation is free to make any
optimization that it chooses to do, as long as the result is
indistinguishable from calculating it in the unoptimized way.

I think that where Eric placed these words make it clear that adding the
CRLF is done during the canonicalization phase, and that the l=
truncation is performed off of the canonicalized text.

I'm happy with this last addition, and I think it polishes off the
issue. "Ship it, Dano." (And now I'm off to change my implementation to
match. :-) )

	Tony Hansen
	[email-masked]

* where N and X come from l=N and X=length(canonicalized(body))

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 11 Jan 2007 15:43:16 +0000

> issue. "Ship it, Dano." (And now I'm off to change my implementation to

If Tony's happy, then I suspect we should all be happy:-)

So, let's consider this issue as closed now. If that's wrong,
please start by telling Barry & I that off-list and we can
bring it back to the list if necessary,
Ta,
Stephen.
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html