Subject: [Fwd: Re: [ietf-dkim] canonicalized null body and dkim]

From: Mark Delany <[email-masked]>
Date: Fri, 15 Dec 2006 10:17:16 -0800

Tony Hansen wrote:
> (Sorry for the truncated message previously. I hit "send" accidentally.)
> 
> You're missing this statement:
> 
>    "In more formal terms, the "simple" body canonicalization algorithm
>     converts "0*CRLF" at the end of the body to a single "CRLF"."
> 
> This states *how* the blank lines at the end of the message are to be
> discarded.

FWIW, this problem was similarly discovered in DK. The early text read:

-01
    o All trailing empty lines are ignored. An empty line is a line of
       zero length after removal of the local line terminator. The
       empty line that separates the header from the body is a to be
       included in this process.

and the later text read:

-06
     o All trailing empty lines are ignored. An empty line is a line of
       zero length after removal of the local line terminator.

       If the body consists entirely of empty lines, then the
       header/body line is similarly ignored.

In short, if the last empty line of the email is the header/body 
separator, then it should not be fed into the canonicalization.

The "simple" in DKIM, as I understand it, is merely re-codifying the
same function.

Mark.

_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Eric Allman <[email-masked]>
Date: Fri, 15 Dec 2006 11:07:47 -0800

> The "simple" in DKIM, as I understand it, is merely re-codifying the

I *think* there are three cases here:

(1) Some body with l=0.  I think we're agreed that this should result 
in an empty input into the hash algorithm.

(2) Header, CRLF, no body; that is, the input to a DATA would be:

        Header: foobar<CRLF>
        <CRLF>
        <CRLF>.<CRLF>

The last line of the example is the separator, so the initial CRLF 
doesn't count.

(3) Header, no CRLF, no body:

        Header: foobar<CRLF>
        <CRLF>.<CRLF>

It seems to me that all three of these should match after 
canonicalization.  I'm not sure the -07 draft is explicit enough to 
ensure this.

In short, I'm not sure we have guarantied DK compatibility based on 
the current wording, regardless of intent.

eric

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 18 Dec 2006 11:56:59 -0000

>> The "simple" in DKIM, as I understand it, is merely re-codifying the

Nope!

> (1) Some body with l=0.  I think we're agreed that this should result in  
> an empty input into the hash algorithm.
>

Nope!

Let us be VERY careful here. Start from RFC 2822:

message         =       (fields / obs-fields)
                         [CRLF body]
body            =       *(*998text CRLF) *998text

So a <body> can be EMPTY, and its last line might not have a CRLF.

The CRLF following the header fields is NOT part of the <body>.

If the <body> is absent (indistinguishable from an empty <body>) that CRLF  
after the header fields can be omitted.

Now look at RFC 2821:

    The mail data is terminated by a line containing only a period, that
    is, the character sequence "<CRLF>.<CRLF>" (see section 4.5.2).  This
    is the end of mail data indication.  Note that the first <CRLF> of
    this terminating sequence is also the <CRLF> that ends the final line
    of the data (message text) or, if there was no data, ends the DATA
    command itself.

So, even if you have a body with no CRLF, as permitted by RFC 2822, you  
can't actually transmit it by RFC 2821 (well, you might transmit it by  
UUCP, and you might encapsulate in in a message/rfc822 within some  
multipart).

So we have the following cases. The dotted lines enclose what is, by RFC  
2822 definition, the <body>, and is therefore what will get hashed or  
canonicalized by dkim-base, as currently worded. The ".CRLF" is the RFC  
2821 DATA terminator.

1) ordinary message with <body> of 1 non-empty line:

Last-Header: foobarCRLF
CRLF
---------------------
barbazCRLF
---------------------
.CRLF

2) <body> consisting of 2 empty lines

Last-Header: foobarCRLF
CRLF
---------------------
CRLF
CRLF
---------------------
.CRLF

3) <body> consisting of 1 empty line

Last-Header: foobarCRLF
CRLF
---------------------
CRLF
---------------------
.CRLF

4) <body> containing no lines

Last-Header: foobarCRLF
CRLF
---------------------
---------------------
.CRLF

5) message with absent <body>

Last-Header: foobarCRLF
.CRLF

Now apply simple canonicalization to all those cases, using:

Making the entirely reasonable assumption that "body" means exactly what  
RFC 2822 defines it to mean, then here is what gets hashed in all of those  
cases:

1) ordinary message with <body> of 1 non-empty line:
---------------------
barbazCRLF
---------------------

2) <body> consisting of 2 empty lines
---------------------
CRLF
---------------------

3) <body> consisting of 1 empty line
---------------------
CRLF
---------------------

4) <body> containing no lines
---------------------
CRLF
---------------------

5) message with absent <body>
---------------------
---------------------

That is undoubtedly what the "formal terms" in dkim-base undoubtedly SAY.

It is NOT what the "informal" words in dkim-base say.
It is NOT what version -01 of DK says.
It is NOT what version -06 of DK says.
It is NOT what Eric's three examples claim.
It is entirely possible that is is NOT what dkim-base was INTENDED to say.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Mark Delany <[email-masked]>
Date: Mon, 18 Dec 2006 05:23:23 -0800

Charles Lindsey wrote:

> RFC 2822 defines it to mean, then here is what gets hashed in all of 
> those cases:
> 

(Did you mean to include Last-Header: in the following examples?)

> 1) ordinary message with <body> of 1 non-empty line:

I believe the intent is that 2, 3, 4 and 5 all canonicalize to the same 
content for c=simple, namely to match 5) as:

-------------------------------
Last-Header: foobarCRLF
-------------------------------

Mark.

Mark.

---

From: Tony Hansen <[email-masked]>
Date: Mon, 18 Dec 2006 10:35:29 -0500

Mark Delany wrote:
>> Making the entirely reasonable assumption that "body" means exactly
>> what RFC 2822 defines it to mean, then here is what gets hashed in all
>> of those cases:
>>
> 

Mark, Charles was intending to show the last header of the message and
the separating CRLF between it and the body. He's trying to nail down
the semantics of applying simple to the body alone, so wants to be clear
about which CRLF belongs to the header, which CRLF belongs to the
separator, and which CRLFs belong to the body. So the body in each
example would NOT include the octets "Last-Header: foobarCRLF".

	Tony Hansen
	[email-masked]

---

From: Mark Delany <[email-masked]>
Date: Mon, 18 Dec 2006 07:48:47 -0800

>> (Did you mean to include Last-Header: in the following examples?)

Sure. In which case the answer is that the canonicalized output of 2-5 
should be:

-----------------------
-----------------------

Mark.

---

From: Tony Hansen <[email-masked]>
Date: Mon, 18 Dec 2006 11:02:55 -0500

Charles, you missed the cases introduced by RFC 3030 and the CHUNKING
ESMTP extension.

Further comments below.

	Tony Hansen
	[email-masked]

Charles Lindsey wrote:
> If the <body> is absent (indistinguishable from an empty <body>) that
> CRLF after the header fields can be omitted.
> 
> Now look at RFC 2821:
> 
> can't actually transmit it by RFC 2821 

Correct, 2822 requires complete lines to be transmitted, that is, lines
ending in CRLF.

> (well, you might transmit it by

Add in RFC 3030 and CHUNKING, and you get the ways of transmitting
messages using ESMTP that are RFC 2822-compliant but not RFC
2821-compliant. Add in the MIME RFCs and you also eliminate the
requirements that the lines be limited to 998 characters and consist of
text.

> canonicalized by dkim-base, as currently worded. The ".CRLF" is the RFC

Correct.

> 1) ordinary message with <body> of 1 non-empty line:

I contend that the current wording in base-07 also requires that example
5 canonicalize into a

---------------------
CRLF
---------------------

Even when the body doesn't exist, it still must be treated as having 0
lines following, which still canonicalize to a CRLF.

But even with my contention on case #5, I don't disagree with your

> It is entirely possible that is is NOT what dkim-base was INTENDED to say.

That's why the issue was raised.

I firmly believe that we *intended* to canonicalize each of these cases
into the empty body

---------------------
---------------------

	Tony Hansen
	[email-masked]

PS. For completeness, the only missing cases, after taking into
consideration RFC 3030 and MIME, are as follows. *These* are the reason
that the 0*CRLF rule was added and where it needs to be applied:

6) ordinary message with <body> of >1 non-empty line, not ending in CRLF

Content-Type: binary
Last-Header: foobarCRLF
CRLF
---------------------
somethingCRLF
anything
---------------------

7) ordinary message with <body> of 1 non-empty line, not ending in CRLF

Content-Type: binary
Last-Header: foobarCRLF
CRLF
---------------------
anything
---------------------

This winds up adding a CRLF to the last line of both of these cases, so
here is what gets hashed in all of these additional cases:

6) ordinary message with <body> of >1 non-empty line, not ending in CRLF

---------------------
somethingCRLF
anythingCRLF
---------------------

7) ordinary message with <body> of 1 non-empty line, not ending in CRLF

---------------------
anythingCRLF
---------------------

---

From: Tony Hansen <[email-masked]>
Date: Mon, 18 Dec 2006 11:03:14 -0500

And that's where the contention comes in.

I firmly believe that what you write below is what we *intended*.
However, it is *not* what is described in the current spec.

	Tony Hansen
	[email-masked]

---

From: Michael Thomas <[email-masked]>
Date: Mon, 18 Dec 2006 09:01:39 -0800

> I firmly believe that we *intended* to canonicalize each of these cases
Well, that may or may not have been the intent, but the current spec 
does say that
and it's my claim that what the current spec is doing is *useful* given 
the mangling
that happens in the infrastructure. That is, we're getting better pass 
rates when we
implement the spec as it is written now.

       Mike

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 18 Dec 2006 18:41:27 +0000

I agree that this clarification should be added whenever we
get a consensus on what to say.

Can we also include the actual sha256 digest resulting (which
goes into bh=, or am I confused?) when we do this? That way
coders can check that they've gotten the right answer.

So the new note might go something like:

"With simple c14n the following examples all result in
the same body hash which has the value..."

S.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 19 Dec 2006 14:25:03 -0000

> I contend that the current wording in base-07 also requires that example
Well that's a philosophical issue :-( .
The present texts instructs you to do various things with "the body". If  
there is no body, then you can't do those things. So there is nothing to  
canonicalize and nothing to hash. The the only sensible thing to do in  
that case is to hash nothing. So I seem to disagree with you :-( .

But as to what we should actually do when this is sorted out should, I  
think, be that which causes the "least astonishment". So one would hardly  
expect the canonicalization to finish up with more lines that you started  
with.

The easiest rule for everyone to understand is an invariant such as:

    "After canonicalization, there will NEVER be an empty line at the end
     of what remains to be hashed."

> I firmly believe that we *intended* to canonicalize each of these cases
Which is indeed the "least astonishing".

> 6) ordinary message with <body> of >1 non-empty line, not ending in CRLF

I think those cases are covered by the rule that tells you to add a CRLF  
if there is no trailing CRLF anyway. But weird things can happen when you  
have binary. Here are some more cases for you all to chew over:

8) Binary text with assorted line endings:

Content-Type: binary
Last-Header: foobarCRLF
CRLF
---------------------
anythingLF
somethingCRLF
nothingCR
---------------------

So do I fix that last line by adding CRLF, or by adding just LF, or by  
leaving it alone?

Moreover, if I specify "l=1" in the Dkim-Signature, exactly which of those  
lines gets included in the process? And if I do any sort of  
canonicalization, is the count for "l=?" done before of after the  
canonicalization (I presume before).

And if I specify "l=0" and then treat my case (5) as you proposed to treat  
it, do I then have to hash that CRLF that appeared out of nowhere?

Canonicalization, you see, is tricky stuff to specify (as indeed is any  
cryptographic process). And agreeing the count for "l=?" is another of the  
things that has to be gotten right.

So, all in all, I think what is needed is an Appendix containing  
interesting test cases for which everybody needs to get the same results,  
and maybe another Appendix containing a model implementation (as the  
punycode people did, which is as well because nobody would have understood  
their algorithm without it).

---

From: Tony Hansen <[email-masked]>
Date: Tue, 19 Dec 2006 14:16:48 -0500

Michael Thomas wrote:
>> I firmly believe that we *intended* to canonicalize each of these
>> cases into the empty body
>>
> does say that and it's my claim that what the current spec is doing
> is *useful* given the mangling that happens in the infrastructure.
> That is, we're getting better pass rates when we implement the spec
> as it is written now.

Ah, but several of us are convinced that the current spec does *not* say
that and we want a clarification added one way or the other.

There's no one disputing that the current spec isn't useful.

If I read your message correctly, your implementation implements
canonicalizing a body with nothing but CRLFs into an empty body. Or is
there a missing "not" in your statement and I'm wrong?

*My* implementation does so. Arvel's does so.

But another implementation I know of changes it into a CRLF. Sigh. So we
are NOT currently getting compatible implementations with the current
wording.

Back to intents: I believe the intent of the extra CRLF rule was to
handle transport modifications to the body that added an extra CRLF to
the end of the message. This was considered such a common modification
that we wanted to reduce the affects of it even in the simple
canonicalization mechanism.

Adding such an extra CRLF during transport could generate any of these
cases:

Last-header: foobarCRLF
CRLF

Last-header: foobarCRLF
CRLF
CRLF

Last-header: foobarCRLF
CRLF
CRLF
CRLF

and they MUST be treated identically because you can't tell which of
those CRLFs were original and which were added later.

So, I believe that the intent was: 1) to strip out all trailing CRLFs.
2) If there *is* body content other than those CRLFs, the last
non-CRLF-only line would be guaranteed to have a CRLF at its end.

And 3.4.3 needs to be adjusted to guarantee that intent.

	Tony Hansen
	[email-masked]

---

From: Michael Thomas <[email-masked]>
Date: Tue, 19 Dec 2006 11:49:48 -0800

> there a missing "not" in your statement and I'm wrong?

My previous implementation did the same as Arvel's (given his recent
mail), which is the same thing that I think that Murray's  is doing. But
to be pedantic:

null body:

Last-header: foo<crlf>
<crlf>

l=2; canon-body: <crlf>

single crlf:

Last-header: foo<crlf>
<crlf>
<crlf>

l=2; canon-body: <crlf>

two trailing crlf's

Last-header: foo<crlf>
<crlf>
<crlf>
<crlf>

l=2; canon-body: <crlf>
> are NOT currently getting compatible implementations with the current

Right. That was my reason to bring this up.
> canonicalization mechanism.

Note that as I read the current draft, it also allows for the _removal_ of
trailing CRLF's too. Which is what I've seen in the field too. Note that
this isn't even a controversial point except for the question of null 
bodies,
but the draft seemingly covers that too with its null rule, though it's not
very clear.

       Mike

---

From: Stephen Farrell <[email-masked]>
Date: Tue, 19 Dec 2006 20:23:52 +0000

Ok, so we've this and we've that. Who's volunteering to
craft new text for the document?

S.

---

From: Michael Thomas <[email-masked]>
Date: Tue, 19 Dec 2006 12:42:20 -0800

Stephen Farrell wrote:

How about this:

Informative Note:

The canonical text can be thought of as a virtual representation of the
actual input. In particular, a body without any lines (ie, a null body) is
canonicalized as a single CRLF, which its canonical length set to 2 (l=2). 
Note that the canonical length still applies to the canonical text so an 
input
of:

Last-Header:<CRLF>
<CRLF>

Would result in a length of 2 and a canonical body of <CRLF>.

If the l= parameter is specified and is set to 0,  the canonical text is
null. That is, the canonical text is virtually created, where the first
length octets (if specified, otherwise all octets) are presented to
the hashing algorithm.

       Mike

---

From: Tony Hansen <[email-masked]>
Date: Tue, 19 Dec 2006 23:14:50 -0500

> craft new text for the document?

I did so in my original note in this thread.

	Tony Hansen
	[email-masked]

---

From: Hector Santos <[email-masked]>
Date: Wed, 20 Dec 2006 00:39:41 -0500

> I did so in my original note in this thread.

I guess what I am not sure about is whether a final <CRLF> is always 
required in the SIMPLE c14n method.

If so, then the minimum is always l=2.

But consider when l > 2, does this still mean the ending 2 bytes for the 
L amount is <CRLF>?

What if the actual text is larger than L?

Example:

12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
<CRLF>

here you have 52 bytes.  I believe canonicalized text size is l=50.

But what if the actual L= size defined in the DKIM-Signature: header is 
l=45? Is the canonicalized text then?

12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
123<CRLF>

??

---
HLS

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 20 Dec 2006 07:34:18 +0000

> I did so in my original note in this thread.

Sorry - I'd forgotten. Though you did have two options there
and we need to pick one, as Mike did in his mail.

So - do we have consensus for Mike's text (below) or not?
(Which cannot be an informative note IMO - it has to be
normative, even if its a corner-case.)

Ta,
S.

Mike's text (slightly edited):

"The canonical text can be thought of as a virtual representation of the
actual input. In particular, a body without any lines (ie, a null body) 
is canonicalized as a single CRLF, which its canonical length set to 2 
(l=2). Note that the canonical length still applies to the canonical 
text so an input of:

Last-Header:<CRLF>
<CRLF>

would result in a length of 2 and a canonical body of <CRLF> and,
with sha-256, a body hash of...

If the l= parameter is specified and is set to 0, the canonical text
is null, and, with sha-256, has a body hash of..."

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 20 Dec 2006 09:37:00 +0000

Mark Delany wrote:
>> actual input. In particular, a body without any lines (ie, a null 
>> body) is canonicalized as a single CRLF, which its canonical length 
>> set to 2 (l=2). Note that the canonical length still applies to the 
>> canonical text so an input of:
>>
>> Last-Header:<CRLF>
>> <CRLF>
> 
> No. a) It's creating a "surprise" CRLF when none previously existed. b) 
> It differs from DK for no good reason.

Fair enough. So, given that there is confusion, what's your counter
proposal for clarifying text?

(BTW - I assume there are only two sensible things to do here, so
we can just get two text proposals and see which achieves rough
consensus.)

S.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 20 Dec 2006 10:52:26 -0000

> mail), which is the same thing that I think that Murray's  is doing. But

No, I don't think that is what Tony was claiming the majority of  
implementations did (I think it is what the current wording says to do,  
but I think Tony was saying all those should result in an empty body to be  
hashed).

Anyway, here is some wording:

    The "simple" body canonicalization removes empty lines from the end of  
the
    body until either the last line is non-empty, or no lines remain. An  
empty
    line is a line of zero length after removal of any terminating CRLF. If
    the body is not now empty and the last line is not already terminated by
    CRLF, a CRLF is added to it.

       INFORMATIVE NOTE: Following [RFC 2822}, the CRLF which separates the
       header fields from the body is NOT part of the body, and therefore is
       never presented to the signing or verification algorithm. In the case
       of a pure binary message (such as one with a  
Content-Transfer-Encoding
       of 'binary') the concept of "lines" may not be meaningful.  
Nevertheless,
       wherever the pair of octets that represent CRLF happens to occur,  
that
       is to be considered as the end of a "line" for the purposes of this
       canonicalization algorithm.

Now, you are all invited to find some way of misinterpreting that :-).

Next, for body length counts which, as I now see from 3.4.5, are to be  
applied _after_ canonicalization. (BTW, I misinterpreted those counts as  
line counts rather than byte counts in an earlier message).

Here is another example to amuse you:

Last-header: foobarCRLF
CRLF
----------------
12345678CRLF
12345678CRLF
12345678
----------------

Now sign that with l=29 :-)
(don't forget to add the CRLF to the last line first)

---

From: Tony Hansen <[email-masked]>
Date: Wed, 20 Dec 2006 07:28:06 -0500

After creating the canonicalized form, no matter how that form winds up,
the l= value is applied directly AFTER the canonicalization.

That is,

12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
<CRLF>

is canonicalized into:

12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>

and l=45 turns this into:

12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345

	Tony Hansen
	[email-masked]

---

From: Tony Hansen <[email-masked]>
Date: Wed, 20 Dec 2006 08:06:06 -0500

Charles Lindsey wrote:
> but I think Tony was saying all those should result in an empty body to
> be hashed).
> 
>    The "simple" body canonicalization removes empty lines from the end
>    of the body until either the last line is non-empty, or no lines 
>    remain. An empty line is a line of zero length after removal of any
>    terminating CRLF. If the body is not now empty and the last line is
>    not already terminated by CRLF, a CRLF is added to it.
> 
>       Content-Transfer-Encoding of 'binary') the concept of "lines" may not
>       be meaningful. Nevertheless,
>       that is to be considered as the end of a "line" for the purposes 
>       of this canonicalization algorithm.
> 

I can live with the above.

I was going to suggest ABNF, but that's where we got into trouble last
time. :-) If we *were* to add ABNF to the above, it would have to have
two cases.

> Here is another example to amuse you:

The text on when l= counts should be applied hasn't changed meaning,
thankfully. It's very specific: it's an octet count applied after
canonicalization.

Canonicalization produces:

----------------
12345678CRLF
12345678CRLF
12345678CRLF
----------------

and l=29 produces:

----------------
12345678CRLF
12345678CRLF
12345678CR
----------------

	Tony Hansen
	[email-masked]

---

From: <[email-masked]>
Date: Wed, 20 Dec 2006 09:18:42 -0500

This sounds fine

Bill Oxley
Messaging Engineer
Cox Communications
404-847-6397
-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Tony Hansen
Sent: Wednesday, December 20, 2006 8:06 AM
To: Charles Lindsey
Cc: DKIM
Subject: Re: [Fwd: Re: [ietf-dkim] canonicalized null body and dkim]

----------------
12345678CRLF
12345678CRLF
12345678CRLF
----------------

and l=29 produces:

----------------
12345678CRLF
12345678CRLF
12345678CR
----------------

	Tony Hansen
	[email-masked]

---

From: Michael Thomas <[email-masked]>
Date: Wed, 20 Dec 2006 06:29:24 -0800

Stephen Farrell wrote:
>> No. a) It's creating a "surprise" CRLF when none previously existed. 
>> b) It differs from DK for no good reason.

There is a perfectly good reason: some things in the mail path strip 
trailing CRLF from the
body. We've experienced it firsthand and it's not uncommon, and would 
reduce the survival
rate if changed.
> proposal for clarifying text?

BTW: we're *waaaaaaay* past last call here. This text has been here for 
time immemorial.
All I propose we do is clarify the meaning of what is actually in the 
draft now surrounding a
corner case. I think the bar should be a lot higher for actually 
changing the text to mean
something that it doesn't currently imply.

       Mike

---

From: [email-masked]
Date: Wed, 20 Dec 2006 14:35:47 -0000 (GMT)

> changing the text to mean

Well, it appears that we've found an interop. glitch, right?
So the spec has to be changed to clarify/fix that (since we can).

And since different coders have done different things, someone
needs to change their code as well, once we decide how to fix
the spec.

Am I missing something?

S.

---

From: Michael Thomas <[email-masked]>
Date: Wed, 20 Dec 2006 07:06:16 -0800

> needs to change their code as well, once we decide how to fix
Yes. In my mind there's a substantial difference in clarifying an edge 
condition
where the current text strongly implies one thing with new normative text
to the contrary. The latter requires a lot more justification, IMO -- 
we're way
past last call. Thus far all I've seen by way of justification is 
aesthetics which
is not very good justification, in my mind. The case for keeping it is 
that it
-- intentionally or not -- does the right thing through some mail 
infrastructure
giving a more robust canonicalization. Why would we want to reduce our
robustness, especially at such a late date?

       Mike

---

From: Mark Delany <[email-masked]>
Date: Wed, 20 Dec 2006 07:23:32 -0800

Charles Lindsey wrote:
> of the
>       never presented to the signing or verification algorithm.

I think I agree with the effect, but I wish I could offer something 
terser, but that seems hard since this is dealing with the interaction 
of potentially different header canon and body canon.

Does the INFORMATIVE NOTE imply that the following two emails 
canonicalize to the same thing?

---------------
Last-Header: blahCRLF
CRLF
lineOne: blah1CRLF
lineTwo: blah2CRLF
---------------

---------------
Last-Header: blahCRLF
lineOne: blah1CRLF
lineTwo: blah2CRLF
---------------

And yes, I purposely have a body that matches the header syntax.

Mark.

---

From: Michael Thomas <[email-masked]>
Date: Wed, 20 Dec 2006 07:23:47 -0800

Mark Delany wrote:
> Michael Thomas wrote:
>> Yes. In my mind there's a substantial difference in clarifying an 
>> edge condition
>
> Last call is intended to find bugs. We found a bug, let's fix it.

Last call ended 4 months ago.

> I don't think the "lateness" has anything to do with it since there 
> are bugs in either direction and no one even noticed until recently. 
> It can hardly be the case that it is breaking much to apply the fix.

Well, we have a deployment and changing this from the current sense would be
breaking messages that would have otherwise survived. That is what I 
recently
discovered. Do you have evidence to the contrary with your DKIM deployment?

       Mike

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 20 Dec 2006 15:27:27 +0000

> Last call ended 4 months ago.

Doesn't matter. He's right that we found a bug and we should fix
it.

> discovered. Do you have evidence to the contrary with your DKIM deployment?

If the two fixes differ in how they affect signature robustness then
that is a significant difference. Can you elaborate? (Sorry, I'm not
a mail person so I don't get all the nuances of what can happen to
the last few bytes of a message.)

S.

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 20 Dec 2006 15:19:11 +0000

> robustness, especially at such a late date?

Those are reasonable points and you may well convince folks.

However, the rough consensus may also end up on the other side this
time, we'll see.

FWIW, I don't see this as fitting the usual "high bar" argument, since
there is a clear, even though very minor, interop issue in this case,
so while you may see one proposed solution as being done for aesthetic
reasons, and you may be right or not, there's a definite need to change
the text at least, and our way to do that is to establish the rough
consensus of the WG.

Meanwhile - are Mike's and Charles' texts the two options we want to
use to see who wants what? If so, I'll send a mail tomorrow and we can
have a poll over the holiday to see who prefers which. (Note: there's no
additional delay to RFC-dom involved since the IESG won't be deciding
base's fate until Jan 11.)

Stephen.

---

From: Michael Thomas <[email-masked]>
Date: Wed, 20 Dec 2006 07:37:44 -0800

> the last few bytes of a message.)
Simply stated, as the draft is currently worded, the simple body 
canonicalization
will be immune to additions *and* deletions of of CRLF's at the end of 
the body
in all cases. The proposed change to the normative behavior, on the 
other hand,
will not be immune to deletions. Deletions are something that happen in 
real life,
and we have experienced then. Changing the normative sense of the draft 
at this
point will reduce the number of verified messages.

I will also add that finding this has been an extremely maddening 
adventure as
a developer. If we change it to what Mark and others are advocating, we are
condemning all future DKIM developers to find this problem themselves just
as I have and be faced with unpleasant alternatives of how to work 
around it,
if they even find it in the first place. I don't wish my pain for this 
one on anybody.

       Mike

---

From: Dave Crocker <[email-masked]>
Date: Wed, 20 Dec 2006 07:45:55 -0800

Michael Thomas wrote:
> Mark Delany wrote:
>> I don't think the "lateness" has anything to do with it since there are 
>> bugs in either direction and no one even noticed until recently. It can 
>> hardly be the case that it is breaking much to apply the fix.
> 
> recently discovered. Do you have evidence to the contrary with your DKIM
> deployment?

Sorry, Mike, but that particular line of argument isn't applicable here.> Hence,
it was pure academic exercise.

Working group specs are subject to semantic change up to the point of IESG
approval. Anyone deploying code based on a spec prior to that moment is taking a
well-advertised risk.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 20 Dec 2006 15:52:54 +0000

> point will reduce the number of verified messages.

Thanks for the explanation.

Sounds like a fairly telling argument to me, if people are generally
seeing such deletions.

> if they even find it in the first place. I don't wish my pain for this 

There you confused me - aren't we going to make it easy for every
subsequent coder by telling them about this in advance? But its ok
that I'm confused, I often am:-)

S.

---

From: Michael Thomas <[email-masked]>
Date: Wed, 20 Dec 2006 07:52:34 -0800

> approval. Anyone deploying code based on a spec prior to that moment 
Huh? I'm saying that changing this is *NOT* academic: there are things 
in the
real world which will cause more message signature to fail if we make this
change. You're not in favor of that are you?

I thought this was the entire point of running code: to find out how the 
spec
works in real life. If that's just an academic exercise, there's 
something seriously
wrong.

       Mike

---

From: Michael Thomas <[email-masked]>
Date: Wed, 20 Dec 2006 07:59:52 -0800

Stephen Farrell wrote:
>> will be immune to additions *and* deletions of of CRLF's at the end 
>> of the body
>> will not be immune to deletions. Deletions are something that happen 
>> in real life,
>> and we have experienced then. Changing the normative sense of the 
>> draft at this
>> if they even find it in the first place. I don't wish my pain for 
>> this one on anybody.
>
Not if we change the current sense of the spec, no. If we change rather
than clarify the current draft, every developer is going to have to go 
through
the same detective work on why signatures with lone CRLF's in the body are
being stripped causing signatures to fail. This is *not* intuitive and 
not easy
work. The best outcome is to have a *robust* and *clear* canonicalization.

       Mike

---

From: Dave Crocker <[email-masked]>
Date: Wed, 20 Dec 2006 08:01:27 -0800

>> Sorry, Mike, but that particular line of argument isn't applicable here.>
>> Hence, it was pure academic exercise.
>> 
>> approval. Anyone deploying code based on a spec prior to that moment is
>> taking a well-advertised risk.
> Huh? I'm saying that changing this is *NOT* academic

Wow. Sorry. Thunderbird got very creative.

That text after the angle bracket was from an entirely unrelated message and I 
know I didn't put it there by an accidental cut and paste.  In other words, my 
note was only the first sentence of the first paragraph and all of the second 
paragraph.

: there are things in the
> change. You're not in favor of that are you?

Two lines of argument.  You were invoking the 'installed base' argument and I 
was noting that it is not valid to use that, at this stage, for this type of issue.

As for the rest of the thread and the effort to resolve this ambiguity, I was 
offering no comment.  (I'm tracking it, but have nothing constructive to add, 
especially since it seems to have developed as a thorough consideration of a 
real problem and real solutions to it.)

> I thought this was the entire point of running code: to find out how the spec
>  works in real life. If that's just an academic exercise, there's something
> seriously wrong.

Yup.  But casting it as "people are already using it" is different than saying 
"it is the best solution to the problem".

d/
-- 

   Dave Crocker

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 20 Dec 2006 16:05:33 +0000

Michael Thomas wrote:
> work. 

Ok, I see what you meant now. The robustness/deletion argument is
still much better IMO.

 > The best outcome is to have a *robust* and *clear* canonicalization.

Well, both seem equally clear to me. I see you robustness argument
too, but I guess we'll see what others have to say in a bit.
(I'm offline from now 'till tomorrow:-)

S.

---

From: Michael Thomas <[email-masked]>
Date: Wed, 20 Dec 2006 08:09:19 -0800

Dave Crocker wrote:
>>> Working group specs are subject to semantic change up to the point 
>>> of IESG approval. Anyone deploying code based on a spec prior to 
>>> that moment is
> That text after the angle bracket was from an entirely unrelated 
> message and I know I didn't put it there by an accidental cut and 
> paste.  In other words, my note was only the first sentence of the 
> first paragraph and all of the second paragraph.
>
>
>>  real world which will cause more message signature to fail if we 
>> make this change. You're not in favor of that are you?
>
> Two lines of argument.  You were invoking the 'installed base' 
> argument and I was noting that it is not valid to use that, at this 
> stage, for this type of issue.

No I was not. My code up until very recently was making the same mistake.
What is strongly implied in the current draft offers *superior* 
robustness in
the real world. That is, it is immune to additions *and* deletions of 
trailing
CRLF's. That is not an appeal to installed base, it's an appeal to a more
robust spec. As it happens, the spec merely needs to make more obvious
what the normative text already says and we will have an improved spec.

       Mike

---

From: Dave Crocker <[email-masked]>
Date: Wed, 20 Dec 2006 08:44:03 -0800

Michael Thomas wrote:
> No I was not. 

ok.  sorry I misread it to mean that.

d/

-- 

   Dave Crocker

---

From: Hector Santos <[email-masked]>
Date: Wed, 20 Dec 2006 11:52:45 -0500

> what the normative text already says and we will have an improved spec.

What you experienced with the "Alpha Code" out there is normal. Whatever 
the final outcome, at the end of the day, the official RFC is whats 
important here and those experiences, insights, etc, needs to be 
transcribed in the technical specs for the reasons you stated - the NEXT 
GUY should not have to be dealing with these protocol design questions.

---
HLS

---

From: Tony Hansen <[email-masked]>
Date: Wed, 20 Dec 2006 23:50:07 -0500

Michael Thomas wrote:
> No I was not. My code up until very recently was making the same
> mistake. What is strongly implied in the current draft offers
> *superior* robustness in the real world. That is, it is immune to
> additions *and* deletions of trailing CRLF's. That is not an appeal
> to installed base, it's an appeal to a more robust spec. As it
> happens, the spec merely needs to make more obvious what the
> normative text already says and we will have an improved spec.

Michael, I don't see where you see the "new and improved algorithm" is
more robust in the face of deleted CRLFs than the proposed fix to the
dkim-base text?

Point 1:
	The original dkim-allman-base and dkim-ietf-base text called for
	trailing CRLFs to be deleted.

Point 2:
	The change to the text introduced in base-04 called for a
	CRLF to be added to a message that doesn't have a CRLF on its
	last line of text. In the process, it changed what happens when
	there is a message that only has trailing CRLFs. That base-04
	included a change to the existing behavior for handling such
	messages was missed by many people, myself included, until now.

Point 3:
	The discussions that led to the change in base-04 did not
	include any intent to change the default behavior when there
	are only CRLFs in the message body. Those discussions were only
	about what should happen with a message whose last line didn't
	end in CRLF. (Such messages are only possible when you are using
	a binary MIME content-transfer-encoding and a binary transport
	protocol.)

Point 4:
	The proposed text changes restore the behavior for when you have
	blank messages, as well as maintaining the change in semantics
	necessary to handle messages whose last line doesn't end in
	CRLF. (This includes the messages that have lost a trailing CRLF
	in transport.)

Point 5:
	No matter which option we choose, someone's going to have to
	change their implementation.

Point 6:
	My preference is to use the algorithm that's compatible with
	what was in dkim-allman-base-* and dkim-ietf-base-00 thru 03,
	while maintaining the change to handle messages with no final
	CRLF.

Point 7:
	Another way of expressing this algorithm that people may find
	easier to understand is:

	"If the last line of the message does not end with CRLF, CRLF is
	added. Then, CRLF 0*CRLF is reduced to a single CRLF."

Tony Hansen
[email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 21 Dec 2006 11:04:33 -0000

> canonicalize to the same thing?

Certainly not! Why should they?

The first is a message with one header and a body of two lines. The header  
canonicalizes into itself, and the body canonicalizes into

---------------
lineOne: blah1CRLF
lineTwo: blah2CRLF
---------------

The second is a message with three headers and no body at all. The headers  
canonicalize as themselves, and for the body you treat it as empty:

---------------
---------------

One could add text to say that an absent body is equivalent to an empty  
body, if you think that is necessary.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 21 Dec 2006 11:20:58 -0000

> body. We've experienced it firsthand and it's not uncommon, and would  

But nobody is proposing any such thing. What stripping are you talking  
about? If it is stripping a CRLF from an otherwise non-empty last line,  
then ALL the rules under discussion require you to put it back, so that  
what you are being asked to hash always has a CRLF at the end of any last  
line that actually has real text in it. But if there is no real text  
anywhere, then you canonicalize into an empty file.

So you start with

------------------
Last-Header: foobarCRLF
CRLF
barbazCRLF
CRLF
------------------

and some stupid system en route strips it down to

Last-Header: foobarCRLF
CRLF
barbaz

so in that case you put back the CRLF, and what you hash is:

------------------
barbazCRLF
------------------

And that is also what you would have hashed if it arrived exactly as  
originally sent, or even if a few more CRLF-only lines had been picked up  
somewhere.

BUT, if you started with

------------------
Last-Header: foobarCRLF
CRLF
------------------

or with

------------------
Last-Header: foobarCRLF
------------------

Then certinaly nothing will be stripped en route (there is nothing to  
strip), but some CRLF-only lines might get added, but nevertheless what  
you hash is still the empty file

------------------
------------------

And, in all those cases, the wording I suggested gives exactly that.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 21 Dec 2006 11:32:53 -0000

> 	added. Then, CRLF 0*CRLF is reduced to a single CRLF."

Well that would be better expressed as "1*CRLF is reduced to a single  
CRLF",
but you should be saying "If the last line of the body" rather than "If  
the last line of the message", because

------------------
Last-Header: foobarCRLF
CRLF
------------------

is a message with an empty body (read RFC 2822).

Or put it another way; the last line of what you hash MUST NEVER be empty  
(i.e. must never consist of CRLF with no preceding real text).

That was why I included my INFORMATIVE NOTE, just to remove any possible  
misinteroretations (and we have just seen how easily people misinterpret  
things :-( ).

---

From: Tony Hansen <[email-masked]>
Date: Thu, 21 Dec 2006 07:26:47 -0500

Yes, see my other note that crossed in the mail. I left off a sentence.
And "last line of the body" is definitely more correct.

	Tony

Charles Lindsey wrote:
> Or put it another way; the last line of what you hash MUST NEVER be
> empty (i.e. must never consist of CRLF with no preceding real text).
> 
> --Charles H. Lindsey ---------At Home, doing my own thing------------------------
> 
> Tel: +44 161 436 6131                     
> NOTE WELL: This list operates according
> tohttp://mipassoc.org/dkim/ietf-list-rules.html

---

From: Tony Hansen <[email-masked]>
Date: Thu, 21 Dec 2006 07:25:44 -0500

I left off a sentence in Point 7.

> 	added. Then, CRLF 0*CRLF is reduced to a single CRLF."

	"If the body only consists of a CRLF after this reduction, that
	too is removed."

Tony Hansen
[email-masked]

---

From: Hector Santos <[email-masked]>
Date: Thu, 21 Dec 2006 12:55:41 -0500

> 	"If the body only consists of a CRLF after this reduction, that

Tony,

For SIGNING......

Does this imply that SIMPLE c14n SHOULD NOT be done when there exist 
only 2 bytes <CRLF> in the body?

For VERIFYING......

When it comes to verification, the l= tag will determine what text was 
canonilized.

if l=2,  that means two <CRLF> were hashed.

if l= missing, that means at minimum two <CRLF> were hashed.

If l=0, no hashing was done.

It sounds to me, that technically, the bottom line the SIMPLE c14n feed
must end with <CRLF>, period.  If missing, it is added to the feed.

--
HLS

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 22 Dec 2006 14:34:25 -0000

> only 2 bytes <CRLF> in the body?

Why ever not? Simple says, roughly, "remove empty trailing lines from the  
body", so you remove them. Ah! you were confused by the "last line of the  
message does not end with CRLF" bit? But if the message body is empty,  
then there IS no last line, so the question does not arise. Probably  
better, then, to use my wording which clearly covers all these edge cases.
> When it comes to verification, the l= tag will determine what text was  

If what was hashed was <empty>, then the signature should have claimed  
l=0. It says in 3.5 that the l= value MUST NOT be larger than the actual  
number of octets in the canonicalized message body (but a liberal verifier  
might, and maybe SHOULD, interpret that as "use the actual length, or the  
l= value, whichever is the smaller).
> if l=2,  that means two <CRLF> were hashed.

But that case cannot arise with the text proposed.
> if l= missing, that means at minimum two <CRLF> were hashed.

No, it means whatever the canonicalization produced would be hashed, which  
would be <empty> in this case.
> must end with <CRLF>, period.  If missing, it is added to the feed.

No! Magically appearing emptry line with CRLFs are _precisely_ what we are  
trying to avoid.

---

From: Michael Thomas <[email-masked]>
Date: Fri, 22 Dec 2006 15:48:23 -0800

> 	"If the body only consists of a CRLF after this reduction, that
I have to say that this is even more opaque to me then what is in the
current draft.

I went through Charles' cases and I specifically didn't see the one that
I'm actually interested in:

before:

Last-Header: foo
<CRLF>
<CRLF>

becomes:

Last-Header: foo
<CRLF>

in transit. What I do know is that the current draft handles this case,
even though it's not entirely clear and could use an informative note
to make it more obvious. What's not clear to me is whether anything
new 1) has that property and 2) is any clearer than just adding informative
text to the current draft. Redoing normative text at this point shouldn't
be done lightly, IMO, and I really don't see what the gain is even if
your text works for this case.

       Mike

---

From: Hector Santos <[email-masked]>
Date: Fri, 22 Dec 2006 19:39:07 -0500

Charles Lindsey wrote:
> On Thu, 21 Dec 2006 17:55:41 -0000, Hector Santos 

> But that case cannot arise with the text proposed.

But what if it is l=2, what if that is what the VERIFY sees?  It means 
whether truly BYTES exist or not, these two hashable bytes must be 
<CR><LF>.

> No, it means whatever the canonicalization produced would be hashed, 
> which would be <empty> in this case.

Ok, again, we are talking VERIFICATION here. If no l= tag is specified 
as part of the DKIM-Signature: header, the specs says the "entire" body 
is hashed. Therefore, according to what I am reading, if the body is 
indeed DKIM-NULL (empty after any <CRLF> trimming) then 2 bytes <CRLF> 
will be hashed.

Are you saying there SHOULD be no hashing and therefore no b= tag?

> No!

No?

 > Magically appearing empty line with CRLFs are _precisely_ what we
 > are trying to avoid.

No line is magically appearing here.

Based on the current SIMPLE c14n specs, it would be FEED into the 
HASHING ENGINE if it didn not exist as part of the original feed.  It is 
not added to the original source.

What I am now hearing is this, given 50 REAL bytes

12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>

if l=25, then the hashing feed is:

12345678<CRLF>
12345678<CRLF>
12345

and it does not include the expected final <crlf> which would currently 
required during a SIMPLE c14n signing process.

If the CRLF is part of the final feed, then the l=25 text would be:

12345678<CRLF>
12345678<CRLF>
123<CRLF>

No?

---
HLS

---

From: <[email-masked]>
Date: Fri, 22 Dec 2006 21:24:06 -0500

You are correct l=N should be avoided at all costs :-)

-----Original Message-----
From: [email-masked] on behalf of Hector Santos
Sent: Fri 12/22/2006 7:39 PM
To: Charles Lindsey
Cc: DKIM
Subject: Re: [Fwd: Re: [ietf-dkim] canonicalized null body and dkim]

No?

12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>
12345678<CRLF>

12345678<CRLF>
12345678<CRLF>
12345

12345678<CRLF>
12345678<CRLF>
123<CRLF>

No?

---
HLS

---

From: Hector Santos <[email-masked]>
Date: Sat, 23 Dec 2006 09:45:32 -0500

Except when its from a trusted source? <g>

** Happy Holidays **

---
HLS

---

From: Tony Hansen <[email-masked]>
Date: Sat, 23 Dec 2006 17:21:44 -0500

> If the CRLF is part of the final feed, then the l=25 text would be:

No.

The canonicalization is done *FIRST*. Then exactly l= octets of that are
presented to the hash algorithm.

> What I am now hearing is this, given 50 REAL bytes

ok

> 12345678<CRLF>

which is the exact same thing that went in. The last line already has a
CRLF at its end, so no additional CRLF needs to be added.

> and it does not include the expected final <crlf> which would
> currently required during a SIMPLE c14n signing process.

The octets shown here are exactly what would be fed to the hash.

> If the CRLF is part of the final feed, then the l=25 text would be:

No, the CRLF being discussed is the one that's added as part of the
canonicalization process if the last pre-canonicalized line did not have
a CRLF present. It's *NOT* one added after the canonicalization is
completed. So what you're showing here would NOT be what is fed to the
hash algorithm.

	Tony Hansen
	[email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Sat, 30 Dec 2006 12:54:39 -0000

> your text works for this case.

OK, let us look at that case:

Last-Header: foo     <- last header
<CRLF>               <- separator between headers & body
<CRLF>               <- a body consisting of one empty line

Here is my proposed text:

So that emtpy line is removed for canonicalization purposes, and no lines  
remain, so that is it.

Your second example already has an empty <body>, so there is nothng to be  
done. So both examples canonicalize into <empty>, as expected.

---

From: "Charles Lindsey" <[email-masked]>
Date: Sat, 30 Dec 2006 13:05:06 -0000

> whether truly BYTES exist or not, these two hashable bytes must be  

No, if what was hashed has zero bytes in it, then you MUST NOT say 'l-2'.
> indeed DKIM-NULL (empty after any <CRLF> trimming) then 2 bytes <CRLF>  

If the body is indeed NULL/empty/whatever, then WHY do people want to do  
anything to it, like adding a <CRLF> out of thin air? The text I have  
proposed says to leave it empty in that case.
> Are you saying there SHOULD be no hashing and therefore no b= tag?

No, you hash <empty> and the result of that goes into the b= tag.

Apparently, the result of hashing an empty file with sha-256 is
    47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU
so that would go in the b= tag.