Subject: Re: GnuPG and trailing whitespace handling

From: [email-masked] (Ulf Möller)
Date: Wed, 10 Feb 99 21:58 +0100

>>    0x01: Signature of a canonical text document.
>>          Typically, this means the signer owns it, created it, or
>>          certifies that it has not been modified.  The signature is
>>          calculated over the text data with its line endings converted
>>          to <CR><LF> and trailing blanks removed.
>
>I can verify that PGP 2.6.2 and PGP 5.0 do *not* do this for detached
>signatures.

I reported that on the OpenPGP list before RFC 2440 was issued, but
for some reason the RFC authors ignored my post.

In PGP, trailing whitespace is removed as part of the clearsig
encoding (which is the only situation where trailing whitespace can
get lost in the transport). It is not part of the signing operation
and should not have been mentioned in the section quoted above.

---

From: Jon Callas <[email-masked]>
Date: Wed, 10 Feb 1999 14:28:58 -0800

At 09:58 PM 2/10/99 +0100, Ulf Möller wrote:

|I reported that on the OpenPGP list before RFC 2440 was issued, but
|for some reason the RFC authors ignored my post.
|
|In PGP, trailing whitespace is removed as part of the clearsig
|encoding (which is the only situation where trailing whitespace can
|get lost in the transport). It is not part of the signing operation
|and should not have been mentioned in the section quoted above.

I didn't ignore it, I'm pretty sure we discussed it.

I believe that for OpenPGP, this is the right thing to do, and that for
*any* object in "canonical text" form, trimming trailing white space is a
Smart Thing To Do.

Now, an *implementation* of OpenPGP has to be always generating thingies
trimmed. When it does, they will interoperate with the old versions just
fine. The only subtle gotcha here is that when *receiving* a signed text
object, a smart implementation won't throw a hissy fit if the text object
hasn't been trimmed; it will just roll its eyes and mutter "get with the
program, geezer" under its breath. This is the old principle of "be liberal
in what you accept and conservative in what you generate."

	Jon

-----
Jon Callas                                  [email-masked]
CTO, Total Network Security                 3965 Freedom Circle
Network Associates, Inc.                    Santa Clara, CA 95054
(408) 346-5860                              
Fingerprints: D1EC 3C51 FCB1 67F8 4345 4A04 7DF9 C2E6 F129 27A9 (DSS)
              665B 797F 37D1 C240 53AC 6D87 3A60 4628           (RSA)

---

From: Werner Koch <[email-masked]>
Date: Thu, 11 Feb 1999 11:05:23 +0100

> for some reason the RFC authors ignored my post.

Yep, I remember that post but it's long ago.

> and should not have been mentioned in the section quoted above.

Okay, so I implement it like PGP and make a remark that GnuPG does not
follow the RFC :-)

  Werner

-- 
ceterum censeo RSA esse delendam

---

From: Werner Koch <[email-masked]>
Date: Thu, 11 Feb 1999 11:22:03 +0100

> in what you accept and conservative in what you generate."

That's right, but it is quite complicated to do in an implementaion which
gets it input from stdin.  One would have to calculate hashs for boths
"versions".   

   Werner

---

From: [email-masked] (Ulf Möller)
Date: Thu, 11 Feb 99 18:58 +0100

>I didn't ignore it, I'm pretty sure we discussed it.

Your reply didn't show up in my mailbox nor in the archive at
www.imc.org, then.

>in what you accept and conservative in what you generate."

That sort of makes sense, but the RFC should have contained a notice
about the incompatibility. When you follow the letter of the RFC, the
obvious thing to do when you verify signatures is to use the same
function for tweaking the line ends for both signature types.
Following the principle of "being liberal in what you accept" without
knowing about this specific incompatibility (there are dozens of ways
a signing implementation could fail) is impossible because there is no
such thing as a "near miss" with a message digest.

---

From: Jon Callas <[email-masked]>
Date: Thu, 11 Feb 1999 12:30:55 -0800

At 11:05 AM 2/11/99 +0100, Werner Koch wrote:

|Okay, so I implement it like PGP and make a remark that GnuPG does not
|follow the RFC :-)

I don't know how to interpret the smiley, so I apologize if this is a
humor-impaired comment.

If you follow the RFC, you will be backwards compatible.

If you want to be forwards compatible (meaning that you accept things old
versions generate), all you have to do is not generate an error when an old
version fails to blank-trim.

IETF standards are not handcuffs. You are always permitted to do more than
them. This is especially true in cases where you accept as valid something
that fails slightly.

	Jon

-----
Jon Callas                                  [email-masked]

---

From: Russ Allbery <[email-masked]>
Date: 11 Feb 1999 16:08:44 -0800

> If you want to be forwards compatible (meaning that you accept things
> old versions generate), all you have to do is not generate an error when
> an old version fails to blank-trim.

Pardon me if I'm confused here, but wouldn't that involve checking any
failing detached text-mode signature a second time with the original data
without trailing whitespace stripped to see if it then passes?

And isn't it impossible to both follow the RFC and generate detached
text-mode signatures that will verify correctly with PGP 5.0 if the data
has trailing whitespace?  In that either the signatures will verify with
PGP 5.0 or they will verify with RFC-compliant checkers, but never with
both?

-- 
Russ Allbery ([email-masked])         <URL:http://www.eyrie.org/~eagle/>

---

From: Werner Koch <[email-masked]>
Date: Fri, 12 Feb 1999 12:21:20 +0100

> version fails to blank-trim.

Ah right that's an idea how to handle it without calculating hashs of both
versions - I have to see how I can do this with my streams like framework.

  Werner

---

From: Jon Callas <[email-masked]>
Date: Fri, 12 Feb 1999 16:27:25 -0800

At 04:08 PM 2/11/99 -0800, Russ Allbery wrote:

|Pardon me if I'm confused here, but wouldn't that involve checking any
|failing detached text-mode signature a second time with the original data
|without trailing whitespace stripped to see if it then passes?
|
|And isn't it impossible to both follow the RFC and generate detached
|text-mode signatures that will verify correctly with PGP 5.0 if the data
|has trailing whitespace?  In that either the signatures will verify with
|PGP 5.0 or they will verify with RFC-compliant checkers, but never with
|both?

Here's my interpretation of what you should do:

Remember that we're not talking about clear-signed messages, we are talking
about a message that is a literal packet and a signature packet (and
possibly this pair has been encrypted).

If you are constructing the message, construct it in canonicalized text.
Line ends are CRLFs, and lines are blank-trimmed. Construct the signature,
send the message. All things being equal (that is, excepting the cases
where you have a DSS sig sent to 2.6 etc.), it will always work. The
literal packet should be a text-mode literal packet.

When you receive a message, don't go interpreting it. Do nothing. Compute
the signature. In the case where you have a non-blank-trimmed literal
packet, don't even look for the flaw here. The signature will work
correctly for anyone who constructed it correctly, and it will work for
anyone who was merely stupid. That's what I meant by rolling your eyes and
proceeding. My suggestion is that by doing zero work, you handle all
correctly contrusted messages, and many incorrectly constructed messages.

There are a number of flawed cases, here, too. I'm using the word "flaw"
only to avoid a more perjoritive term like "erroneous" because I don't want
to get into those arguments.

One flawed case is a text mode literal that has not been blank trimmed, but
is otherwise correctly textmode. It's followed by a textmode sig packet.
The "do no work" precept lets this one work.

Another is a textmode literal that contains carriage control -- for example
one that underscores by using a bare CR. The "do no work" precept lets this
one work, and a smart implementation is likely to damage this packet.

Another is a mal-formed textmode literal. For example, it just put unix
text (LF endings) into a textmode packet. The DNW precept makes for a
correctly formed signature, but there may be errors when the file is
written out (or maybe not, if the texmode literal processor has
error-adaptions in it). But that's a different problem.

More problematic is a textmode literal with trailing blanks, with a sig
calculated by trimming them. Here, DNW fails. This is also where the real
crux of the problem is. I believe that it's possible to have "minimal"
implementations and "friendly" implementations. A minimal implementation
will fail to verify the signature. This is in my opinion fine, because I
believe that a compliant 2440 application will be *generating*
blank-trimmed signed text. If the trailing blanks were that important to
you, there's always binary mode, y'know? A friendly implementation,
however, is free to see that the signature failed and try again using a
different strategy (such as blank trimming or blank trimming with line-end
conversion) and see if the signature works with this.

It's my opinion that if a 2440 application wants to send a textmode message
with trailing blanks then the obvious, right way to do it is to put a
binarymode signature after the textmode literal. That's what binarymode
means -- don't mess with the content, just hash it straight. To me,
textmode signatures are an *assurance* that the literal packet it signs met
those criteria when it was generated. If it doesn't, you could have
constructed a textmode message and signed it in binary mode. To tie in what
I said before, this means that a 2440 app conservatively generates a
message by trimming a textmode packet if it is going to textmode sign it,
so there will be *no* *question* how to interpret it.

Now then, having said that, reality is that we live in a world with a mass
of implementations of PGP, all of which are more-or-less 2440ish. A
friendly application can consider the textmode signature packet when it
*receives* it to be an alert that it may have to futz with the hashing to
be fully forwards compatible. Note that a 2440 application is not required
to do that. That's the difference between minimal and friendly. No one says
you have to be friendly.

(Here's another comment of mine that only relates to comments that I've
made here and before that you're permitted to go beyond the spec. 2440 says
that textmode literals have CRLF line ends. It's my considered opinion that
textmode should also trim blanks. You're free to disagree, but if I were
writing a 2440 app, I'd trim blanks. I probably would also allow textmode
conversions to do tab/blank conversions and possibly paragraph
reformatting. 2440 is silent on all of these. That does not mean that one
is forbidden from doing them. If you're the implementer, you get to make
these decisions yourself.)

	Jon

-----
Jon Callas                                  [email-masked]

---

From: [email-masked] (Ulf Möller)
Date: Sun, 14 Feb 1999 01:24:51 +0100 (MET)

> packet, don't even look for the flaw here.

It is not that simple for detached signatures (a detached signature is
one where the PGP message consists of the signature packet only and
the signed text is specified externally). When you verify a detached
text-mode signature, you *have to* convert the text to canonical
format first. And, conversely, when you generate a detached signature,
modifying the original file usually is not an option.

> binarymode signature after the textmode literal.

RFC 2440 doesn't expressly say whether or not you can do that, but as
far as I remember, PGP just throws funny error messages at you when
you try it.