Subject: [ietf-822] Base64 encoding details

From: Alessandro Vesely <[email-masked]>
Date: Fri, 18 Sep 2020 11:32:01 +0200

Hi all,

it has become customary to encode text messages in base64.  This is presumably 
an attempt to avoid breaking the body hashes of DKIM signatures.

However, notwithstanding RFC 2045 saying:

    Care must be taken to use the proper octets for line breaks if base64
    encoding is applied directly to text material that has not been
    converted to canonical form.  In particular, text line breaks must be
    converted into CRLF sequences prior to base64 encoding.

I note that, apparently based on the OS where the encoding is done, text is 
often encoded using LF line endings, instead of CRLF.  This list, for one, does so.

Would it make sense to note this feature explicitly in the header?  For 
example, one could write:

     Content-Transfer-Encoding: base64; nocr; column-width=76

That data would allow to reproduce the exact encoding that was signed, which 
was the reason to use base64 in the first place.

Thoughts?

Best
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Fri, 18 Sep 2020 06:35:44 -0700 (PDT)

> an attempt to avoid breaking the body hashes of DKIM signatures.

The usual reason it's done is so that you don't have to fix your HTML
generator to produce shorter lines, or lines at all.

I have not noted any increase in doing this, although YMMV.

> was the reason to use base64 in the first place.

Huh? Why are you decoding and then re-encoding the parts of your message, while
attempting to preserve a signature over the encoding? This entire process is
bound to fail, as there is no required canonical form for base64 output, let
alone quoted-printable.

And for that matter, why are you attempting to preserve DKIM signatures while
playing games with the message content? DKIM is intended to sign messages in
transit. It is not intended, or designed for, other signature applications.

All that said, if you really want to allow this sort of thing, the way to do it
isn't to try and figure out how to describe the myriad representations allowed
in our current set of encodings. That's never going to work, because people
will never stop coming up with variations you haven't thought of.

What you do instead is sign the material under the encoding, eliminating the
encoding variations from the signature. And while you're at it, you do it in
stages, so that when you're dealing with really large parts you can reuse
previosuly computed hashes.

More specifically, you sign a MIME message body by first hashing the unencoded
leaf body parts and the part headers minus the CTE fields and removing the
boundary= parameters from the CT headers. You then compute a Merkle tree of the
hashes corresponding to the MIME tree structure. (There are some details
missing here, and there are various security issues that have to be dealt with
some care, but this is the basic idea.)

You now have a hash that will survive the sort of manipulations you seem to
want to do.

Nick Shelness and I proposed this scheme more than 20 years ago, and I doubt we
were the first to come up with it - given the tree structure of MIME, a Merkle
tree is an obvious fit. The problem with it is the benefits have never been
seen to outweigh the costs. Sure, there are cases where not having to
compute the hash is a real win, or not having to store the original message is
a win, but given hardware advances it never seems to be enough?

> Thoughts?

First and foremont, changing the syntax of the CTE header in an incompatible
way, as this proposal does, would cause no end of trouble.

As far as line terminators go, the standard is clear. If you don't want to
follow it you're depending on other implementations' tolerance of your crap.

Tolerating this sort of crap is fairly easy, which of course is why folks can
get away with it enough of the time that they aren't forced to fix it. And
tolerating this sort of crap isn't made any easier by providing a means
of announcing it. So there's no benefit to your scheme there.

So even if this propsal was modified to, say:

   Content-transfer-encoding-notes: crap=nocr; column-width=76

all it really does is introduce possible silly states.

Of course nothing prevents you from doing all this locally. If you want to take
messages apart and store only the pieces, nothing prevents you from writing
down the format details of the encodings that were used so you can try and
reproduce them later. You aren't really helped at all by asking the people
generating the encodings to produce these sorts of labels.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Fri, 18 Sep 2020 07:41:03 -0700

> previosuly computed hashes. 

this.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Ned Freed <[email-masked]>
Date: Fri, 18 Sep 2020 09:54:30 -0700 (PDT)

> this.

I think I have a proto-draft writeup of the mechanism, FWIW. I can post it if
there's any real interest.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Fri, 18 Sep 2020 10:05:34 -0700

> there's any real interest.

+1

d/

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 18 Sep 2020 19:10:20 +0200

> bound to fail, as there is no required canonical form for base64 output, let

I agree qp is not quite reproducible.  However, base64 is most often encoded at 
a fixed column width, even if it's not required to be done that way.  In many 
cases the width is 76, but not always.

> transit. It is not intended, or designed for, other signature applications.

This list appends a footer, and then encodes the body in base64.  That way, 
even if I signed using l=, the signature is broken.  If I convert the body back 
to 8bit, my signature verifies.

However, occasional failures are more likely to occur with 8bit.  For example, 
lines starting with "From ".  So, what if I base64 encoded the before signing? 
I'd have to unencode ietfa.amsl's base64 encoding, strip the footer, and then 
re-encode.  If I use the same column-width and line terminators, the signature 
should verify.

> [...Merkle tree idea elided ...]

I appreciate the scheme you proposed.  I too saw proposals to have 
MIME-compatible DKIM signatures.  That ship has apparently sailed.

I'm looking at how to verify standard DKIM signatures after a "footer" 
transformation.  See:
https://tools.ietf.org/html/draft-kucherawy-dkim-transform-02#section-6

>> Thoughts?
> 
> First and foremost, changing the syntax of the CTE header in an incompatible

Yes.

> So even if this proposal was modified to, say:
> 

Those notes may emerge from a mailing list forwarding unscathed, thereby 
allowing to still verify the original signature.  Cool, no?

> generating the encodings to produce these sorts of labels.

I think I agree... To be clear, I agree that I can hardly ask to ietfa.amsl to 
detect and annotate my base64 encoding details.  Presumably, the mail filter 
that produced and signed the original message should also annotate encoding 
peculiarities.

Best
Ale
--

---

From: John Bucy <[email-masked]>
Date: Fri, 18 Sep 2020 11:31:29 -0700

> generator to produce shorter lines, or lines at all.
I don't believe it is possible to conform arbitrary html to 998+2
transparently due to things like pre. I tried a couple years ago but the
best I could come up with was a kludge to wrap lines with <!--
--> which seemed fairly horrible to me and I gave up. If it was just pre,
you could probably do it 90% of the time with a simple implementation but
if it has css, you have to understand that which I think is a dealbreaker.

This all happened before my time but AIUI 8BITMIME came before utf8 had won
out so it allowed nulls which must have been pretty traumatic for software
based on null-terminated strings to support but still requires 998+2 which
hampers its utility for html today. It's simple enough to devise a binary
-> 8bit cte but for the trouble to deploy it, you might as well go all the
way to BINARYMIME.

cheers
john

---

From: Ned Freed <[email-masked]>
Date: Sat, 19 Sep 2020 06:40:19 -0700 (PDT)

> --> which seemed fairly horrible to me and I gave up.

I'm afraid I don't see the relevance. There's a ton of stuff in HTML that works
fine in a web browser but for various reasons doesn't work if at all in email,
and I suspect avoiding all of that is far more of an inconvenience than not
being able to have really long lines of preformatted text.

> If it was just pre,

That's all fine and dandy, but the reality is that encodings are going to get
removed sooner or later, and in certain environments that rightly or wrongly
impose restrictions things like line length, restrictions which will result in
suboptimal behavior.

In any case, my point wasn't that you can't or shouldn'tuse encodings to send
text material with arbitrarily long lines through email. Of course you can, and
there are plenty of cases where there's no useful alternative. But in the
specific case of HTML, the results can be less than ideal, so some
consideration should be given to avoidance. (Of course there's always the "view
this email in a web browser" trick...)

> -> 8bit cte but for the trouble to deploy it, you might as well go all the

We tried throwing this particular party. Only a few RSVPs, and then nobody
showed up.

				Ned