Subject: Re: [ietf-smtp] Compressing SMTP streams

From: Tony Finch <[email-masked]>
Date: Mon, 1 Feb 2016 12:47:12 +0000

Carl S. Gutekunst <[email-masked]> wrote:
>
> We assume that binary MIME (ala RFC 3030) is dead.

BURL is designed to work with BDAT, so that you can submit a message in
chunks with verbatim parts (BDAT) and out-of-line attachments (BURL).

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Fair Isle, Faeroes, Southeast Iceland: Cyclonic gale 8 to storm 10, decreasing
5 to 7 at times. Very rough or high, becoming very high for a time. Rain or
squally showers. Poor, occasionally good.

---

From: Martijn Grooten <[email-masked]>
Date: Sat, 6 Feb 2016 11:32:26 +0000

On Sat, Feb 06, 2016 at 11:15:54AM +0100, Aaron Zauner wrote:
> Do you guys have any numbers on this? I.e. what the advantage and
> compression ratio for your average mail traffic will be? I suspect
> compression is helpful in SMTP but it may also introduce
> vulnerabilities in combination with TLS. CRIME wasn't the only attack
> on compression, there's also been application layer specific attacks
> BREACH for example (breachattack.com). A team is currently working on
> improving these attacks in application layer protocols, circumvent
> counter-measures in clients et cetera (from a talk at             
> RealWorldCrypto2016 -
> https://drive.google.com/file/d/0Bzm_4XrWnl5zMkJJdHo0Rml4bXM/view?usp=sharing).                                                                                                                                                                          

I think it's fair to say (as others have done already) that none of
these attacks work against SMTP as they all require the attacker to
force the client to make specific requests to the target.

But these attacks also show that compression and encryption don't go
well together. And crypto is hard and provides plenty of opportunities
to mess up. For that reason, I would suggest following TLS 1.3 and not
combine the two, as it would teach people bad habits.

Martijn.

---

From: Aaron Zauner <[email-masked]>
Date: Sat, 6 Feb 2016 11:15:54 +0100

Hi,

> On 29 Jan 2016, at 19:07, John Levine <[email-masked]> wrote:
> 
>> Compression has been removed completely from TLS v1.3, the outcome of
>> the room consensus at IETF-89.
> 
> Bummer.

No, it's a security *feature*.

> 
> Well, in that case, here's a straw man proposal.
> 
> The extension name is COMPRESS, the EHLO keyword is COMPRESS and is
> followed by a space-separated list of compression schemes, currently
> consisting only of DEFLATE (RFC 1951.)
> 
> There's one new command, COMPRESS which takes as an argument the type
> of compression to be used.  If you want to do both STARTTLS and
> COMPRESS, the results of doing COMPRESS before STARTTLS are
> aggessively undefined.
> 
> The responses to COMPRESS are:
> 
> 500 compress not supported
> 501 compression scheme unknown
> 220 go ahead

I'm strongly opposed to this.

Do you guys have any numbers on this? I.e. what the advantage and compression ratio for your average mail traffic will be? I suspect compression is helpful in SMTP but it may also introduce vulnerabilities in combination with TLS. CRIME wasn't the only attack on compression, there's also been application layer specific attacks - BREACH for example (breachattack.com). A team is currently working on improving these attacks in application layer protocols, circumvent counter-measures in clients et cetera (from a talk at RealWorldCrypto2016 - https://drive.google.com/file/d/0Bzm_4XrWnl5zMkJJdHo0Rml4bXM/view?usp=sharing).

Another problem with SMTP extensions is that mail daemons are rarely updated thus it takes quite some years to have real support on the internet.

Aaron

---

From: Aaron Zauner <[email-masked]>
Date: Sat, 6 Feb 2016 12:40:21 +0100

> On 06 Feb 2016, at 12:32, Martijn Grooten <[email-masked]> wrote:
> 

Well, yes. You just can't do JavaScript via SMTP (yet) :)

> combine the two, as it would teach people bad habits.

There aren't that many people looking into this, with renewed research efforts I fear that these attacks may improve significantly. The authors of mentioned work on improving these attacks explicitly mentioned other application layer protocols than HTTP(S) in the Q/A after the talk. They will also be presenting new attacks at upcoming conferences. As I see it - adding compression might reduce traffic load for some providers but at the same time may introduce new vulnerabilities in the future to already poorly-secured protocols like SMTP.

Aaron

---

From: Paul Smith <[email-masked]>
Date: Mon, 8 Feb 2016 09:50:43 +0000

On 06/02/2016 11:32, Martijn Grooten wrote:
> combine the two, as it would teach people bad habits. Martijn.

AFAICS, CRIME isn't really a issue of 'compression & encryption not 
going together', but that the nature of compression lets you infer 
information about the compressed data. It doesn't seem to attack the 
encryption at all.

So, if you came up with a compression algorithm and gave me the 
compressor but not the decompressor, I could still (potentially) work 
out what some compressed data is by using the 'CRIME' method. The 
encryption acting on top of the compression essentially converts the 
known compression algorithm into an unknown one (unless you know the 
decryption key).

(I may totally be misunderstanding it, but from what I've read, that 
seems to be the case).

If you do per-message compression, then I can't really see how something 
like CRIME could possibly work. CRIME seems to work by using the same 
compression stream over known and unknown data, so by seeing how it 
compresses, you can infer something about the unknown data because you 
know the known data and how the compressor works. Also, it's a 'divide 
and conquer' system, so needs lots of transactions - possibly 
exponentially more the bigger the compressed data is.

So, to do it in SMTP message compression, you'd need to inject some 
arbitrary known text into an unknown message. I can't see how you could 
do that unless you have actually had control of the original message - 
in which case the message wouldn't be unknown... In the known HTTPS 
CRIME exploit, the HTTPS client can inject known data into the HTTP 
request which gets added to unknown content (the web cookies) by the web 
browser so you can see how they interact via the compression.

You'd also need to do it lots of times with the same unknown message. 
Probably many more times than the HTTPS exploit because a typical RFC822 
message is a lot bigger than a typical web cookie.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 08 Feb 2016 08:55:22 -0800

> exponentially more the bigger the compressed data is.

The potential for an attack like this on per-message encryption comes from
the fact that the leading data in email messages is fairly predictable
(the same few headers, and a given client is likely to generate them in
roughly the same order).  That's where you get the known data from.

Of course, this is way worse with the full SMTP dialog, since the various
SMTP-level commands are even more predictable.  But at least in theory one
might be able to derive some information about the contents of messages by
using the known text in the message headers.  (In practice, I think this
would be substantially harder than CRIME.)

-- 
Russ Allbery ([email-masked])              <http://www.eyrie.org/~eagle/>

---

From: Martijn Grooten <[email-masked]>
Date: Mon, 8 Feb 2016 18:07:59 +0000

> information about the compressed data. It doesn't seem to attack the

Well, the idea behind CRIME is that data strings compress to smaller
strings if there is more repetition in the original string and that
encryption doesn't change the length of a plaintext. So if you're able
to make small modifications to an otherwise unknown plaintext and you
can read (the length of) the encrypted output, you can determine whether
the bit you controlled occurred elsewhere in the plaintext. CRIME uses
this to step-by-step reconstructs a cookie.

These kinds of attacks are the reason why people say that compression
and encryption don't go together, even if you are technically right that
it's all compression's fault. So perhaps it would be better to say: if
it's sensitive, don't compress it.

> If you do per-message compression, then I can't really see how
> something like CRIME could possibly work. CRIME seems to work by
> using the same compression stream over known and unknown data, so by
> seeing how it compresses, you can infer something about the unknown
> data because you know the known data and how the compressor works.
> Also, it's a 'divide and conquer' system, so needs lots of
> transactions - possibly exponentially more the bigger the compressed
> data is.
> 
> arbitrary known text into an unknown message. I can't see how you
> could do that unless you have actually had control of the original
> message - in which case the message wouldn't be unknown... In the
> known HTTPS CRIME exploit, the HTTPS client can inject known data
> into the HTTP request which gets added to unknown content (the web
> cookies) by the web browser so you can see how they interact via the
> compression.
> 
> You'd also need to do it lots of times with the same unknown
> message. Probably many more times than the HTTPS exploit because a
> typical RFC822 message is a lot bigger than a typical web cookie.

This is all true. I can't see how a CRIME-like attack on SMTP+TLS could
work either.

But cryptography tends to set itself very high standards, in which "I
can't see how" just isn't good enough. Even if you'd be absolutely
certain(*) that such attacks would never be found, I still think it's a
bad idea: implementing cryptography is hard enough as it is; people make
mistakes all the time. Teaching them bad practices should thus be
avoided, even in cases where it doesn't really matter.

Unless, of course, network connections around the world are falling over
because of all the emails. I don't run any large mail servers, so please
correct me if I'm wrong, but a back-of-the-envelope calculation suggests
that this is unlikely to be the case.

(*) and given that, as Aaron pointed out, mail servers aren't updated
often, you would really have to be absolutely certain.

Martijn.

---

From: Aaron Zauner <[email-masked]>
Date: Tue, 9 Feb 2016 00:44:37 +0100

> roughly the same order).  That's where you get the known data from.

Previous posters to the SHUTUP mailing list (maybe also this one?
just joined) suggested only compressing the message data, that would
improve security there by a bit.

I've done implementations of CRIME on custom protocols a few years
back, to be effective an attacker must be able to alter parts of the
response ciphertext in order to guess using compression for the
attack. This isn't easy in SMTP as other people pointed out in the
thread already.

> would be substantially harder than CRIME.)

Most certainly. But I'm still unsure what the added value of
compression in SMTP is there. Ideally I'd like to see some
measurements, these days most messages are rather short.

Aaron

---

From: Brandon Long <[email-masked]>
Date: Mon, 8 Feb 2016 16:07:34 -0800

On Mon, Feb 8, 2016 at 3:44 PM, Aaron Zauner <[email-masked]> wrote:

> something

Your message, as received by me:
 9520  0.txt
 4679  0.txt.gz

Small messages compress great!

But I'm guessing that's not the question you're asking.

As pointed out upthread, for historical reasons, most binary attachments
are base64 encoded, which increases their size of 4/3rds, and with
compression you get nearly all of that back... and that's with original
data which is relatively uncompressible (jpeg/mp4).

And, for many providers, their email streams are a relatively small
fraction of their bandwidth, certainly here at Google we spend a lot more
bandwidth on video.  That said, our bandwidth isn't free, and we have
enough mail volume where reducing that by 25% would mean real savings.
Will we ever reach that level?  I don't know.  We never will if we don't at
least start the discussion.

I think the MSA case, especially for mobile devices, may be even more
interesting.  I know that when we added support for COMPRESS for IMAP,
mobile clients adopted it almost immediately.

As for the security implications, I won't pretend that I know enough there
to be a good judge, though it's also not clear to me that we should
immediately shy away from this since there may be dragons there.

Brandon

---

From: "John Levine" <[email-masked]>
Date: 9 Feb 2016 01:10:08 -0000

>immediately shy away from this since there may be dragons there.

Someone pointed out this could be bad news for spam filtering.  At
this point, it's typical to partly or completely skip the filtering on
very large messages, partly for efficiency, partly because it's not
cost effective for spammers to send lots of giant messages.

But imagine a compressed spam that consists of the payload followed by
a very large image consisting of 50MB of the same byte.  With zlib
compression, that 50MB will compress down to 50KB, so now it's easy
for spammers to send. (If your maximum message size is less than 50MB,
substitute that in, the result doesn't change much.) When you consider
the extra load on the spam filters the compression might not be such a
great tradeoff.  I suppose you could invent new heuristics that score
against highly compressible messages, but who knows how they'd game
that.

R's,
JOhn

---

From: Brandon Long <[email-masked]>
Date: Mon, 8 Feb 2016 17:23:22 -0800

> against highly compressible messages, but who knows how they'd game

We've already gone through phases of image spam, pdf spam, powerpoint spam,
many of which were quite a bit larger than regular spam messages.

I guess one could disable compression against unknown/low reputation/low
volume IPs in that case.

It seems to me any "break out" spam like this is typically short lived, as
filters adjust and then it becomes a small niche or goes away entirely as
too much trouble for no real gain.

Brandon

---

From: Chris Lewis <[email-masked]>
Date: Mon, 8 Feb 2016 22:01:45 -0500

> cost effective for spammers to send lots of giant messages.

When you have 10s of thousands of infected machines at your disposal, 
cost effectiveness isn't terribly important.

Eg: the high volume spams I saw that contain 1Mb of obfuscatory gunk 
just to push a single line of payload.

---

From: Paul Smith <[email-masked]>
Date: Tue, 9 Feb 2016 09:24:35 +0000

On 08/02/2016 23:44, Aaron Zauner wrote:
> measurements, these days most messages are rather short. Aaron

Not a perfect metric, but our mail server software stores messages in a 
database which compresses the messages individually. We typically find 
it compresses to around 50% of the original size. I haven't yet seen a 
customer installation where it's significantly different from that. So, 
for me, that seems reasonable as an average. Obviously some messages 
will compress a lot more, and others less.

---

From: Paul Smith <[email-masked]>
Date: Tue, 9 Feb 2016 09:49:26 +0000

On 08/02/2016 18:07, Martijn Grooten wrote:
> and encryption don't go together, even if you are technically right 
> that it's all compression's fault. So perhaps it would be better to 
> say: if it's sensitive, don't compress it. 

You can't say that...

If 'compression and encryption don't go together', then you have to stop 
people sending JPG, DOCX, ZIP, GIF etc files as well. Those are 
compressed, and they are sent over encrypted links.

Should HTTPS servers stop delivering JPEG files? Do we force people to 
download images as BMP files?

No, because it's not that 'compression and encryption don't go 
together', but that compression can be used *in a very specific way* to 
help find out the content of the encrypted data. It does not break the 
encryption, and it is not a weakness of the encryption.

(AFAICT, CRIME/etc will not help determine the content of JPG/DOCX/etc 
files. It is purely "if I send known data along with unknown data and 
can tell the compressed size, then, after lots of tries I can guess what 
the unknown data is that was sent with my known data". If I am simply 
receiving something that I have no control over (or sniffing a stream 
that I can't inject into), then I have nothing to work with, and the 
data is safe. SMTP is just "receiving something".)

As far as I can see, the CRIME method isn't really even an 'attack on 
HTTPS'. It's actually an attack on the web browser. If you write a 
standalone 'CRIME client' it'd be useless. The CRIME attack is 
attempting to determine the cookies stored in a web browser. It doesn't 
allow the attacker to determine the content downloaded from the HTTPS 
server, just the cookie content sent by the browser. It's very specific, 
and there's nothing like it, even in concept, in SMTP.

> This is all true. I can't see how a CRIME-like attack on SMTP+TLS 
> could work either. But cryptography tends to set itself very high 
> standards, in which "I can't see how" just isn't good enough.

But it is.

I "can't see how" TLS can be snooped on usefully, but that doesn't mean 
we throw it away. You can't prove a negative. Until someone has 
discovered a weakness, or at least a theoretical weakness, then you have 
to assume it's good enough.

All the documentation I've found on CRIME/etc require specific 
behaviour, which is just not there in SMTP. I may be wrong, and someone 
may have worked out a theoretical way to exploit compression in SMTP, 
but just saying 'there may be a way to break this that no one's 
discovered yet' isn't (IMHO) a good enough reason not to do something. 
Otherwise we may as well just abandon all attempts at encryption, as 
there may be ways to break those that no one has discovered yet.

Until someone comes up with an SMTP extension which lets a third party 
say to an unrelated MTA "send that last message again with this 
arbitrary data added at the start" (so it can be sniffed to try to 
determine the newly compressed message size and compare it to other 
compressed sizes of the same message with different data at the start), 
then I can't see how CRIME or anything like it could work with SMTP.

It has to be a third party telling the MTA to send again, because the 
two MTAs involved in the transaction can already see the 
uncompressed/unencrypted data, so any attempt for them to do a 
'CRIME-like' attack is totally pointless. In CRIME, there are three 
parties involved - the browser, the server and the page content/script. 
it's this third thing which does the attack. In SMTP there are 
(currently) only two, so there is no agent available to perform the attack.

---

From: Tony Finch <[email-masked]>
Date: Tue, 9 Feb 2016 10:48:12 +0000

Martijn Grooten <[email-masked]> wrote:
>
> I can't see how a CRIME-like attack on SMTP+TLS could work either.

If you can identify an auto-responder which sends using SMTP AUTH PLAIN or
LOGIN, and if the SMTP AUTH and message envelope are in the same
compression context, you can use the coupling between the credentials and
your choice of recipient address to attack the credentials. Everything
else in the SMTP transaction up to that point is fixed.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Viking, North Utsire: Westerly at first in south, otherwise cyclonic, becoming
northerly later in Viking, 5 or 6. Moderate or rough. Showers. Good.

---

From: "John Levine" <[email-masked]>
Date: 9 Feb 2016 16:07:24 -0000

>just to push a single line of payload.

It's an order of magnitude issue.  A megabyte isn't large these days,
but that 1Mb of gunk could now turn into 1Gb of gunk, or whatever size
the bad guys think is enough to get less stringent filtering.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 9 Feb 2016 16:09:05 -0000

>else in the SMTP transaction up to that point is fixed.

Wouldn't my CDAT that only compresses the data prevent that?

If I were a bad guy, the auth credentials would usually be the most
valuable thing to steal out of the session.

R's,
John

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 9 Feb 2016 16:16:39 +0000

> Wouldn't my CDAT that only compresses the data prevent that?

So it would. But so would a stream-level compression command provided it's 
only available after authentication.

IMO it all doesn't matter. The two are very, very similar, "done" vs 
"vapourware" is the only significant difference.

Arnt

---

From: Tony Finch <[email-masked]>
Date: Tue, 9 Feb 2016 17:35:47 +0000

John Levine <[email-masked]> wrote:
> >your choice of recipient address to attack the credentials.
>

Yep! :-)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Sole, Lundy, Fastnet, Irish Sea, Shannon, South Rockall: Westerly or
northwesterly 7 to severe gale 9, decreasing 5 to 7 later. High at first in
Sole, Shannon and Rockall, otherwise rough or very rough, but becoming
moderate in Irish Sea. Wintry showers. Good, occasionally poor.