Subject: [ietf-smtp] Compressing SMTP streams

From: Brandon Long <[email-masked]>
Date: Thu, 28 Jan 2016 13:42:36 -0800

Has there been any previous discussion on having a COMPRESS like extension
for SMTP?

Similar to RFC 4978 for IMAP.

I was hoping we could get this for free with TLS, but few libraries
implement it and https://en.wikipedia.org/wiki/CRIME seems to have made
them leary about it.  I don't think SMTP is implicated because it doesn't
use cookies/etc, but anyways.

There seems to have been some discussion ~13 years ago, but it went no
where.

As for my reasoning, at least for us:

1) CPU is cheaper than bandwidth
2) message sizes are increasing, but base64 isn't going away

Theoretically, switching to some combination 8BITMIME, CONTENTBINARY and
BDAT, we'd save nearly as much bandwidth since a good fraction of the
compression is going to be recovering the base64 encoding overhead on
attachment types that aren't likely to compress much.  That said, with the
advent of DKIM, we can't easily change the encoding of messages without
breaking signatures, so we're kind of stuck using base64.

DKIM and ARC are also leading to larger headers, which should compress
relatively well, not to mention larger HTML text parts.

Of course, email is only a fraction of modern networks, but the overall
volumes at medium to large esps would still make a useful reduction.

Brandon

---

From: "John Levine" <[email-masked]>
Date: 29 Jan 2016 06:17:51 -0000

>Has there been any previous discussion on having a COMPRESS like extension

Now and then, but the general feeling has been that it's not worth it.

>use cookies/etc, but anyways.

I agree that the CRIME cookie issue seems irrelevant to SMTP since
there isn't any context saved from one session to the next.  TLS has
had perfectly good compression features defined for over a decade, so
I'd rather that we encourage people to use those rather than invent
yet another layer of goop for SMTP.

It looks to me like openssl includes the compression code, and a lot
of mail software uses openssl, so it might be an interesting
experiment to try enabling it in some SMTP clients and see how many
servers are willing to do it.

R's,
John

---

From: "Carl S. Gutekunst" <[email-masked]>
Date: Thu, 28 Jan 2016 23:12:28 -0800

> 2) message sizes are increasing, but base64 isn't going away

Are they?

It's an honest question; you have vastly better data to work from than I 
do. But I have not seen much growth in message size once we got past the 
bump from plain text to HTTP, and the occasional bump from another 
vendor adding more cruft to the header. Attachments can be large, but 
the user community seems to have generally accepted that Email isn't an 
appropriate transport for "large" things and tends to avoid it. The most 
common attachment types also are not compressible: PDF, JPEG, pkzip.

The exceptions can be spectacular; I see marketing types Emailing 150MB 
powerpoint files every day, and those seem to be quite compressible. But 
as annoying as they can be, they do seem to be the exception.

<csg>

---

From: Gilles Chehade <[email-masked]>
Date: Fri, 29 Jan 2016 08:44:10 +0100

> yet another layer of goop for SMTP.

very much agreed :-)

-- 
Gilles Chehade

https://www.poolp.org                                          @poolpOrg

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 29 Jan 2016 07:57:18 +0000

Brandon Long writes:
> Has there been any previous discussion on having a COMPRESS 
> like extension for SMTP?

The matter was discussed at an eatery across the street from an IETF venue 
a few hours after IMAP COMPRESS had been discussed at the meeting. Summary: 
SMTP has less upside than IMAP in terms of compression ratio and is more 
difficult since there are so many implementations, "so let's do just IMAP".

At the moment I think there are many more SMTP implementations than TLS 
implementations, so reaching critical mass on TLS might be easier.

> Theoretically, switching to some combination 8BITMIME, 
> CONTENTBINARY and BDAT, we'd save nearly as much bandwidth since 
> a good fraction of the compression is going to be recovering the 
> base64 encoding overhead on attachment types that aren't likely 
> to compress much.

Announce that messages to gmail from compressing clients will get a bonus 
by your spam filter, and wide deployment will follow overnight ;)

Arnt

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 29 Jan 2016 12:01:02 +0100

On Fri 29/Jan/2016 08:12:28 +0100 Carl S. Gutekunst wrote:
> 
> The most common attachment types also are not compressible: PDF, JPEG,
> pkzip.

I seem to worsen compression by ~102.5% on zip files, running something like:

   for f in *.zip; do
      os=$(stat -c %s $f)
      vs=$(base64 < $f | gzip -c | wc -c)
      printf '%-28s %10d %10d %8.2f%%\n' \
         "$f" $os $vs "$(echo 100\*$vs/$os | bc -l)"
   done | sort -rnk 2

Ale

---

From: Tony Finch <[email-masked]>
Date: Fri, 29 Jan 2016 13:33:42 +0000

John Levine <[email-masked]> wrote:
> there isn't any context saved from one session to the next.

I'm not so sure.

You might be able to perform a CRIME attack against SMTP AUTH if you can
cause an AUTH user to send messages with envelope addresses under your
control, for instance if the AUTH user has an auto-responder.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Tyne, Dogger, Fisher, German Bight: West or southwest 7 to severe gale 9,
occasional storm 10 at first except in German Bight. Rough or very rough,
becoming high at times except in Tyne. Rain or showers. Good, occasionally
poor.

---

From: "Carl S. Gutekunst" <[email-masked]>
Date: Fri, 29 Jan 2016 09:01:24 -0800

> yet another layer of goop for SMTP.

Compression has been removed completely from TLS v1.3, the outcome of 
the room consensus at IETF-89. It's already gone from LibreSSL and Rich 
Salz indicated it would be removed from OpenSSL by the end of the year. 
It's not just the CRIME attack, but a whole range of vulnerabilities 
(mostly side-channel attacks, but others too) inherent to compression 
that the crypto community seems to believe cannot be resolved. 
Basically, they're just tired of it.

There was some interesting discussion on content-based compression, but 
the TLS WG just didn't want to go there.

<csg>

---

From: Brandon Long <[email-masked]>
Date: Fri, 29 Jan 2016 09:34:56 -0800

>> 2) message sizes are increasing, but base64 isn't going away

The averages aren't, but the distribution tends to have multiple peaks
where they include the most common sizes of images.

I guess, what I'm really saying, is the volume is increasing, but maybe the
difference only matters at a high enough volume.

> bump from plain text to HTTP, and the occasional bump from another vendor
> adding more cruft to the header. Attachments can be large, but the user
> community seems to have generally accepted that Email isn't an appropriate
> transport for "large" things and tends to avoid it. The most common
> attachment types also are not compressible: PDF, JPEG, pkzip.
>
> powerpoint files every day, and those seem to be quite compressible. But as
> annoying as they can be, they do seem to be the exception.

Most Office document types switched to be a compressed format several years
back, but if a powerpoint has a video or something in it, that's not going
to compress.

And the GfW folks are always pushing us to increase the max message size we
allow.

My main point was that with base64 encoding of a jpg, there's still a
recovery ratio in the 25% range.

Ie, a local mp4:
blong@blong-x1:~$ ls -l tom_nhl.mp4*
-rw-r----- 1 blong eng 21566364 Aug 10 02:09 tom_nhl.mp4
-rw-r----- 1 blong eng 29133510 Jan 29 09:29 tom_nhl.mp4.b64
-rw-r----- 1 blong eng 22092559 Jan 29 09:29 tom_nhl.mp4.b64.gz

so, not quite back to the original size, but close, 24% reduction.

Brandon

---

From: "John Levine" <[email-masked]>
Date: 29 Jan 2016 18:07:13 -0000

>the room consensus at IETF-89.

Bummer.

Well, in that case, here's a straw man proposal.

The extension name is COMPRESS, the EHLO keyword is COMPRESS and is
followed by a space-separated list of compression schemes, currently
consisting only of DEFLATE (RFC 1951.)

There's one new command, COMPRESS which takes as an argument the type
of compression to be used.  If you want to do both STARTTLS and
COMPRESS, the results of doing COMPRESS before STARTTLS are
aggessively undefined.

The responses to COMPRESS are:

500 compress not supported
501 compression scheme unknown
220 go ahead

After a 220 response, subsequent traffic is compressed.  It's up to
each end to ensure that there's a compression block boundary every
time the transmission direction changes, i.e., after each command or
response, or if pipelining after each pipelined group.  If the
compress command fails, the client can at its option continue an
uncompressed session or give up.

Example

S: 220 mx.example smail2 ESMTP

C: EHLO client.example

S: 250-mx.example
S: 250-STARTTLS
S: 250-8BITMIME
S: 250 COMPRESS DEFLATE

C: STARTTLS

S: 220 go ahead

<TLS negotiation>

C: EHLO client.example

S: 250-mx.example
S: 250-8BITMIME
S: 250 COMPRESS DEFLATE

C: COMPRESS DEFLATE
S: 220 go ahead

<subsequent traffic is compressed>
 ... blah blah ...

C: QUIT

S: 221 sayonara

R's,
John

---

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2016 14:45:30 -0500

> The takeaway here is to 1) not allow compression until after any
> authentication has been done and 2) flush the compression state between
> messages (if the sever supports sending multiple messages over the same
> SMTP session).

Thanks.  Those are easy enough to do, as advice if not as part of the 
protocol.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

---

From: Brandon Long <[email-masked]>
Date: Fri, 29 Jan 2016 12:11:23 -0800

That's about exactly what I was thinking.

Brandon

On Fri, Jan 29, 2016 at 10:07 AM, John Levine <[email-masked]> wrote:

> _______________________________________________
> Shutup mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/shutup
>

---

From: "Carl S. Gutekunst" <[email-masked]>
Date: Fri, 29 Jan 2016 13:04:26 -0800

> After a 220 response, subsequent traffic is compressed.

Yes, order must be STARTTLS, then AUTH, then COMPRESS.

We assume that binary MIME (ala RFC 3030) is dead.

Alternative: Is there any benefit to compressing the envelope? With 
pipelining, probably; otherwise no. If we don't compress the envelope, 
then we could make compression an option on the DATA command. Now we 
have something that starts to look like RFC 3030, but without the 
assumption that the sender should not use Base64 (with its attendant and 
DKIM-breaking content transfer encoding conversion at each hop).

<csg>

---

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2016 16:10:20 -0500

> That's about exactly what I was thinking.

Here's a wild'n'crazy idea -- how about only compressing the 
client->server direction?  SMTP responses are never very long.

---

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2016 16:12:53 -0500

> pipelining, probably; otherwise no. If we don't compress the envelope, then 
> we could make compression an option on the DATA command.

Interesting idea.  In principle, you can pipeline a blat of a hundred RCPT 
TO and a hundred responses, but in practice if a message has three 
recipients it's a lot.

R's,
John

---

From: Brandon Long <[email-masked]>
Date: Fri, 29 Jan 2016 13:14:06 -0800

On Fri, Jan 29, 2016 at 1:04 PM, Carl S. Gutekunst <[email-masked]> wrote:

> we could make compression an option on the DATA command. Now we have
> something that starts to look like RFC 3030, but without the assumption
> that the sender should not use Base64 (with its attendant and DKIM-breaking
> content transfer encoding conversion at each hop).

1000 recipients at ~30 bytes each with each domain likely the same... maybe
sometimes.

If you're only compressing the data, what's the interaction with RFC 1870?
Do you use the compressed size or the original, and if you use the
compressed size, does that mean the server will accept larger uncompressed
messages?

And I'd think it's an alternative to BDAT, same thing, but compressed?
Unless we extend the BDAT syntax for compression...

Brandon

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Jan 2016 13:19:38 -0800

+ietf-nntp, who may be interested

> Well, in that case, here's a straw man proposal.

[snip proposal for COMPRESS extension to smtp]

BTW, in the NNTP world, we're currently testing:

    http://tools.ietf.org/id/draft-murchison-nntp-compress-01.html

and will probably have a new version of that draft based on early interop
results.

-- 
Russ Allbery ([email-masked])              <http://www.eyrie.org/~eagle/>

---

From: "John Levine" <[email-masked]>
Date: 29 Jan 2016 21:42:19 -0000

>1000 recipients at ~30 bytes each with each domain likely the same... maybe

Can you make any estimates of how common large recipient lists are?  My
impression is that since most bulk mail is customized per recipient, big
lists in stuff that isn't spam are rare.

>Do you use the compressed size or the original,

It has to be the original size.  If you're using a stream compressor
like deflate you can't tell what the compressed size will be unless
you make a prepass over the data.  And anyway, my impression is that
the motivation for using 1870 is still the internal storage, not the
transmission time.

>And I'd think it's an alternative to BDAT, same thing, but compressed?

Same problem, client generally doesn't know the compressed size in advance.

R's,
John

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Jan 2016 14:04:39 -0800

> and will probably have a new version of that draft based on early interop

Apologies, there's a current version out already:

    http://tools.ietf.org/id/draft-murchison-nntp-compress-02.html

---

From: Brandon Long <[email-masked]>
Date: Fri, 29 Jan 2016 15:21:58 -0800

> lists in stuff that isn't spam are rare.

Pretty rare, I think our average is in the 1.3 per message range.

> the motivation for using 1870 is still the internal storage, not the

Makes sense.

> Same problem, client generally doesn't know the compressed size in advance.

Hmm, so we'd be stuck with trying to find the EOD marker, or doing some
streaming decompression til we reach the right end size.

I think I prefer the full stream compression.

Brandon

---

From: Brandon Long <[email-masked]>
Date: Fri, 29 Jan 2016 15:30:28 -0800

> Pretty rare, I think our average is in the 1.3 per message range.

Looking so far at today, apparently our 99th percentile is 1 and 99.9th is
7, so yeah, pretty rare.

> I think I prefer the full stream compression.

And the nntp compress extension sounds also identical, a stream one.  Now,
nntp can be used in both an smtp like state between servers and an imap
like state from client to server, but total stream seems to be the common
choice.

I would be remiss if I didn't mention that one of my colleague's here at
Google thinks this is all silly, and we should instead figure out how to
not base64 encode the messages in the first place, but I think that's a
higher barrier than this.

Brandon

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Jan 2016 16:21:13 -0800

> higher barrier than this.

We specified the NNTP protocol as being basically binary-clean (you still
have to do dot-stuffing, and you have to have a newline at the end of the
message, but it's mostly there), and most implementations *mostly* cope,
but clients are another matter entirely.  (Ideally, we'd want an even more
binary-clean transport, but the NNTP one is close enough.)

Note that, in the NNTP world, yEnc has some traction as an encoding
format.  I tried to convince the author to write it up as a MIME encoding,
with absolutely no luck, but it should be possible to do.  I think Ned
Fried took a crack at it?  It's basically a minimal encoding to deal only
with the flaws found in actual NNTP implementations (such as not coping
with nuls and unmatched CR or LF characters) and dot-stuffing, and is
pretty close to not compressing.

It's kind of a horrible hack, but at least for a while it was very widely
used in the binary Usenet groups since it avoids having to bloat things
with base64 and is pretty close to the unencoded size.  However, I've not
seen it used for anything more general-purpose.

http://www.yenc.org/

---

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2016 20:55:29 -0500

> higher barrier than this.

The good news is that we figured that out in 1995 in RFC 1830, very 
slightly updated in 2000 in RFC 3030.  But the bad news, as you are surely 
aware, is that it's been 20 years, nobody uses it, and since it'd require 
changes to all of the MUAs that create and display MIME messages, it's 
unlikely to happen.

Given that the bloat from base64 is only 25%, how much real benefit is 
there likely to be from compression?  I know that people mail around giant 
Powerpoint and Autocad files, but I have no idea how compressible they 
are.  Recent MS office formats pptx, xlsx, and so forth are actually zip 
files, so they're not going to compress at all beyond what you get from 
undoing base64.

R's,
John

---

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2016 22:09:37 -0500

Here's an alternative hack that only compresses the message body.  I think 
it avoids the CRIME problem since it seems it'd be pretty hard to predict 
where in a stream the encoded data would start.  It is about as 
effective as compressing the whole stream since the SMTP commands and 
responses are generally very short, while making the code considerably 
simpler since only the part that sends or receives the message body needs 
to understand compression.

The EHLO keyword is still COMPRESS followed by compression schemes, 
currently DEFLATE only.  Since you can't add arguments to the DATA 
command, this invents a new one CDAT which takes as its argument the 
compression scheme.  CDAT is like DATA, except that after a 354 response, 
the client sends compressed data up to and including the \r\n.\r\n at the 
end of the message.  The responses to CDAT are the same as to DATA and in 
addition

Example:

C: EHLO client.example
S: 250-mx.example
S: 250-STARTTLS
S: 250-8BITMIME
S: 250 COMPRESS DEFLATE

C: STARTTLS
S: 220 go ahead

<TLS negotiation>

C: EHLO client.example
S: 250-mx.example
S: 250-8BITMIME
S: 250 COMPRESS DEFLATE

C: MAIL FROM:<[email-masked]>
S: 250 OK

C: RCPT TO:<[email-masked]>
S: 250 OK

C: CDAT DEFLATE
S: 354 send your compressed message
C: ... compressed message and \r\n.\r\n ...
S: 250 message received

C: QUIT
S: 221 sayonara

Regards,

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Jan 2016 19:42:08 -0800

"John R Levine" <[email-masked]> writes:

> Here's an alternative hack that only compresses the message body.  I
> think it avoids the CRIME problem since it seems it'd be pretty hard to
> predict where in a stream the encoded data would start.  It is about as
> simpler since only the part that sends or receives the message body
> needs to understand compression.

We considered this in NNTP and abandoned it because the implementation
complexity seemed a lot higher for the typical NNTP implementation.  The
advantage of compressing the whole stream in a protocol that already
supports TLS and SASL is that you just stack the compression into your
transport layer, and you already have a system for dealing with that.
This requires building knowledge higher in the stack in the command
processor.

That said, NNTP and SMTP differ here in that NNTP has multiple different
commands that send or receive larger data, whereas SMTP basically has only
the one, so the issues don't apply in the same way.  Also, there's the
CRIME problem -- the NNTP proposal tries to deal with the most obvious
implications by requiring AUTHINFO happen before compression is
negotiated, but there are probably other issues even after that one.

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 30 Jan 2016 08:23:58 +0000

John,

> On 30 Jan 2016, at 03:09, John R Levine <[email-masked]> wrote:
> 
> Here's an alternative hack that only compresses the message body.  I think it avoids the CRIME problem since it seems it'd be pretty hard to predict where in a stream the encoded data would start.  It is about as effective as compressing the whole stream since the SMTP commands and responses are generally very short, while making the code considerably simpler since only the part that sends or receives the message body needs to understand compression.
> 
> The EHLO keyword is still COMPRESS followed by compression schemes, currently DEFLATE only.  Since you can't add arguments to the DATA command, this invents a new one CDAT which takes as its argument the compression scheme.  CDAT is like DATA, except that after a 354 response, the client sends compressed data up to and including the \r\n.\r\n at the end of the message.  The responses to CDAT are the same as to DATA and in addition

If we are not doing the full stream compression (which I prefer), I would really like a BDAT variant, it is easier to code. (Speaking from experience implementing both client and server side.)
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 30 Jan 2016 08:25:40 +0000

Hi Russ,

> On 30 Jan 2016, at 03:42, Russ Allbery <[email-masked]> wrote:
> 

Exactly.

> the one, so the issues don't apply in the same way.

Although there are various SMTP extensions adding more commands.

>  Also, there's the

---

From: Paul Smith <[email-masked]>
Date: Sat, 30 Jan 2016 09:59:25 +0000

> This requires building knowledge higher in the stack in the command
I prefer the "extra command to compress the message body" method. It 
just seems more SMTP-y to me. Also, I'd have thought it'd be easier to 
implement - anyone could just GZIP the raw message and then send the 
output of that. Doing stream compression would seem to be a lot harder.

---

From: Paul Smith <[email-masked]>
Date: Sat, 30 Jan 2016 10:05:30 +0000

On 29/01/2016 23:30, Brandon Long wrote:
>
>
> I would be remiss if I didn't mention that one of my colleague's here 
> at Google thinks this is all silly, and we should instead figure out 
> how to not base64 encode the messages in the first place, but I think 
> that's a higher barrier than this.
>
The problem is that that requires ALL mail software to support that, 
otherwise you need to have recoding of messages by MTAs, which is 
problematic (eg for DKIM). If you compress messages, then that is done 
on a hop-by-hop basis, and the original message is unaltered.

I know we don't support the 8BITMIME extension in our MTA because I 
don't want to faff around with recoding messages for a small gain, but 
we'd probably do a message-based compression extension (might have to 
think hard about whether to implement stream based compression, and 
would probably decide it's not worth it).

---

From: "John R Levine" <[email-masked]>
Date: 30 Jan 2016 12:36:27 -0500

> If we are not doing the full stream compression (which I prefer), I would really like a BDAT variant, it is easier to code. (Speaking from experience implementing both client and server side.)

Having sketched out some code, I agree.

The problem with doing full stream comprssion is that the compressor is a 
coroutine that sits between the application and the network.  If you also 
are doing TLS, that's already a coroutine between the application and the 
network, so this is now a coroutine in front of the other coroutine.  The 
TLS coroutine code in my MTA is a kludge, and doubling the kludge and 
making it work both with and without TLS would be fragile.  Besides, if we 
only do it for the data I think we can ignore the CRIME issue.

For the DATA only part, it's easy enough to start up zlib or its 
equivalent and tell it to deflate or reflate blocks of data.  But for my 
proposal where the compression stops after \r\n.\r\n it's harder to tell 
it to stop and then to continue processing uncompressed data after that. 
It'd probably work since the client will flush the output but again, it's 
fragile.

So the revised straw man invents a CDAT command which is like BDAT except 
that the data is compressed.  The length is the length of the transmitted 
compressed data, and if your compressor produces blocks of data, it's 
specifically allowed to split a compressed block between two CDAT blocks. 
(This poses no problem to zlib.)  All the other rules for BDAT apply, if 
the message ends in the middle of a MIME chunk or the like, the results 
are undefined.

For performance reasons you want the chunks to be as big as possible, but 
on today's computers I expect that a 10Mb compression buffer is no big 
deal.

Regards,

---

From: "Robert A. Rosenberg" <[email-masked]>
Date: Sat, 30 Jan 2016 14:01:36 -0500

At 15:30 -0800 on 01/29/2016, Brandon Long wrote about Re: 
[ietf-smtp] Compressing SMTP streams:

>I would be remiss if I didn't mention that one of my colleague's 
>here at Google thinks this is all silly, and we should instead 
>figure out how to not base64 encode the messages in the first place, 
>but I think that's a higher barrier than this.

What purpose is served by using BASE64 beyond making the content able 
to travel through a 7-bit-only hop? Since most (all?) hops are now 
8-bit ones, why not just send the actual 8-bit version of the data 
instead of converting it to the 7-bit BASE64 format?

---

From: "John R Levine" <[email-masked]>
Date: 30 Jan 2016 21:48:01 -0500

> What purpose is served by using BASE64 beyond making the content able to 
> travel through a 7-bit-only hop? Since most (all?) hops are now 8-bit ones, 
> why not just send the actual 8-bit version of the data instead of converting 
> it to the 7-bit BASE64 format?

The answer to this question is quite complex.  The 8BITMIME extension was 
defined in 1994, and nearly all current MTAs can handle it.  But all it 
says is that the MTA can handle 8 bit characters.  It doesn't affect the 
rule that the message must consist of lines no more than 1000 characters 
long with \r\n at the end of each.  As far as I can tell, the main effect 
is that people can send ISO-8859-x and UTF-8 without encoding, which is 
useful but generally not a big deal.

BINARYMIME was defined in 2000 to avoid that issue, and invents a new BDAT 
command that uses a byte count rather than escape sequence, so the message 
body can be an arbitrary sequence of octets.  Gmail, hotmail and 
icloud/me.com support it, Yahoo and AOL don't, but I've never seen client 
software that would take advantage of it.

What we've never seen is a quoted-unprintable encoding, which is like QP 
but intended for binary data.  It could be like QP without soft line 
breaks and \r\n pairs are ignored.  It's an obvious idea so there must be 
some reason.  Dave Crocker or John Klensin or Ned Freed would know.

Regards,

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 31 Jan 2016 10:10:54 +0000

> On 31 Jan 2016, at 02:48, John R Levine <[email-masked]> wrote:
> 
> BINARYMIME was defined in 2000 to avoid that issue, and invents a new BDAT command that uses a byte count rather than escape sequence, so the message body can be an arbitrary sequence of octets.  Gmail, hotmail and icloud/me.com support it, Yahoo and AOL don't, but I've never seen client software that would take advantage of it.

BINARYMIME is the preferred mode of operation for Isode's Android email client. I hope others would follow.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sun, 31 Jan 2016 21:41:35 +0000

John R Levine writes:
> BINARYMIME was defined in 2000 to avoid that issue, and invents 
> a new BDAT command that uses a byte count rather than escape 
> sequence, so the message body can be an arbitrary sequence of 
> octets.  Gmail, hotmail and icloud/me.com support it, Yahoo and 
> AOL don't, but I've never seen client software that would take 
> advantage of it.

Zmailer did too, but all told it seems like one of those RFCs where the 
specification's reviewers outnumber its implementations.

Arnt

---

From: Ned Freed <[email-masked]>
Date: Sun, 31 Jan 2016 17:11:33 -0800 (PST)

> Fried took a crack at it?

I did. See:

    https://datatracker.ietf.org/doc/draft-freed-mime-newenc/

for details.

> It's basically a minimal encoding to deal only

Correct. I never pursued it because there didn't seem to be anywhere near
the necessary level of interest.

And it's not hard to see why. The problem with this scheme is that you have
to have support in the clients at both ends, or you end up sending what
amounts to garbage as far as the recipient is concerned.

Of course you could have some way to determine whether or not a given
recipient is capable of receiving such objects. But any scheme that requires
users to maintain this information is pretty clearly a nonstarter, and
how you'd do it automatically without a large chunk of new infrastructure
is unclear to me.

Another concern is that AS/AV software will have to be updated to support
the encoding. Without this support being in place you're going to see messages
blocked because they contain a big blog of stuff the scanner doesn't
understand. I note that this makes any sort of recipient capability
discovery much less valuable.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 1 Feb 2016 02:22:43 -0000

Someone was wondering what sort of nastiness one could do by putting
maliciously crafted data into allegedly compressed data.  I took a look
and using deflate, the malice appears limited.

One thing you could do is to make something that blows up to enormous
size.  It appears to me is that the limit of deflats is about 1000:1.
I compressed a 5 gig file all of the same character, the compressed
version was 5 meg.  That's a lot of expansion, but the obvious response
is to limit the size, the same as with uncompressed data.  Using zlib
it's easy enough to track the decompressed size and stop if you have
more than you want.

The other possibility would be to do something that hangs the
decompressor or at least makes it really slow.  Deflate is pretty
simple, decoding Huffman encoded bit strings, and LZ pointers back to
previous data.  There are bit combinations that are forbidden, but at
someone else's suggeston I looked to see of people have tried fuzz
testing zlib, with randomly permuted data, it appears lots of people
have and nobody's found any bugs for a long time.

So compression may not buy us much, but it seems unlikely to do anything
very evil.

R's,
John

---

From: "Carl S. Gutekunst" <[email-masked]>
Date: Sun, 31 Jan 2016 22:44:57 -0800

>> pretty close to not compressing.

Which do you think makes more sense for encapsulating John's CDAT data, 
something that looks more like BDAT (length indicator), or something 
that looks more like yEnc ("quoted unprintable")?

My intuition is that something like yEnc will be easier to code and 
perhaps more robust, at the considerable expense of having to scan for 
the dot and such. But the yEnc-line encoding could include dot handling. 
And CPU cost doesn't bother me at all when we're already talking about 
compression.

<csg>

---

From: Russ Allbery <[email-masked]>
Date: Sun, 31 Jan 2016 23:28:24 -0800

"Carl S. Gutekunst" <[email-masked]> writes:

> something that looks more like BDAT (length indicator), or something that
> looks more like yEnc ("quoted unprintable")?

BDAT is way cleaner if you can change the protocol.  yEnc is a hack around
not being able to define a new POST or ARTICLE command in NNTP with saner
data handling and instead working around line length limits, CRLF pairing,
and dot stuffing, all of which is painful for binary data.  It's way more
likely to have bugs, whereas BDAT's counted binary data is far cleaner and
easier to implement correctly.