Subject: Re: [ietf-smtp] Draft for compressing SMTP streams

From: Julien ÉLIE <[email-masked]>
Date: Tue, 16 Aug 2016 23:24:11 +0200

Hi John,

> To move the discussion along, here's a draft for the CDAT compressed
> data command.
>
> https://datatracker.ietf.org/doc/draft-levine-smtp-compress/?include_text=1

I see that the draft has recently expired.  Are there any news about it?

In case a -01 version is scheduled, I would like to point out a fairly 
recent updated draft regarding the COMPRESS extension for NNTP:
     https://tools.ietf.org/html/draft-murchison-nntp-compress-05

Section 3 about "Compression Efficiency" may be of interest, especially 
if you wish to add something like that in your draft.
Of course Section 6 about "Security Considerations" could also be a 
useful source of inspiration for SMTP (notably the advice not to 
compress two different messages together, so as not to possibly leak 
information about them, as they should be considered "private").  You 
could say it explicitly:  when you have an end-marker (LAST), the 
reset-marker (RESET) is mandatory.  At least, if I read well the ABNF 
syntax.

As for your "open issues":

    o  Is it worth making provision for multiple compression schemes?
       After 20 years, there still isn't anything much better than
       DEFLATE and zlib.

=> COMPRESS for IMAP permits to use other compression schemes.  We also 
permit that for NNTP.  Better foresee future better algorithms from the 
start.

    o  If a server supports both COMPRESS and CHUNKING [RFC3030], can you
       mix compressed and uncompressed chunks of data?  I don't see why
       not, but ugh.

=> Whatever the final choice is (to allow or not to allow), I believe it 
should be explicitly mentioned in the draft.

    o  Do we need new 5xx codes for bad compressed data, or can we use
       554 for bad data and 552 for too big?

=> According to RFC 5321:
    552  Requested mail action aborted: exceeded storage allocation

    554  Transaction failed (Or, in the case of a connection-opening
       response, "No SMTP service here")

Both of these codes are valid in response to a DATA command, so I think 
it is OK to use them for CDAT compression errors.

Incidentally, are you sure a separate CDAT command is the wisest 
solution for SMTP (instead of a compression layer, like TLS)?
Keep in mind that one of the advantages of a compression layer is that 
you can also decide not to compress data (setting the compression level 
at 0 for commands or answers that do not need being compressed).  The 
client may know that the contents is already compressed (jpg or zip 
attachment for instance) and decide not to compress it again.

As far as NNTP is concerned, we already had separate commands like 
XZVER, XZHDR or XFEATURE COMPRESS (enabling compression for a subset of 
subsequent NNTP commands).  The problem is that they do not work for new 
NNTP extensions; one has to create again other separate commands dealing 
with compression.  That's why I reckon a more global solution (like a 
compression layer) should really be encouraged.  Or, if you really want, 
both CDAT and COMPRESS :)

-- 
Julien ÉLIE

« Il ne faut jamais pleurer d'avoir perdu la lune car les larmes
   empêchent de voir les étoiles. »

---

From: "John Levine" <[email-masked]>
Date: 16 Aug 2016 22:37:56 -0000

>I see that the draft has recently expired.  Are there any news about it?

No.  If there are people who would actually implement it, it's easy
enough to send in a new rev.

>     https://tools.ietf.org/html/draft-murchison-nntp-compress-05

I've looked at it, being one of the 14 people in the world who still
runs a news server.  Despite the superficial syntactic similarities,
SMTP and NNTP are not all that similar.  Most notably, SMTP traffic is
very asymmetric, with only tiny amounts from server to client, and
potentially large messages from client to server.  NNTP can go either
way.

>permit that for NNTP.  Better foresee future better algorithms from the 

Does anyone actually use other compression algorithms?  Options that
nobody actually uses are about the best way to create hard to diagnose
incompatibilities.

>solution for SMTP (instead of a compression layer, like TLS)?

Yes, see previous discussion.  The compression layer requires a
three-way coroutine system which is a lot more fragile than sending
the data through already debugged TLS connection code, and this
completely avoids the CRIME issue and its relatives.

>attachment for instance) and decide not to compress it again.

How is this different from CDAT?  The client can send anything that
DEFLATE can be persuaded to produce.  If you're using zlib, you call
deflateParams to tell it how to change its compression strategy and
keep going.

R's,
John

---

From: Sean Leonard <[email-masked]>
Date: Tue, 16 Aug 2016 23:05:01 -0700

> nobody actually uses are about the best way to create hard to diagnose

LZMA and LZMA2 (container for both uncompressed data and multiple runs 
of LZMA data) are strong general-purpose compression algorithms. They 
are considered better than bzip2 for the same computational overhead, 
and can take advantage of multicore/multithreaded processing. AFAIK the 
compression family is also free.

In crypto, a new algorithm or protocol is invented, and then vetted, and 
then...people don't implement it (widely) except to fulfill some arcane 
government requirements. Then someone figures out how to break the old 
algorithm and everybody jumps onto the new bandwagon because "the sky is 
falling!"

This cycle doesn't happen so much with general-purpose compression 
algorithms. The driver for adopting new compression schemes seems to be 
tied more to the utility of the container format that uses the scheme. 
(See, e.g., LZW and GIF.)

Regards,

Sean

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 17 Aug 2016 13:02:26 +0530

FWIW I wrote 4790, which is fairly widely implemented, and so far noone 
has suggested adding anything beyond Deflate for that, so I guess that 
SMTP implementers will be happy with similar simplicity.

Or perhaps 
not want compression at all. SMTP has different problems than IMAP, 
which I think simple compression will not fix: Submit sees a single 
message per connection, plain SMTP sees mostly spam. Why implement 
compression, then?

Arnt

Arnt

---

From: Julien ÉLIE <[email-masked]>
Date: Wed, 17 Aug 2016 14:55:39 +0200

Hi John,

> potentially large messages from client to server.  NNTP can go either

OK, I understand that COMPRESS would not be that useful for SMTP 
(contrary to IMAP or NNTP).
Regarding implementation of COMPRESS in the NNTP world, interoperability 
is already proven:  we have a widely used news server (INN) and a news 
client (flnews) that implement it.  Cyrus NNTP server also added support 
for it.
If the draft ever becomes a proposed standard, we can hope other clients 
and servers will adopt it.  Only 13 news servers left to upgrade :)

> nobody actually uses are about the best way to create hard to diagnose

Sean spoke about LZMA/LZMA2 here.  That algorithm also arrived in the 
discussion in the news.software.nntp newsgroup.  Though still not 
supported, it could in the future (or any other algorithm).

Supporting from the start a syntax permitting to use other algorithms 
than DEFLATE is easy to do.  I do not see why supporting that would 
create incompatibilities.

If the client asks for an unknown algorithm, the server will just 
decline it:

       [C] COMPRESS X-SHRINK
       [S] 503 Compression algorithm not supported

Besides, the server advertises the algorithms it knows:

       [C] CAPABILITIES
       [S] 101 Capability list:
       [S] VERSION 2
       [S] READER
       [S] AUTHINFO USER
       [S] COMPRESS DEFLATE
       [S] LIST ACTIVE NEWSGROUPS
       [S] .

Let's activate a compression layer using DEFLATE:

       [C] COMPRESS DEFLATE
       [S] 206 Compression active
       [Henceforth, all traffic is compressed]

> completely avoids the CRIME issue and its relatives.

A separate command will be "easy" to implement in case SMTP software 
cope well with null bytes, as Russ hinted at in the discussion.
Otherwise, a compression layer is easier to add (the same way a TLS 
layer or an SASL layer are).

-- 
Julien ÉLIE

---

From: John C Klensin <[email-masked]>
Date: Wed, 17 Aug 2016 14:19:45 -0400

--On Wednesday, August 17, 2016 13:02 +0530 Arnt Gulbrandsen
<[email-masked]> wrote:

> Or perhaps not want compression at all. SMTP has different
> problems than IMAP, which I think simple compression will not
> fix: Submit sees a single message per connection, plain SMTP
> sees mostly spam. Why implement compression, then?

That seems to me to be a key question, perhaps _the_ key
question.

Relays in SMTP are also much more under the control of the
recipient domain, while NNTP, at least from my perspective, acts
like more of a mesh.   Other issues put more of a premium on
rapid delivery with SMTP than with NNTP.   For many situations,
email also gains much more from viewing compression as something
dealt with as an end-to-end compressed body part issue rather
than as a transport option.

Noting that transport-time compression capability and associated
SMTP extensions have been proposed many times and never gone
anywhere, can we take this discussion up a notch and address the
question of what the requirement for this capability looks like
and what problem it solves?  "We can figure out how to do it" is
obviously an important piece of the puzzle, but it seems to me
that is the easy part... and a part that doesn't lead to a lot
of demand or deployment.

     john
   (who still believes that fewer options promote simplicity
and, in turn, leads to fewer bugs, less unanticipated
vunerability to attack, and better interoperability)

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 18 Aug 2016 12:36:13 +0530

Fwiw I find the arguments below unpersuasive, but here they are and 
maybe we can do this round very quickly:

Compressing can let mobile 
clients upload faster to submission servers, achieving a better 
approximation of atomicity. I find this weak because they do not seem 
to pipeline and hardly anyone seems to have shown any interest in 
lemonade, not even to consider and reject it.

Compressing can help 
ESPs and big receivers. Well, perhaps, but both of those have enough 
bandwidth to do it the simple, reliable way.

Compressing can help 
spammers avoid detection when on botnets by using less bandwidth 
😅

Arnt

---

From: Paul Smith <[email-masked]>
Date: Thu, 18 Aug 2016 09:07:07 +0100

On 18/08/2016 08:06, Arnt Gulbrandsen wrote:
> Compressing can let mobile clients upload faster to submission 
> servers, achieving a better approximation of atomicity. I find this 
> weak because they do not seem to pipeline and hardly anyone seems to 
> have shown any interest in lemonade, not even to consider and reject it.
>
> Compressing can help ESPs and big receivers. Well, perhaps, but both 
> of those have enough bandwidth to do it the simple, reliable way.
>
> Compressing can help spammers avoid detection when on botnets by using 
> less bandwidth 😅 

I think points against it are:

  - it will take several years before any new option is widely 
implemented enough to make any difference. By then it is probable that 
mobile clients will have better access to fast Internet, so the 
percieved benefit to them will be reduced, reducing incentive to 
implement the option.
  - compression will give little real gain for small messages. The 
biggest potential gains would be for big messages. However, big messages 
will usually contain attachments. Many common attachments are already 
compressed (eg JPG, PNG, DOCX etc) so further attempts to compress those 
will gain little. (AIUI base64 encoding, even of uncompressed data, also 
reduces compressability).

---

From: Jeremy Harris <[email-masked]>
Date: Thu, 18 Aug 2016 09:29:04 +0100

On 18/08/16 09:07, Paul Smith wrote:
>  (AIUI base64 encoding, even of uncompressed data, also

Presumably a base64-aware compression method is not impossible?
-- 
Jeremy

---

From: Paul Smith <[email-masked]>
Date: Thu, 18 Aug 2016 09:48:51 +0100

> Presumably a base64-aware compression method is not impossible?

No, but then it would probably be a non-standard compressor, which will 
make implementation more complex. When gains are dubious and/or minimal, 
then you want to make the cost of implementation & possibility of 
introducing bugs as low as possible.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 18 Aug 2016 14:27:11 +0530

It is not quite that simple.

Most compression algorithms expect the 
input to be characters from some alphabet, such as 8-bit bytes. Base64 
changes that, the bytes are not really visible. However, many of the 
input formats do not use bytes anyway, they are more properly seen as 
bit strings composed of variable length tokens. It does not really 
matter whether a compression algorithm sees six of eight bits from the 
middle of that string, when it cannot even tell where each token 
ends.

Au this point you can think of ways to turn six back to eight. 
But does it really matter to anyone?

Arnt

---

From: "John Levine" <[email-masked]>
Date: 18 Aug 2016 14:10:51 -0000

>message per connection, plain SMTP sees mostly spam. Why implement 

I think the motivation is people sending around large documents in
formats like Word and Excel.  Pictures and videos aren't very
compressible, but they're base64 encoded so compressing that tends to
get you back to the original size.  (You don't have to tell me that
BINARYMIME and BDAT solve that particular problem better.)

Early on people from Gmail experessed interest, but I haven't heard
from them for a while.

R's,
John

---

From: Paul Smith <[email-masked]>
Date: Thu, 18 Aug 2016 15:21:24 +0100

On 18/08/2016 15:10, John Levine wrote:
> get you back to the original size.

DOCX and XLSX formats are compressed as well. (They're basically ZIP 
archives of XML & binary data)

So, you can't further compress those, and you can't further compress 
pictures and videos.

Pretty much the only common things you can compress are PDF files. OK 
some people still use DOC & XLS formats (even though they were replaced 
9 years ago), but they're (hopefully) becoming less popular to send by 
email because they can contain macros which DOCX and XLSX files can't, 
so are a common malware vector.

---

From: Sean Leonard <[email-masked]>
Date: Thu, 18 Aug 2016 10:30:37 -0700

+1 to smime@

> On Aug 18, 2016, at 7:10 AM, John Levine <[email-masked]> wrote:
> 
>> Or perhaps 

Modern Word and Excel files are in ZIP container formats. Therefore, they are already compressed. [Looks like Paul Smith already posted about this.]

I would like to see implementations of SMTP compression, but more important than that is binary SMTP (aka BINARYMIME / BDAT), RFC 3030. The difficulty here in addition to general SMTP deployment issues, is that storage has to be binary-clean, which has effects on POP and IMAP. But that is an easy way to save 37%+ on bandwidth.

If the entire payload is base64-encoded and is in standard form (72 characters per line, no weird spaces, etc.), then it would be straightforward to convert the base64 part (of which only 1 exists) to binary, and then de-convert it back for storage.

Servers that support EAI have to support 8-bit clean channels, except for control characters and NULL. Therefore they can be seen as part of the same (not-small) infrastructure upgrade.

S/MIME supports compression, RFC 3274. It is not widely supported, but I know of at least one mail client that supports it. It would be good to support that as well, especially since end-to-end encrypted content will not be further compressible. OpenPGP already supports compression widely (standard algorithms: DEFLATE etc.).

Sean

> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: "John R Levine" <[email-masked]>
Date: 18 Aug 2016 13:40:11 -0400

Took out smime, since this discussion has nothing to do with it.

> Modern Word and Excel files are in ZIP container formats. Therefore, 
> they are already compressed. [Looks like Paul Smith already posted about 
> this.]

Oh, right, forgot about that.

It's looking to me like there are two places where CDAT might be useful. 
One is to squeeze base64 encoded stuff back to about its original size. 
The other, since the compressor is deliberately not reset between 
messages, is when you send approximately but not quite the same message to 
several people in the same session, e.g., different text with the same 
attachments.  Imagine you're an ESP sending tarted up mail with lots of 
images, and each recipient's name is spliced into the text of his message.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

---

From: John Bucy <[email-masked]>
Date: Thu, 18 Aug 2016 14:09:19 -0700

On Thu, Aug 18, 2016 at 10:40 AM, John R Levine <[email-masked]> wrote:

> Modern Word and Excel files are in ZIP container formats. Therefore, they
>> are already compressed. [Looks like Paul Smith already posted about this.]
>>
>
> One is to squeeze base64 encoded stuff back to about its original size. The
> other, since the compressor is deliberately not reset between messages, is
> when you send approximately but not quite the same message to several
> people in the same session, e.g., different text with the same

Agree with this. We (gmail) are still interested but it isn't our highest
priority. The draft seems straightforward, I can probably prototype it on
our smtp gateway in a few days when I can find the time.

cheers
john

---

From: Brandon Long <[email-masked]>
Date: Fri, 19 Aug 2016 15:14:51 -0700

We wouldn't mind switching to a world of using BDAT/BINARYMIME (we support
both, but don't advertise BINARYMIME), but when coupled with DKIM, it
becomes harder, since if we switch to generating cte:binary messages and
signing them, or accepting such messages, how do we forward them to hosts
that don't handle them?

It's kind of the same problem with SMTPUTF8, of course.

And that doesn't include what happens downstream with POP/IMAP clients as
well, in our testing many of them didn't handle binary attachments well
either.

Brandon

---

From: Dave Crocker <[email-masked]>
Date: Fri, 19 Aug 2016 15:21:27 -0700

On 8/19/2016 3:14 PM, Brandon Long wrote:
> if we switch to generating cte:binary messages and signing them, or
> accepting such messages, how do we forward them to hosts that don't
> handle them?

I believe that's the sort of issue that motivated the CTE choices for 
DKIM, however unfortunate.

> And that doesn't include what happens downstream with POP/IMAP clients
> as well, in our testing many of them didn't handle binary attachments
> well either.

Just amazing how much a few decades of legacy baggage can weigh, isn't 
it...?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: "John R Levine" <[email-masked]>
Date: 19 Aug 2016 18:30:19 -0400

> signing them, or accepting such messages, how do we forward them to hosts

I get the impression that for most people, the 25% bloat for base64 isn't 
a big enough deal to be worth a lot of effort to upgrade all the required 
software to handle binary attachments.  If you really care that much, you 
might as well put the attachments on a web server somewhere and use 
message/external body with a pointer to where it's stored.  That's been in 
the MIME standard for 20 years.

R's,
John

> It's kind of the same problem with SMTPUTF8, of course.

Yes, although with SMTPUTF8 we made a fairly clear distinction that you're 
on the bus or you're off, and an entire mail site either supports it or 
not.  The experimental version had a bunch of fallback options that failed 
miserably.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sat, 20 Aug 2016 19:19:36 +0200

> Presumably a base64-aware compression method is not impossible?

It would be, but I would expect the difference to be slight. Assuming
that at least some of the largest mails in my office mailbox consist
mostly of an uncompressible blob (either an MS-office document
(essentially a ZIP file) or a PDF), an ideal compression would be
compress these to about 74% of their size (undoing the base64
expansion). Doing some stats on the last 30 days of mail I see some
mails which cannot be compressed to 74%, but none which are more than
76% of their original size after compression. I used gzip defaults, gzip
-9 or bzip or lzma might do slightly better. An improvement of 2%
probably doesn't warrant a non-standard compression algorithm.

        hp

-- 
   _  | Peter J. Holzer    | A coding theorist is someone who doesn't
|_|_) |                    | think Alice is crazy.
| |   | [email-masked]         | -- John Gordon
__/   | http://www.hjp.at/ |    http://downlode.org/Etext/alicebob.html

---

From: Brandon Long <[email-masked]>
Date: Tue, 23 Aug 2016 17:34:15 -0700

> the MIME standard for 20 years.

I think the Google Drive folks would be happy with that, but I'm not sure
the rest of the internet would be.

Also, there's the complication of spam/malware/virus scanning, now you rely
on the end user's computer for that instead of the service, unless the
service downloads the message/external and checks it, but that has other
problems with authorization.

Also, the actual support for message/external may be a bit lacking in most
mail clients, just a link in an html message probably works better for most
folks.

R's,
> not.  The experimental version had a bunch of fallback options that failed

Not having a fallback is probably the right choice, but it does make roll
out harder for any service which does forwarding/relaying.

OTOH, as we're pointing out, it's hardly the only smtp extension with that
issue.

Brandon

---

From: "John R Levine" <[email-masked]>
Date: 23 Aug 2016 21:17:35 -0400

> the rest of the internet would be.

Depends how big the attachments are.  I know people who would mail around 
100 meg presentations every day if there were some way to do it.

Do you have any stats on how many large attachments people use?

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 24 Aug 2016 01:21:17 -0000

>problems with authorization.

If the attachment were in the message you'd have no trouble scanning it,
so I don't see why that's a problem.

>mail clients, just a link in an html message probably works better for most

Yeah, that too.

R's,
John

---

From: Dave Crocker <[email-masked]>
Date: Wed, 24 Aug 2016 08:14:43 -0700

On 8/23/2016 5:34 PM, Brandon Long wrote:
> I think the Google Drive folks would be happy with that, but I'm not
...
> Also, there's the complication of spam/malware/virus scanning, now you
...
> Also, the actual support for message/external may be a bit lacking in
...

Just to round out the list of concerns, there are patents on that mode 
of operation, and they are very aggressively (and unfortunately 
successfully) defended.

d/

-- 

   Dave Crocker

---

From: "John Levine" <[email-masked]>
Date: 24 Aug 2016 15:58:16 -0000

>of operation, and they are very aggressively (and unfortunately 

Given that the MIME standard was written 20 years ago, I'd think that
the patents should be expiring by now.

R's,
John