Subject: Re: [ietf-smtp] Compressing SMTP streams

From: Paul Smith <[email-masked]>
Date: Mon, 1 Feb 2016 10:53:21 +0000

On 31/01/2016 02:48, John R Levine wrote:
>> What purpose is served by using BASE64 beyond making the content able 
>> to travel through a 7-bit-only hop? Since most (all?) hops are now 
>> 8-bit ones, why not just send the actual 8-bit version of the data 
>> instead of converting it to the 7-bit BASE64 format?
>
> The answer to this question is quite complex.  The 8BITMIME extension 
> was defined in 1994, and nearly all current MTAs can handle it.  But 
> all it says is that the MTA can handle 8 bit characters.  It doesn't 
> affect the rule that the message must consist of lines no more than 
> 1000 characters long with \r\n at the end of each.  As far as I can 
> tell, the main effect is that people can send ISO-8859-x and UTF-8 
> without encoding, which is useful but generally not a big deal.
>
> BINARYMIME was defined in 2000 to avoid that issue, and invents a new 
> BDAT command that uses a byte count rather than escape sequence, so 
> the message body can be an arbitrary sequence of octets.  Gmail, 
> hotmail and icloud/me.com support it, Yahoo and AOL don't, but I've 
> never seen client software that would take advantage of it.
>
> What we've never seen is a quoted-unprintable encoding, which is like 
> QP but intended for binary data.  It could be like QP without soft 
> line breaks and \r\n pairs are ignored.  It's an obvious idea so there 
> must be some reason.  Dave Crocker or John Klensin or Ned Freed would 
> know. 

The problem with any of these is what to do if YOU accept 8 bit 
characters, but you have to send the message to a mail server which 
doesn't say it does. Some just pass it on and hope (which is against the 
rules AFAICS - e.g. we regularly receive messages with NULL characters 
in), other than that you can recode the message (which risks breaking 
DKIM etc) or reject the message (which risks upsetting someone).

That's why we don't support these extensions.  If every server supported 
BINARYMIME it wouldn't be a problem, but the transition period is nasty.

A compression scheme shouldn't have the same problem.

---

From: Paul Smith <[email-masked]>
Date: Mon, 1 Feb 2016 10:53:27 +0000

---

From: John C Klensin <[email-masked]>
Date: Mon, 01 Feb 2016 08:20:24 -0500

>...
>> The 8BITMIME
>> extension  was defined in 1994, and nearly all current MTAs
>> can handle it.  But  all it says is that the MTA can handle 8
>> bit characters.  It doesn't  affect the rule that the message
>> must consist of lines no more than  1000 characters long with
>> \r\n at the end of each.

It also says that the MTA will do something sensible --downgrade
and encode or reject-- if some next-hop system will not accept
the extension.   As noted elsewhere, a lot of systems ignore
that requirement and just adopt what we used to call "just send
8", but that really isn't a good idea, if only because, today, a
supposedly 8-bit-clean system that doesn't bother to advertise
8BITMIME probably suffers from other symptoms of an ancient or
lazy implementation.

>> As far as I can  tell, the main
>> effect is that people can send ISO-8859-x and UTF-8 without
>> encoding, which is useful but generally not a big deal.

It certainly improves readability of the raw stream.  I agree it
is generally not a big deal for transmission message size, but
beware of falling into the same reasoning that brought us Q-P:
while it improves readability for Latin script with some
decorated characters, it tends to make an even bigger mess of
messages in which substantially every code point is outside the
ASCII range.

>> BINARYMIME was defined in 2000 to avoid that issue, and
>> invents a new  BDAT command that uses a byte count rather
>> than escape sequence, so  the message body can be an
>> arbitrary sequence of octets.  Gmail,  hotmail and
>> icloud/me.com support it, Yahoo and AOL don't, but I've 

IIR, much of what motivated BINARYMIME at the time was concern
about non-text messages in which concepts like "line" were
meaningless and people tampering with characters that might
denote line-end (in other contexts) could cause a mess.   Yes,
it also allows arbitrary-length lines in text messages, but the
advantages of that are rarely worth the trouble and risks (see
below).

>> What we've never seen is a quoted-unprintable encoding, which
>> is like  QP but intended for binary data.  It could be like
>> QP without soft  line breaks and \r\n pairs are ignored.
>> It's an obvious idea so there  must be some reason.  Dave
>> Crocker or John Klensin or Ned Freed would  know. 

Obvious or not, I don't remember it coming up.  Remembering that
Q-P has very poor readability and encoding density properties
for anything but mostly-ASCII text with a fairly small number of
decorated Latin characters and is _far_ worse than Base64 for
non-text data in which the high-order bit is set in many or most
of the octets, the pain of having to deal with yet another
encoding almost certainly exceeds the small advantages.

On the other hand, if one knows the message is entirely ASCII
and can transmit "binary", there is another compression method
that is fast, cheap, and yields about a 12% improvement with no
dictionary overhead.  That is, of course, to toss the first bit
of every octet, transmitting only the seven payload bits.   I
have vague memories of that being done by some systems
(especially those that natively encoded ASCII as five seven-bit
characters in a 36 bit word) during the mail-over-FTP period,
but don't recall seeing any use with SMTP.  Again, a pain with
not very much gain.

> The problem with any of these is what to do if YOU accept 8
> bit characters, but you have to send the message to a mail
> server which doesn't say it does. Some just pass it on and
> hope (which is against the rules AFAICS - e.g. we regularly
> receive messages with NULL characters in), other than that you
> can recode the message (which risks breaking DKIM etc) or
> reject the message (which risks upsetting someone).

And, today rather than in the 90s, any rejection risks getting
tangled up with the backscatter or other spam issues and being
discarded, thereby having the message neither delivered nor the
sender notified of the problem.  
 
> That's why we don't support these extensions.  If every server
> supported BINARYMIME it wouldn't be a problem, but the
> transition period is nasty.

And that is the problem with every SMTP extension that causes a
failure if every subsequent MTA doesn't accept it.  To the
extent that message validity and integrity methods (DKIM and
friends included) take recoding, translation, etc., off the
table in order to solve other problems, we've made the problem
even more difficult.  The transition periods are nasty and
probably only worth it for features that either bring huge
advantages (as others have pointed out, in "normal" SMTP
environments, a bit more compression doesn't) or that are
visible and important to end users (many people believe
non-ASCII addresses fall into that category, but the deployment
rate is not exciting and their importance is still an unproven
hypothesis).

best,
     john

---

From: "John Levine" <[email-masked]>
Date: 1 Feb 2016 16:39:36 -0000

>Which do you think makes more sense for encapsulating John's CDAT data, 
>something that looks more like BDAT (length indicator), or something 
>that looks more like yEnc ("quoted unprintable")?

Counting, definitely.  Look at zlib and try and figure out how you
would switch back to processing uncompressed data after you saw
the in-line end marker.

R's,
John

---

From: Steve Atkins <[email-masked]>
Date: Mon, 1 Feb 2016 09:03:30 -0800

> 
> On Feb 1, 2016, at 2:53 AM, Paul Smith <[email-masked]> wrote:
> 
>>> What purpose is served by using BASE64 beyond making the content able to travel through a 7-bit-only hop? Since most (all?) hops are now 8-bit ones, why not just send the actual 8-bit version of the data instead of converting it to the 7-bit BASE64 format?
>> 
>> The answer to this question is quite complex.  The 8BITMIME extension was defined in 1994, and nearly all current MTAs can handle it.  But all it says is that the MTA can handle 8 bit characters.  It doesn't affect the rule that the message must consist of lines no more than 1000 characters long with \r\n at the end of each.  As far as I can tell, the main effect is that people can send ISO-8859-x and UTF-8 without encoding, which is useful but generally not a big deal.
>> 
>> BINARYMIME was defined in 2000 to avoid that issue, and invents a new BDAT command that uses a byte count rather than escape sequence, so the message body can be an arbitrary sequence of octets.  Gmail, hotmail and icloud/me.com support it, Yahoo and AOL don't, but I've never seen client software that would take advantage of it.
>> 
>> What we've never seen is a quoted-unprintable encoding, which is like QP but intended for binary data.  It could be like QP without soft line breaks and \r\n pairs are ignored.  It's an obvious idea so there must be some reason.  Dave Crocker or John Klensin or Ned Freed would know. 
> 
> The problem with any of these is what to do if YOU accept 8 bit characters, but you have to send the message to a mail server which doesn't say it does. Some just pass it on and hope (which is against the rules AFAICS - e.g. we regularly receive messages with NULL characters in), other than that you can recode the message (which risks breaking DKIM etc) or reject the message (which risks upsetting someone).

Or silently discard it, which was my experience with Exchange forwarding 8 bit messages to a server which while it was 8 bit clean didn’t advertise the fact a few years ago.

There are a lot of CRM systems behind firewalls in a lot of companies that have Exchange as a primary MX, and some of those accept inbound tickets over SMTP.

> 
> That's why we don't support these extensions.  If every server supported BINARYMIME it wouldn't be a problem, but the transition period is nasty.

Very nasty. And it can easily last (checks RFC 1869) 20 years or more.

> A compression scheme shouldn't have the same problem.

Yes. I’m not a terribly relevant ESMTP implementor, and I’m likely to implement whatever a customer wants me to implement rather than anything driven by protocol elegance … but stream level compression or a counted-byte BDAT + compression both seem like something I’d be comfortable adding to the state machine.

Cheers,
  Steve

---

From: Ned Freed <[email-masked]>
Date: Mon, 01 Feb 2016 11:16:42 -0800 (PST)

> BINARYMIME it wouldn't be a problem, but the transition period is nasty.

This argument works in the case of BINARYMIME for SMTP, but fails in every
other case. There just isn't enough support for BINARYMIME in this situation
to matter, so there's no compelling reason for you to add support for it.

In the case of 8BITMIME, however, suppport for the extension is sufficiently
widespread that it's often seen on SMTP relay operations for 8bit messages,
which IME are the majority these days. So the result of your not providing the
extension is that you force other compliant agents to either transcode, reject,
or violate the standards when relaying to you, with all that implies.

In other words, you may be making your life a little (very little given how
easy it is to properly support 8bitMIME) easier, but you're making your user's
lives harder, because most of those messages would never needed to be
transcoded.

The case for SUBMIT isn't as compelling, but it's there: First, Since message
modification is explicitly allowed in the SUBMIT context, claiming that you
shouldn't be doing that as an excuse doesn't fly.

Second, for 8bitMIME you'll be forcing the client to encode, which given the
widespread support for 8bitMIME (3/4 of MAGY, most US ISPs, etc.) means you're
exploring the lesser used code paths in clients, which is rarely a good idea.

As for BINARYMIME, the case against that is that unless your software is binary
clean to begin with it can be a royal PITA to add support. That plus the
fact that few clients have support for it is a fairly compelling argument not
to bother, but it's not the argument you gave.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Mon, 01 Feb 2016 22:01:35 -0800 (PST)

> >
> >> It's basically a minimal encoding to deal only
> >> with the flaws found in actual NNTP implementations (such as not coping
> >> with nuls and unmatched CR or LF characters) and dot-stuffing, and is
> >> pretty close to not compressing.

> that looks more like yEnc ("quoted unprintable")?

Definitely something like CDAT. Although the choice between a CDAT command
and session-level compression isn't clear to me.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 2 Feb 2016 09:24:53 +0000

> and session-level compression isn't clear to me.

Session-level compression will get ~99% compression for the 2nd to nth 
message in a bulk mailing. Apart from that CDAT and stream compression 
should be the same.

Arnt

---

From: "John Levine" <[email-masked]>
Date: 2 Feb 2016 16:47:51 -0000

>and session-level compression isn't clear to me.

The choices aren't hugely compelling, but I like CDAT for two reasons:

* avoids compressing AUTH info and the CRIME issue

* for those of us with MTAs that do a process per connection ( that's
plenty fast if your MTA isn't too big) it avoids fragile three-way
coroutine code among the MTA, TLS, and the compressor

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 2 Feb 2016 16:52:13 -0000

>message in a bulk mailing. Apart from that CDAT and stream compression 

CDAT could get the same compression if you don't reset the compressor
between messages in the session.  With deflate that's trivial, just
skip the call to the initialization routine after the first message.

Also, how many MTAs reuse sessions these days?  I have no idea, our
friends at Yahoo and Goog probably have useful numbers.

R's,
John