Subject: [ietf-smtp] quoted-unprintable ?

From: "John Levine" <[email-masked]>
Date: 21 Mar 2021 13:41:02 -0400

Over on another list we have been musing about CHUNKING and BINARYMIME.

CHUNKING is easy to implement (I added it to my MTA in about an hour) but BINARYMIME
is extremely painful on systems where the native line ending is not \r\n because you
have to parse MIME bodies to figure out when to change line endings and when not to.

The advantage of BINARYMIME over base64 is that base64 is 33% bigger since it encodes
six bits per octet rather than 8.  It occurs to me that since everone these days
supports 8BITMIME, one could invent a quoted-unprintable encoding that encodes only
the characters that are special, CR LF NUL.  (To play it safe I'd also encode 0xff).
This gets you about a 2% size increase and stays compatible with 8BITMIME.

This seems totally obvious.  Has anyone proposed it before?

I realize that there is a severe chicken and egg problem here since you wouldn't use it
unless you knew your recipient could handle it.  I suppose one could add an EHLO keyword,
but MTAs don't downgrade on the fly any more, particularly since that means they would have
to redo DKIM and ARC signatures.

R's,
John

PS: I also realize that the sensible way to send a message with a giant attachment is
to put the attachment on a web server and use message/external-body, but that doesn't seem
all that well supported, either.

---

From: Russ Allbery <[email-masked]>
Date: Sun, 21 Mar 2021 12:29:20 -0700

"John Levine" <[email-masked]> writes:

> The advantage of BINARYMIME over base64 is that base64 is 33% bigger
> since it encodes six bits per octet rather than 8.  It occurs to me that
> since everone these days supports 8BITMIME, one could invent a
> quoted-unprintable encoding that encodes only the characters that are
> special, CR LF NUL.  (To play it safe I'd also encode 0xff).  This gets
> you about a 2% size increase and stays compatible with 8BITMIME.

> This seems totally obvious.  Has anyone proposed it before?

Yes, on Usenet, and it looks like the authors also thought of using it on
email as well.  It's called yEnc:

http://www.yenc.org/

I don't think there was ever an RFC, though.  The deployment on Usenet was
very YOLO and at least at the start openly eschewed MIME in favor of using
it similar to uuencode, which made me leery of it.  I'm not sure if that
was subsequently fixed because I lost track of further developments since
I didn't care much about binary Usenet and the author didn't participate
in standardization discussions very much.

-- 
Russ Allbery ([email-masked])             <https://www.eyrie.org/~eagle/>

---

From: "John R Levine" <[email-masked]>
Date: 21 Mar 2021 16:08:32 -0400

> I don't think there was ever an RFC, though.

I found a discussion on this very list in 2003 of gzip-8bit which was more 
or less gzip with yEnc escapes, and a draft that Ned wrote:

https://mailarchive.ietf.org/arch/browse/ietf-822/?gbt=1&index=VmGPBP83tzuzAzdKOwtckalMipE

https://datatracker.ietf.org/doc/draft-freed-mime-newenc/

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Ned Freed <[email-masked]>
Date: Sun, 21 Mar 2021 12:18:41 -0700 (PDT)

> This gets you about a 2% size increase and stays compatible with 8BITMIME.

It's actually kind of tricky if you want to avoid pathological cases. Combining
the encoding with compression is the simplest way to avoid that, since it's
unlikely in the extreme that the compression scheme will spit out a high
percentage of any specific character.

> This seems totally obvious.  Has anyone proposed it before?

At least five times that I know of. Probably a bunch more than I don't. One
scheme that's actually used on the News side of thing is yEnc:

  http://www.yenc.org/

I wrote a draft at one point for one that combined the encoding part of yEnc
with deflate:

  https://tools.ietf.org/html/draft-freed-mime-newenc-00

It never went anywhere.

> to redo DKIM and ARC signatures.

First, there are still MTAs that downgrade, just not for 8bitMIME. It's pretty
much a requirement if you're going to try and deploy smtputf8. However, as long
as the downgrade simply ignores encodings it doesn't recognize, it shouldn't
be a problem.

Second, an EHLO keyword isn't useful, since what matters is that the
*recipient*'s ability to decode. And antivirus scanners, which won't like not
being able to see into all the parts of a message.

And this is pretty much why none of these schemes ever went anywhere.

Anyway, I would be fine with reviving the draft if people either have a way around
the chicken and egg problem, or think it can be ignored.

> all that well supported, either.

Antivirus scanners don't like message/external-body either, because the body can
be changed after scanning. The few that support it do things like replace the
URL with one that redirects through a server that either does the scan every
time the object is fetched or has a checksum to make sure the object hasn't
changed. But all of this is pretty high overhead, and for reasons I don't fully
understand doesn't work all that well in practice.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Thu, 25 Mar 2021 16:32:49 +0000

Ned Freed <[email-masked]> wrote:
> > since it encodes six bits per octet rather than 8.  It occurs to me
> > that since everone these days supports 8BITMIME, one could invent a
> It's actually kind of tricky if you want to avoid pathological cases.
> Combining the encoding with compression is the simplest way to avoid
> that, since it's unlikely in the extreme that the compression scheme
> will spit out a high percentage of any specific character.

Stuart Cheshire and Mary Baker have a super elegant binary escaping scheme
called "consistent overhead byte stuffing" (COBS) that has a guaranteed
overhead of less than 1%.

http://www.stuartcheshire.org/papers/COBSforToN.pdf

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  https://dotat.at/
Fisher, German Bight, Humber, Thames, Dover, Wight, Portland:
Southwest 4 to 6, occasionally 7 later. Slight or moderate, becoming
mainly moderate, occasionally rough later except in Dover and Thames.
Showers. Mainly good.

---

From: "John R Levine" <[email-masked]>
Date: 25 Mar 2021 13:11:12 -0400

> http://www.stuartcheshire.org/papers/COBSforToN.pdf

That's quite clever, although it only avoids a single value and it's not 
immediately obvious how to extend it to multiple values.  Applying it
iteratively for different values wouldn't work.

This was his PhD thesis published over 20 years ago.  Has anyone ever used 
it in practice?

Regards,

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 25 Mar 2021 14:58:53 -0400

> iteratively for different values wouldn't work.

There's an easy generalisation to 3 forbidden code points: use 6 bits
for the chunk length, with 2 high bits for which of the 3 forbidden
sequences terminates the chunk.

    01 - NUL |
    10 - NL  | 6-bit byte count
    11 - CR  |

A non-zero value (n) of the 6-bit count indicates a run of (n-1) literal
bytes none of which are the reserved bytes NUL, LF or CR, followed by the
indicated reserved byte.  When (n == 0) the 2 high bits signal, respectively,
63, 126 or 189 literal bytes without a following reserved byte.

This encoding adds at most one byte for every 63, generating 8-bit
data that is free of NUL, CR and LF characters, which can then
be used for framing.

One can then generate output lines that are 78 bytes long, adding
another factor of 1/40, for a combined cost in the range [2.56%, 4.2%).

-- 
    Viktor.

---

From: Ned Freed <[email-masked]>
Date: Thu, 25 Mar 2021 10:47:12 -0700 (PDT)

> iteratively for different values wouldn't work.

If you really think this is a problem worth solving, the simplest way to do it
is encrypt the dats using a fast encryption scheme and a fixed key. Encryption
pretty much guarantees that there won't be too many of any particular byte
value.

For a key I suggest this hex value:

35AEAB767A46AA9B232C229E3CDFE4E97C67F4BA82A95E3ED4B97805DCC34268 

That's the SHA-256 hash of "This may be a little overengineered."

If you're paranoid about some sort of chosen plaintext attack you can
generate a random IV and put it at start of the data.

As for the encryption scheme, given the hardware assists and numerous excellent
libraries that take advantage of it, AES-256 in OFB mode is probably a good
choice. (OFB eliminates the need to do anything special at the end of the data.)
However, if support on really low end hardware is important, Speck would be a
viable alternative.

The only potential problem I see is that using encyption completely trashes your
ability to apply compression to the encoded part and get anything useful. But
given the use of compression in most media formats plus the efficiency of the
resulting encoding, it isn't clear to me that there's reason to care.

But supposing you do care, or supposing that you're one of those people who
insisted on running your CSMA network way under capacity out of fear of
collision storms, or who after generating their maximum length MIME boundary
out of 256 bits of random data obtained through observation of radiation decay,
still insist on checking for collisions, there are deterministic algorithms
that will work.

Let's first note that simple counting arguments suffice to demonstrate
that there's no possible input that can make it impossible for a combination
of swapping and stuffing to be ineffective unless the set of
octets that have to be avoided is large - certainly much larger than
the set of 4-6 we're contemplating here. 

So the goal is to find a set of swaps and a quoting character that will
deal with the pathological cases. This is easy: Pick a block size, read
in a (possibly partial) block, and then count the number of times each
octet  appears. (The obvious way to do this is with a 256 element array.)
Now scan the array for values with the lowest counts, omitting ones
that are associated with octets you can't use. You'll need one
for the quoting octet plus one for each octet you can't use.

The octet that shows up the least is your quoting octet. The others you pair up
with the octets you can't use, say by associating the least used one with the
most used unusable octet. Encoding is then done by first writing the
quoting/pairing/blocking information (format left as an exercise for the
reader), then taking each octet from the block, swapping it if it's one that
needs swapping, quoting if it's now one that needs quoting, and output the
result.

I think that does it. 

					Ned

---

From: "John R Levine" <[email-masked]>
Date: 25 Mar 2021 18:45:45 -0400

> If you really think this is a problem worth solving, the simplest way to do 
> it is encrypt the dats using a fast encryption scheme and a fixed key. 
> Encryption pretty much guarantees that there won't be too many of any particular byte
> value.

Look at the paper, which is solving a different problem.  He's dealing 
with a situation where there is one special byte (zero in his example) 
which is used to frame blocks, and this twiddles the data in a way that 
gets rid of zeros very cheaply.

Our problem is different. There's at least one value we need to avoid 
(zero), and two that we need to quote when they're in the data, and ignore 
otherwise (CR and LF).  COBS doesn't help either with quoting CR and LF 
when they are significant or ignoring them when they are not.

Regards,

---

From: Gene Hightower <[email-masked]>
Date: Thu, 25 Mar 2021 16:41:34 -0700

On 25/03/2021 15:45, John R Levine wrote:

> There's at least one value we need to avoid (zero)

NUL ('\0') is a valid ASCII character in message bodies.

---

From: Ned Freed <[email-masked]>
Date: Thu, 25 Mar 2021 16:32:25 -0700 (PDT)

> gets rid of zeros very cheaply.

I'm aware of the problem the paper solves and that it isn't the problem we need
solved.

> when they are significant or ignoring them when they are not.

I agree, which is why I proposed two approaches that both address the problem
we're actually trying to solve. More specifically, the bloat associated with
quoting is only a problem when the characters you're quoting comprise a
significant fraction of the input data. Any competent encryption scheme produces
something close to uniformly distributed output, which is about as good as
you're going to get.

The second proposal actually analyzes the input, shifts characters around
to avoid the problematic cases, and selects a good quoting character.

				Ned

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 26 Mar 2021 03:33:38 -0400

On Thu, Mar 25, 2021 at 04:32:25PM -0700, Ned Freed wrote:

> Any competent encryption scheme produces something close to uniformly
> distributed output, which is about as good as you're going to get.
> 

Nothing nearly that sophisticaed is needed, if one is willing to keep it
simple and accept ~4.2% total cost (output folded to 78 bytes + CRLF).
In particular encryption really feels much too heavyweight.  And there
is a malicious counter-example in the form of binary data that happens
to be the decryption of a run of NULs. :-)

Instead, the COBS scheme generalises easily.  The encoding can be more
efficient if we e.g. chunk the output as 998 bytes + CRLF, in which case
the worst case expansion is 1.8% for an input stream entirely devoid of
NUL, LF or CR bytes.  This is the absolute worst case, not just for
likely, but all possible inputs, and inputs where all lines are 63 bytes
or less is not expanded at all!

-- 
    Viktor.

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 26 Mar 2021 14:03:12 -0400

> 
> 
> On Mar 25, 2021, at 7:41 PM, Gene Hightower <[email-masked]> wrote:
> 

Yes, in SMTP, but sadly rejected by a non-negligible fraction IMAP servers,
so it ultimately isn't actually safe in email bodies.

-- 
	Viktor.

---

From: "John Levine" <[email-masked]>
Date: 26 Mar 2021 16:56:36 -0400

>NUL ('\0') is a valid ASCII character in message bodies.

This would be a good time to reread RFC 5322, particularly the
first paragraph of section 2.1, and and the penultimate paragraph
of section 4.

R's,
John

---

From: Gene Hightower <[email-masked]>
Date: Fri, 26 Mar 2021 15:12:21 -0700

On 26/03/2021 13:56, John Levine wrote:

> first paragraph of section 2.1,

Which says “characters with values in the range of 1 through 127” which,
just by itself, means you have to escape *most* values.

I thought you were thinking up a better transfer-encoding for 8-bit MIME
Transport. RFC-6152 lifts that particular restriction allowing “messages
containing arbitrary octet-aligned material” which I took to include the
value 0 and (if considered unsigned) the values 128…255.

Maybe RFC-6152 meant 128…255 and *not* zero by “arbitrary”—that's just
not how I read it.

Escaping zero seems like a good idea given it's traditional use as a
terminator in C code.

---

From: Gene Hightower <[email-masked]>
Date: Sun, 28 Mar 2021 22:13:58 -0700

For use in place of Quoted-Printable when SMTP Service Extension for
8-bit MIME Transport is supported.

For text content in UTF-8 or 8859 (or whatever), a version of
quoted-printable that allowed an octet with decimal value greater than
126 to be represented as itself. For data that is text, but not ASCII
text, this would be a more efferent and more readable/useful version of
quoted-printable.

This would be appropriate for the vast majority of text/plain and
text/html message parts I see in modern email streams.

This is not an answer for John Levine's quoted-unprintable project,
which I think he is envisioning as a replacement for Base64 in the 8-bit
MIME Transport case.

Until everyone does CHUNKING…

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 29 Mar 2021 03:44:48 -0400

> On Mar 29, 2021, at 1:13 AM, Gene Hightower <[email-masked]> wrote:
> 

CHUNKING is very unlikely to be supported on systems with LF line-endings.
In Postfix, Exim, ... my best estimate of the timeline for CHUNKING support
is "never":

  https://media.newyorker.com/photos/59095105ebe912338a372666/master/w_1548,c_limit/How-about-never-cartoon.jpg

-- 
	Viktor.

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 29 Mar 2021 10:39:27 +0100

> In Postfix, Exim, ... my best estimate of the timeline for CHUNKING support

Exim supports CHUNKING.
-- 
Cheers,
   Jeremy

---

From: Sam Varshavchik <[email-masked]>
Date: Mon, 29 Mar 2021 07:04:12 -0400

> In Postfix, Exim, ... my best estimate of the timeline for CHUNKING support

I don't think line ending is the driving force. CHUNKING is an extension,  
it's not mandatory. I do not see a lot of situations where CHUNKING brings  
much value-added to the equation. Even if CHUNKING is implemented it's still  
necessary to support peers that do not. So, what exactly does CHUNKING give  
me?

If a particular SMTP implementation can use CHUNKING far more efficiently to  
send or receive than without it, then, yes, full speed ahead with CHUNKING.  
But I'm hard pressed to come up with such a scenario. Dot-stuffing is not  
a herculean task.

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 29 Mar 2021 08:07:07 -0400

Sorry, OOPS, I meant BINARYMIME...

> On Mar 29, 2021, at 5:39 AM, Jeremy Harris <[email-masked]> wrote:
> 

-- 
	Viktor.

---

From: Gene Hightower <[email-masked]>
Date: Mon, 29 Mar 2021 08:56:55 -0700

> Sorry, OOPS, I meant BINARYMIME...

BINARYMIME requires the CHUNKING extension; it doesn't work with the
DATA/dot-stuffing method of transferring a message.

Quoted-Printable-8bit & quoted-unprintable are ideas that work with the
8-bit MIME Transport extension (RFC-6152).
That is, more efficient and/or otherwise more useful ways of transport
encoding for 8-bit channels.

Interestingly, 8BITMIME is almost universally adopted. BINARYMIME not so
much.

---

From: Gene Hightower <[email-masked]>
Date: Mon, 29 Mar 2021 10:57:52 -0700

Or perhaps just re-define Quoted-Printable to not escape octet values
above 126. Would this break any (many?) existing decoder implementations?

Introducing a new Content-Transfer-Encoding might be impossibly hard.

---

From: "John Levine" <[email-masked]>
Date: 29 Mar 2021 15:09:50 -0400

>encoding for 8-bit channels.

Yes, that was the idea. As Ned pointed out, people have proposed many
versions of quoted-unprintable over the years.

>Interestingly, 8BITMIME is almost universally adopted. BINARYMIME not so

It's not very surprising. 8BITMIME is trivial to implement on systems
with 8 bit bytes (all of them, this millenium), BINARYMIME very
difficult if your internal line ending is not CR-LF.

R's,
John

---

From: Sam Varshavchik <[email-masked]>
Date: Mon, 29 Mar 2021 19:43:18 -0400

> Interestingly, 8BITMIME is almost universally adopted. BINARYMIME not so

I asked myself the same question again: what is the value-added here. 
8BITMIME has clear benefits to both the sender and the recipient. The sender  
does not need to transcode 8bit to quoted-printable. The receiver can  
actually receive all PGP-signed mail. Transcoding breaks PGP signatures (I  
think this was a mistake – applying signatures to the wire content, the PGP  
signature should be calculated for the decoded content, so transcoding to a  
different MIME encoding won't break the signature, but that's water under  
the bridge).

So, what is the answer to the same question, for BINARYMIME? This is mostly  
conjecture, but I'd say that email implementations tend to fall into two  
broad categories.

The first one is where mail is stored in its original unaltered RFC 2?822  
format. Access to it is exposed by POP3, IMAP, and Webmail servers that  
read and parse it. I see no upside to sending and receiving BINARYMIME,  
here. This requires more work, and more opportunities for breakage.

The second one is where mail is transformed into an internal format that's  
specific to the mail store. Attachments get decoded and saved. Mail access  
is with custom clients. I am talking about stuff like Exchange and Outlook;  
or going back earlier in time, Lotus Notes.

Mail is likely to be processed, from on-the-wire format. Seems to be there's  
less work to send and receive BINARYMIME, here.

But, again, you can't assume BINARYMIME availability and you still have to  
implement sending and receiving mail in RFC 2?822 format. So, given that  
this needs to be implemented anyway, again the question is whether it's  
worth the effort to support two code bases that do the same thing. I suppose  
that if the implementations' particulars means that doing BINARYMIME  
requires very little additional work, it might be worth it. But, how likely  
is that…

---

From: "John Levine" <[email-masked]>
Date: 29 Mar 2021 21:00:27 -0400

>conjecture, but I'd say that email implementations tend to fall into two  

CHUNKING and BINARYMIME were in RFC 1830.

RFC 2045 says:

   Mail transport for unencoded 8bit data is defined in RFC 1652.  As of
   the initial publication of this document, there are no standardized
   Internet mail transports for which it is legitimate to include
   unencoded binary data in mail bodies.  Thus there are no
   circumstances in which the "binary" Content-Transfer-Encoding is
   actually valid in Internet mail. ...

I get the impression that BINARYMIME was a placeholder they threw in
with the hope that some other kind of framing would come along. The
advantage of BDAT over DATA for 8BITMIME is pretty modest, and can
even be negative for the implementations that send small 8K BDAT
chunks.

The fact that nothing like quoted-unprintable has ever gotten any traction
tells me that even the overhead of base64 vs 8BITMIME isn't something people
care about.

R's,
John

---

From: Gene Hightower <[email-masked]>
Date: Mon, 29 Mar 2021 18:12:47 -0700

On 29/03/2021 16:43, Sam Varshavchik wrote:
> So, what is the answer to the same question, for BINARYMIME?

It's sort of analogous, as I see it.

On 29/03/2021 16:43, Sam Varshavchik (also) wrote:
> The sender does not need to transcode 8bit to quoted-printable.

The sender does not need to encode binary as base64.

> But, again, you can't assume BINARYMIME availability and you still
> have to implement sending and receiving mail in RFC 2?822 format.

How can we assume 8-bit end-to-end?

I think the theoretical answer is the same, you can't.

I think the practical answer is that most email is transferred over
TCP these days which is a stream of 8-bit bytes. You sort of get
8-bits for free without doing anything special. You have to work at
*not* supporting 8BITMIME. So the bar for this feature was very low.

I will say even without announcing support for 8BITMIME, you still get
a bunch of 8-bit MIME messages from sources all over the Internet.
(The "postal" server does not announce 8BITMIME)

---

From: Paul Smith <[email-masked]>
Date: Tue, 30 Mar 2021 09:52:56 +0100

> *not* supporting 8BITMIME. So the bar for this feature was very low.

So, what do you do if you have an 8-bit message and try to send it to a 
server which doesn't support 8BITMIME?

This is the main thing that has stopped us implementing it.

It seems there are three options:

- fail to deliver the message (not going to be popular with users)

- deliver it anyway, and stuff the standards (will *probably* work, but 
if it doesn't, we have no excuse for our customers)

- transcode it to 7-bit (complex and breaks DKIM/DMARC and any other 
signatures)

Each of these options 'raises the bar' considerably if you provide 
support to users.

-- 
Paul
Paul Smith Computer Services
[email-masked] - 01484 855800

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: Jeremy Harris <[email-masked]>
Date: Tue, 30 Mar 2021 12:31:33 +0100

On 30/03/2021 09:52, Paul Smith wrote:
> So, what do you do if you have an 8-bit message and try to send it to a server which doesn't support 8BITMIME?

https://cr.yp.to/smtp/8bitmime.html

Summary: 8BITMIME has been irrelevant for years.
You send it.

-- 
Cheers,
   Jeremy

---

From: Hector Santos <[email-masked]>
Date: Tue, 30 Mar 2021 08:07:07 -0400

On 3/30/2021 7:31 AM, Jeremy Harris wrote:
>> So, what do you do if you have an 8-bit message and try to send it 
>> to a server which doesn't support 8BITMIME?
>

+1,   we totally assume 8 bit!!  We have nothing that needs 7 bit masking.

-- 
Hector Santos,
https://secure.santronics.com
https://twitter.com/hectorsantos

---

From: Sam Varshavchik <[email-masked]>
Date: Tue, 30 Mar 2021 08:22:15 -0400

> Summary: 8BITMIME has been irrelevant for years.

Dan Bernstein wrote that as an explanation of why he didn't implement  
8BITMIME in Qmail. But I don't think this was the real reason. It was really  
the same reason as to why Qmail also didn't implement TLS (OpenSSL was  
already in use back then), and also why Qmail used one connection per Email  
when sending mail, look up the MX, connect, send email, disconnect; and  
reconnect again, for another email to the same domain; even when the same  
message had multiple recipients in the same domain, Qmail would still use  
connection per recipient, per email, to the same domain.

It's really the same reason for all of this, as well as 8BITMIME, and that  
documentation was merely his explanation for the 8BITMIME part of that.

---

From: Paul Smith <[email-masked]>
Date: Tue, 30 Mar 2021 15:00:56 +0100

> +1,   we totally assume 8 bit!!  We have nothing that needs 7 bit 

We dont either, but RFC 6152 (written in 2011) has the text that [if the 
server does not advertise 8BITMIME support]:

    "then
    the client SMTP must not, under any circumstances, attempt to
    transfer a content that contains characters outside of the US-ASCII
    octet range (hex 00-7F)."

This seemed quite a strongly worded requirement, so we'd made the 
decision that that requirement made it too complicated to handle 
gracefully, and as it was so strongly worded, we'd better not ignore it.

But, if people are ignoring that requirement, it's actually interesting. 
Not because people aren't following the standards, but that suggests 
that 8BITMIME isn't actually that widely used...

I'd assumed that the reason we very rarely see incoming 
"content-transfer-encoding: 8bit" MIME messages is because we don't 
advertise that we support 8BITMIME so most senders were transcoding them 
as RFC 6152 requires. But, if servers are just sending them in the 8-bit 
form anyway, I'd expect to see loads of 'cte: 8bit' messages, but the 
vast majority of '8 bit' messages we receive are QP encoded with a few 
Base64 encoded. A few (single digit percentage) are 'cte: 8bit', mostly 
spam, with some transactional messages from the same few senders.

--

---

From: Ned Freed <[email-masked]>
Date: Tue, 30 Mar 2021 07:05:11 -0700 (PDT)

> with the hope that some other kind of framing would come along.

Nope. MIME was carefully designed to support both 8bit and binary format. We
briefly considered defining the necessary transport - and in hindsight not doing
so was an error - but back when RFC 1521 and 1522 came out the situation
surrounding 8BITMIME was unclear, so we didn't pursue it. Then other
things got in the way, and it only happened because the voicemail
people wanted to avoid the overhead of base64.

> The

Acutally, the advantages can be quite significant client-side, depending
on your implementation.

> tells me that even the overhead of base64 vs 8BITMIME isn't something people

On this we agree. We thought the need to avoid the overhead in voicemail
messages would be sufficient to get BINARYMIME deployed. We were wrong.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Tue, 30 Mar 2021 08:51:15 -0700

> messages would be sufficient to get BINARYMIME deployed. We were wrong.

This ought to be a lesson about assumed incentives, versus checking for 
actual incentives.  We aren't very good at learning that lesson.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: "John R Levine" <[email-masked]>
Date: 30 Mar 2021 12:00:40 -0400

> surrounding 8BITMIME was unclear, so we didn't pursue it.

For binary parts you just scan for --delimiter\r\n ?  I guess that would 
work.

>> The advantage of BDAT over DATA for 8BITMIME is pretty modest, and can

I see how you could use sendfile() to blat a message to a socket if it's 
stored with \r\n line endings.  Is it more than that?

Regards,

---

From: Ned Freed <[email-masked]>
Date: Tue, 30 Mar 2021 09:34:51 -0700 (PDT)

> For binary parts you just scan for --delimiter\r\n ?  I guess that would

Having implemented it, I can say it works very well indeed.

> stored with \r\n line endings.  Is it more than that?

That plus pipelining.

				Ned

---

From: Gene Hightower <[email-masked]>
Date: Tue, 30 Mar 2021 12:19:21 -0700

> stored with \r\n line endings.  Is it more than that?

I think TLS precludes sendfile().

---

From: Jeremy Harris <[email-masked]>
Date: Tue, 30 Mar 2021 20:47:45 +0100

> I think TLS precludes sendfile().

However, see:

   https://www.openssl.org/docs/manmaster/man3/SSL_sendfile.html

-- 
Cheers,
   Jeremy

---

From: "John Levine" <[email-masked]>
Date: 30 Mar 2021 18:09:29 -0400

It appears that Paul Smith  <[email-masked]> said:
>    "then the client SMTP must not, under any circumstances, attempt to

I think you're right.  If your user base speaks English, it is not
likely to be sending around a lot of text where 8BITMIME would help,
and binary data is all sent as base64 because as we have seen,
the 25% bloat hasn't been enough to invent something like quoted-unprintable.

For really large stuff, people use Dropbox and its competitors.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Tue, 30 Mar 2021 19:12:14 -0400

--On Tuesday, March 30, 2021 18:09 -0400 John Levine
<[email-masked]> wrote:

>> We dont either, but RFC 6152 (written in 2011) has the text
>> that [if the  server does not advertise 8BITMIME support]:
>> 
>>    "then the client SMTP must not, under any circumstances,
>>    attempt to transfer a content that contains characters
>>    outside of the US-ASCII octet range (hex 00-7F)."
>> 
>> This seemed quite a strongly worded requirement, so we'd made
>> the  decision that that requirement made it too complicated
>> to handle  gracefully, and as it was so strongly worded, we'd
>> better not ignore it.
>> 
>> But, if people are ignoring that requirement, it's actually
>> interesting.  Not because people aren't following the
>> standards, but that suggests  that 8BITMIME isn't actually
>> that widely used...
> 
> I think you're right.  If your user base speaks English, it is
> not likely to be sending around a lot of text where 8BITMIME
> would help, and binary data is all sent as base64 because as
> we have seen, the 25% bloat hasn't been enough to invent
> something like quoted-unprintable.
> 

I don't have a strong position on the 8BITMIME question itself,
but, as people think about changing or ignoring the requirement,
they should probably keep in mind that SMTPUTF8 (RFC 6530ff)
require 8BITMIME, i.e., advertising SMTPUTF8 without also
advertising 8BITMIME is invalid.  That is, of course, consistent
with the 6152 requirement quoted above because one cannot comply
with RFC 6531 and 6432 without transmitting non-ASCII header
field material and...

    john

---

From: "John R Levine" <[email-masked]>
Date: 30 Mar 2021 20:39:03 -0400

> advertising 8BITMIME is invalid. ...

I couple of years ago I did some surveys of MTAs for the UASG.  Of 56,901 
MTAs I survyed, 56,262 advertised 8BITMIME.  The only identifiable MTA not 
offering it was an obscure Windows MTA called Mailenable.

There were a few Postfix and Exim servers not advertising it (165 out of 
28,000 for Exim, 15 out of 14,739 for Postfix) which I assume were 
configuration errors.

It would be resonable to make 8BITMIME a MUST now.  Ever since the last 
PDP-10 left the net about 30 years ago, every machine on the Internet uses 
8 bit bytes which makes 8BITMIME close to free to implement.

Regards,

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 30 Mar 2021 22:09:52 -0400

> 8 bit bytes which makes 8BITMIME close to free to implement.

Not much point, unless one also at the same time deprecates 7bit
downgrade when the peer does not advertise 8BITMIME.  Otherwise, the
behaviour when it is a MUST is indistinguishable from the status quo.

Are you proposing that too?  I believe most user-friendly MUAs send
quoted-printable, so the issue is typically moot.  Though "mutt",
which I am using for this message, does send unencoded UTF-8 as an
8bit body (I have send_charset="us-ascii:utf-8:iso-8859-1").

-- 
   Виктор.

---

From: John C Klensin <[email-masked]>
Date: Tue, 30 Mar 2021 22:22:16 -0400

--On Tuesday, March 30, 2021 22:09 -0400 Viktor Dukhovni
<[email-masked]> wrote:

>> I couple of years ago I did some surveys of MTAs for the
>> UASG.  Of 56,901  MTAs I survyed, 56,262 advertised 8BITMIME.
>> The only identifiable MTA not  offering it was an obscure
>> Windows MTA called Mailenable.
>> 
>> There were a few Postfix and Exim servers not advertising it
>> (165 out of  28,000 for Exim, 15 out of 14,739 for Postfix)
>> which I assume were  configuration errors.
>> 
>> It would be resonable to make 8BITMIME a MUST now.  Ever
>> since the last  PDP-10 left the net about 30 years ago, every
>> machine on the Internet uses  8 bit bytes which makes
>> 8BITMIME close to free to implement.
> 
> Not much point, unless one also at the same time deprecates
> 7bit downgrade when the peer does not advertise 8BITMIME.
> Otherwise, the behaviour when it is a MUST is
> indistinguishable from the status quo.
> 
> Are you proposing that too?  I believe most user-friendly MUAs
> send quoted-printable, so the issue is typically moot.  Though
> "mutt", which I am using for this message, does send unencoded
> UTF-8 as an 8bit body (I have
> send_charset="us-ascii:utf-8:iso-8859-1").

Victor, 
Not speaking for John, but the point I was trying to make is
that, if anyone is offering support for non-ASCII addresses
and/or header fields, 8BITMIME (and advertising it) is already a
MUST.  And, if anyone is not offering support for non-ASCII
addresses and header fields and is trying to appeal to, or
interact with, a diverse international audience, it is probably
time to get with the program... or expect to be called upon to
explain why not (if not already, then soon).

   john

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 31 Mar 2021 10:38:49 -0400

> On Mar 30, 2021, at 10:22 PM, John C Klensin <[email-masked]> wrote:
> 

I take no issue with that.  Just observing that whether it is MUST
or not is mostly moot, unless we're also deprecating 7-bit downgrade.

$ posttls-finger dukhovni.org
posttls-finger: Connected to smtp.dukhovni.org[100.2.39.101]:25
posttls-finger: < 220 straasha.imrryr.org ESMTP Postfix
posttls-finger: > EHLO straasha.imrryr.org
posttls-finger: < 250-straasha.imrryr.org
posttls-finger: < 250-PIPELINING
posttls-finger: < 250-SIZE 157286400
posttls-finger: < 250-VRFY
posttls-finger: < 250-ETRN
posttls-finger: < 250-STARTTLS
posttls-finger: < 250-ENHANCEDSTATUSCODES
posttls-finger: < 250-8BITMIME
posttls-finger: < 250-SMTPUTF8
posttls-finger: < 250 CHUNKING
posttls-finger: > STARTTLS
posttls-finger: < 220 2.0.0 Ready to start TLS
posttls-finger: smtp.dukhovni.org[100.2.39.101]:25: Matched DANE EE certificate at depth 0: 3 1 1 DB950F0E0030900B7E5F29FB80D843268985F386D1915EE2000D527B5F369C17
posttls-finger: smtp.dukhovni.org[100.2.39.101]:25: subject_CN=mournblade.imrryr.org, issuer_CN=R3, fingerprint=7D:01:E9:7D:3F:AF:25:C2:99:51:16:93:67:B1:6E:FE:AA:23:0E:43:E3:C3:17:7A:5A:B5:2B:50:E1:63:5D:E4, pkey_fingerprint=DB:95:0F:0E:00:30:90:0B:7E:5F:29:FB:80:D8:43:26:89:85:F3:86:D1:91:5E:E2:00:0D:52:7B:5F:36:9C:17
posttls-finger: Verified TLS connection established to smtp.dukhovni.org[100.2.39.101]:25: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256
posttls-finger: > EHLO straasha.imrryr.org
posttls-finger: < 250-straasha.imrryr.org
posttls-finger: < 250-PIPELINING
posttls-finger: < 250-SIZE 157286400
posttls-finger: < 250-VRFY
posttls-finger: < 250-ETRN
posttls-finger: < 250-ENHANCEDSTATUSCODES
posttls-finger: < 250-8BITMIME
posttls-finger: < 250-SMTPUTF8
posttls-finger: < 250 CHUNKING
posttls-finger: > QUIT
posttls-finger: < 221 2.0.0 Bye

-- 
	Viktor.

---

From: Tony Finch <[email-masked]>
Date: Wed, 31 Mar 2021 17:47:46 +0100

>   https://www.openssl.org/docs/manmaster/man3/SSL_sendfile.html

And the FreeBSD / Netflix work on in-kernel TLS with sendfile() and
hardware offload:

https://papers.freebsd.org/2019/vBSDCon/baldwin_In-Kernel_TLS_Framing_and_Encryption_for_FreeBSD.files/slides.pdf

https://papers.freebsd.org/2019/EuroBSDCon/shwartsman_gallatin-Kernel_TLS_harware_offload.files/slides.pdf

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  https://dotat.at/
Shetland Isles: North backing northwest, 3 to 5. Slight or moderate.
Fair. Good.

---

From: "John Levine" <[email-masked]>
Date: 31 Mar 2021 15:40:48 -0400

>or not is mostly moot, unless we're also deprecating 7-bit downgrade.

Interesting question.  I presume there are still MTAs that can do downgrades
but how common is it in practice?

It breaks DKIM signatures unless the MTA can find the signing keys and resign
on the fly which seems a bit much.

R's,
John

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 31 Mar 2021 16:53:17 -0400

> On Mar 31, 2021, at 3:40 PM, John Levine <[email-masked]> wrote:
> 

Postfix will do 7bit downgrade by default if the content is designated
or detected as 8bit and the remote MTA does not advertise 8BITIME support:

  http://www.postfix.org/postconf.5.html#disable_mime_output_conversion

> on the fly which seems a bit much.

For outbound mail, where the MTA/MSA is also the one doing the signing, the
safest thing is to just downgrade all messages before signing, so that
there's never any post-signature dynamic downgrade during delivery.  This
can be done by passing through an internal SMTP content filter that does
not offer 8BITMIME.  I expect most administrators don't do this.  In which
case any submitted 8BITMIME message may get downgraded after signing.

Since most MUAs do quoted-printable defensively, actual downgrading at
the MSA is rare.

One can also of course configure "disable_mime_output_conversion = yes",
but I expect most administrators don't.

I don't do DKIM, so I have personal experience to report.

-- 
	Viktor.

---

From: Ned Freed <[email-masked]>
Date: Wed, 31 Mar 2021 20:05:05 -0700 (PDT)

>   http://www.postfix.org/postconf.5.html#disable_mime_output_conversion

So will Oracle's Messsaging Server. Whether or not this matters given the
widespread deployment of 8BITMIME is another matter. I checked my own logs just
now and in the past 3 years I couldn't find a single case where a downgrade
actually occurred.

> case any submitted 8BITMIME message may get downgraded after signing.

This is trivial to do in our MTA - a single option setting, as a matter of fact.
I don't set it because I want to see if there's any downgrading to be done,
but we recommend that sites doing DKIM signing consider using it.

> Since most MUAs do quoted-printable defensively, actual downgrading at

Not IME. I get quite a few unencoded text/html parts.

I also see a lot of 8bit text (both plain and html), lack of non-English usage
notwithstanding. It's mostly fancy quotes, ellipses, and similar punctuation
characters, although use of emoji is steadily increasing.

And as I pointed out >30 years ago during the MIME work, there are plenty of
English words that employ diacritics. In fact in some cases diacritics are added
to imported words that originally didn't have them. 

  https://en.wikipedia.org/wiki/English_terms_with_diacritical_marks

				Ned

P.S. I have to say I find the fancy quotes quite annoying, especially when some
client decides to use them in sample code. They may look better, for some value
of "better", but compilers and validators don't much care for them.

Inline Emoji are also an issue for me because of font size - Unicode display
rules for inlined Emoji produce intricate little blobs my crappy eyes
are incapable of deciphering.