Subject: [ietf-smtp] [Proposal] 250-MARKDOWN

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 12:29:43 +0530

Hello Everyone,

I'm the guy who proposed SMTP Over TLS on Port 26. Looks like that was a
dead end. So, now coming with another proposal.

I joined IETF day before yesterday. So not sure if this was already
discussed before and got rejected. If this was rejected already, then my
apologies.

I wanna bring Markdown support to SMTP.  So instead of transferring both
"text/plain" and "text/html" versions, future mails can use only
"text/markdown".

I listed the benefits in my proposal. Let me if it is worth going forward.
So I can create an Internet Draft.

https://gist.github.com/mistergiri/603766f5295802d44cc1a5752ebfa21f

Thanks

-- 
Best Regards,

Viruthagiri Thirumavalavan
Dombox, Inc.

---

From: [email-masked]
Date: Wed, 09 Jan 2019 02:33:49 -0500

> "text/plain" and "text/html" versions, future mails can use only

Well, right off the top, there's the end to end problem.

Say you support text/markdown, and your mail server accepts it to deliver. What
happens if my mail server doesn't support markdown?  Do you send a text/markdown
and hope my MUA can deal with it? If you do that, why bother with an SMTP intervention?

Also, see RFC1341, and text/richtext. Ask yourself why you never see e-mails with it.

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 13:10:34 +0530

>
> What happens if my mail server doesn't support markdown?

That's why I'm proposing the MARKDOWN extension. MARKDOWN command should be
presented in the EHLO response only if your server supports markdown.

If your server supports markdown, the mail will be transferred as
text/markdown. Else mail will be transferred as "text/plain" and
"text/html" as usual.

I hope I understood you correctly.

-- 
Best Regards,

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 13:12:20 +0530

I think you are saying, problem will arise when there are multiple hops.
That makes sense.

On Wed, Jan 9, 2019 at 1:10 PM Viruthagiri Thirumavalavan <[email-masked]>
wrote:

> That's why I'm proposing the MARKDOWN extension. MARKDOWN command should
> be presented in the EHLO response only if your server supports markdown.
>

-- 
Best Regards,

---

From: Gilles Chehade <[email-masked]>
Date: Wed, 09 Jan 2019 08:44:26 +0100

Hello,

After reading your proposal, I don't quite understand what it is about and what is expected from SMTP implementations.

The content type is part of the message, not part of the protocol, also your rationale seems to equate one SMTP session to one message but a single session may transport multiple messages with different messages with different content types.

Gilles

Le 9 janv. 2019 07:59, Viruthagiri Thirumavalavan <[email-masked]> a écrit :

Hello Everyone,

I'm the guy who proposed SMTP Over TLS on Port 26. Looks like that was a dead end. So, now coming with another proposal.

I joined IETF day before yesterday. So not sure if this was already discussed before and got rejected. If this was rejected already, then my apologies.

I wanna bring Markdown support to SMTP.  So instead of transferring both "text/plain" and "text/html" versions, future mails can use only "text/markdown".

I listed the benefits in my proposal. Let me if it is worth going forward. So I can create an Internet Draft.

Thanks

--

Best Regards,

Dombox, Inc.

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 13:25:43 +0530

>
> After reading your proposal, I don't quite understand what it is about and
> what is expected from SMTP implementations.

I wanted to introduce the keyword "MARKDOWN" in the list of EHLO response.
So the client can understand and transfer the messages directly with
content type "text/markdown"

The content type is part of the message, not part of the protocol

I'm using the content type in the message right?

also your rationale seems to equate one SMTP session to one message but a
> single session may transport multiple messages with different messages with
> different content types.

Ok what am I missing here? When a server says that "I recognize markdown
format", then the client gonna send markdown messages as it is. I
understand a session can transport multiple messages. But why do you think
it won't work?

In my proposal, I provided content type in the message header to simplify
the example.

On Wed, Jan 9, 2019 at 1:14 PM Gilles Chehade <[email-masked]> wrote:

> Hello,
>
> After reading your proposal, I don't quite understand what it is about and
> what is expected from SMTP implementations.
>
> The content type is part of the message, not part of the protocol, also
> your rationale seems to equate one SMTP session to one message but a single
> session may transport multiple messages with different messages with
> different content types.
>
> Gilles
>
>
>

-- 
Best Regards,

---

From: Gilles Chehade <[email-masked]>
Date: Wed, 9 Jan 2019 09:20:34 +0100

> content type "text/markdown"

but what is preventing the client from doing so today ?

why does the server need to tell the client that it can send in text/markdown,
when the client can already do so and the server doesn't actually care for the
content type since it's part of the DATA and not the transport ?

> I'm using the content type in the message right?

yes, which is why I don't understand why it also needs to be part of protocol.

put it differently, what do you expect the SMTP implementation to do when this
MARKDOWN extension is used that it doesn't do when it isn't used ?

> understand a session can transport multiple messages. But why do you think

well, as someone writing a server implementation:

      When a server says that "I recognize markdown format"

that sentence doesn't make sense to me because the server does not recognize a
format or another, it's in charge of the transport, not of rendering or making
any sense out of the DATA part. it feels like a violation of layer, DATA spill
over transport.

Now as to why it wouldn't work, well, assuming the transport protocol actually
had to know what's inside DATA, if you use the MARKDOWN extension then how can
the same session transport another message that's not a markdown message ? The
session is "tainted" as MARKDOWN session when you use that extension... so why
would I bother implementing it if DATA already contains the content-type ?

How comes MARKDOWN is needed and HTML wasn't given that they are basically the
same use-case with a different encoding format ?

Not to mention that many implementations establish SMTP sessions based on what
is on the envelope (FROM/TO) and not what's in DATA, so sessions would require
that implementations look at DATA to determine what it contains before setting
up the SMTP session.

-- 
Gilles Chehade						       @poolpOrg

https://www.poolp.org                 tip me: https://paypal.me/poolpOrg

---

From: [email-masked]
Date: Wed, 09 Jan 2019 03:30:14 -0500

> understand a session can transport multiple messages. But why do you think

OK. 129 people at 37 different .coms that are all hosted on Office365 or
whatever Microsoft is calling that product now send e-mails to 217 people at 94
different .coms that are hosted by Google.

Microsoft bundled all 217 mails into one stream and connects to Google. Google
says "I do markdown".  Microsoft squishes 217 emails from a multipart/
alternative that contains dual text/(plain,html) down to text/markdown,
breaking any PGP or S/MIME signatures on the multipart/alternative that just
got squished into one body part.

Google now has 217 e-mails to deliver to users - and of the 94 domains, 37 of
them use a corporate email product that doesn't understand text/markdown and
they really wanted a multipart with plain,html.  Another 12 of them are
complaining that Google broke the S/MIME on an e-mail that *needed* to be
digitally signed because it was confirmation of a purchase that involved real
money (nevermind that it was Microsoft that did the squishing, the users always
complain to their own vendor, not the guilty party.  And even though the
complaining users *do* understand e-mail enough to say "I can't act on a
purchase confirmation that's not digitally signed and has a big red X on it
because the signature is bad", they don't understand that multiple servers
handled it along the way.

What does the Google server do now?

And as others will confirm, I'm not making this sort of issue up - these sorts
of problems crop up all the time.  And no matter how clear you thought you
wrote the RFC, you'll find some vendor who had a crack-addicted chimpanzee
writing code that manages to Get It Totally Wrong.

Consider this snippet from my .procmailrc that I had to put in to be able to
open e-mails from IBM:

# Lotus Notes is a boil on the buttocks of e-mail.,...
:0 hwf
*^X-Mailer:.*(IBM|Lotus) Notes
| sed -e 's/boundary="=_alternative  /boundary="=_alternative /' -e 's/boundary="=_mixed  /boundary="=_mixed /' -e 's/boundary="=_related  /boundary="=_related /'

Not only did they put blanks in the boundary separator for multipart, but they
had *two* blanks in the declaration in the RFC822 message headers, but only one
blank in the actual separators between body parts in the mail body.

And this was a problem for me because the emails that wouldn't open were from
an IBM support team working on a Sev 1 (system is totally down) issue with a
server of mine on a $1.2M project.

I never understood how that product managed to interoperate with the rest of
the world with that bug in it.

---

From: Gilles Chehade <[email-masked]>
Date: Wed, 9 Jan 2019 09:38:07 +0100

> I hope I understood you correctly.

still unsure I understand how it works, sorry

my understanding now is that the server translates DATA between markdown
and other content types based on MARKDOWN extension at peer.

did i understand correctly ?

> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

-- 
Gilles Chehade						       @poolpOrg

---

From: Peter Bowyer <[email-masked]>
Date: Wed, 9 Jan 2019 08:41:47 +0000

On Wed, 9 Jan 2019 at 07:40, Viruthagiri Thirumavalavan <[email-masked]> wrote:

> That's why I'm proposing the MARKDOWN extension. MARKDOWN command should be presented in the EHLO response only if your server supports markdown.

What piece of the *server* has to support markdown? The content of the
message isn't the server's business. Nor is knowing what content
type(s) are acceptable to each final destination for each recipient of
each message in a session. That's a matter for the originating MUA and
the receiving MUA to worry about. The server's job is to move the
(mostly) opaque content from one to the other.

PB

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 14:11:35 +0530

You are right Mr. Kletnieks.

My proposal would break with stuffs like DKIM, PGP etc

Thanks for taking your time and explaining all those stuffs.

On Wed, Jan 9, 2019 at 2:00 PM <[email-masked]> wrote:

> | sed -e 's/boundary="=_alternative  /boundary="=_alternative /' -e
> 's/boundary="=_mixed  /boundary="=_mixed /' -e 's/boundary="=_related
> /boundary="=_related /'
>

-- 
Best Regards,

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 14:12:39 +0530

Please take a look here.
https://gist.github.com/mistergiri/603766f5295802d44cc1a5752ebfa21f#steps-need-to-be-taken

But it is a flawed proposal.

On Wed, Jan 9, 2019 at 2:08 PM Gilles Chehade <[email-masked]> wrote:

> be
> > > Say you support text/markdown, and your mail server accepts it to
> deliver.
> > > What
> > > and hope my MUA can deal with it? If you do that, why bother with an
> SMTP
> > > intervention?
> > >

-- 
Best Regards,

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 14:21:53 +0530

You are right. I was worried that receiving server would not recognize
"text/markdown" as a media type since it was registered recently in the
RFC. So if a server started to recognize that as media type, I thought they
should do that via SMTP extension.

But yes, as Gilles Chehade said, it's related to DATA and shouldn't be
there. Even if that's possible, as Mr. Kletnieks said it's gonna break
digital signatures.

Hence, my proposal would not work.

-- 
Best Regards,

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 14:25:06 +0530

Thanks Guys for your time.

I'll try to come up with new proposals when I can.

On Wed, Jan 9, 2019 at 2:21 PM Viruthagiri Thirumavalavan <[email-masked]>
wrote:

>> > That's why I'm proposing the MARKDOWN extension. MARKDOWN command
>> should be presented in the EHLO response only if your server supports
>> markdown.
>>

-- 
Best Regards,

---

From: Gilles Chehade <[email-masked]>
Date: Wed, 9 Jan 2019 09:58:11 +0100

> But it is a flawed proposal.

I read it, but the following doesn't make sense to me:

	Sender should transfer the mail in "text/markdown" if the receiver support that format (250-MARKDOWN).
	Else they can go for multipart/alternative with "text/plain" and "text/html" format as usual.

What's the original content type for the mail here ?

There are multiple ways to understand this depending on what is the type
of the mail to start with. In one case, it might imply that servers need
to convert text/* to text/markdown based on a peer extension, in another
it implies that a text/markdown content should be converted to plain and
html parts, etc...

In cases were no transforms are required in either way, then I don't see
how a session with or without MARKDOWN would differ.

That being said, and as mentioned in another mail, SMTP is for transport
and I think having content-related informations spill in the protocol is
a violation of layers.

-- 
Gilles Chehade						       @poolpOrg

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 14:39:04 +0530

I'll explain off-list.

On Wed, Jan 9, 2019 at 2:28 PM Gilles Chehade <[email-masked]> wrote:

> should
> > > > > > I wanna bring Markdown support to SMTP.  So instead of
> transferring
> > > both
> > > > > Also, see RFC1341, and text/richtext. Ask yourself why you never
> see
> > > > > e-mails with it.
> > > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > >
> > > Gilles Chehade
>  @poolpOrg

-- 
Best Regards,

---

From: [email-masked]
Date: Wed, 09 Jan 2019 04:52:34 -0500

> any sense out of the DATA part. it feels like a violation of layer, DATA spill

And you're totally correct - it *is* a layering violation.  For ages, there's
been a *very* strict separation between the SMTP transport layer and the
RFC822/MIME in the DATA portion. The only extension listed in the below EHLO
that isn't *strictly* SMTP is '8bitmime' - and *that* only gets included
because there is a documented non-lossy and invertable way to convert from
'Content-Transfer-Encoding: 8bit' down to 7bit if a non-8bit-clean path is
encountered.

Consider that the following SMTP transaction is actually *legal* and accepted
by Sendmail:

[~] telnet ::1 25
Trying ::1...
Connected to ::1.
Escape character is '^]'.
220 turing-police.cc.vt.edu ESMTP Sendmail 8.15.2/8.15.2; Wed, 9 Jan 2019 04:26:52 -0500
EHLO localhost
250-turing-police.cc.vt.edu Hello turing-police.cc.vt.edu [IPv6:0:0:0:0:0:0:0:1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5
250-DELIVERBY
250 HELP
MAIL FROM:<>
250 2.1.0 <>... Sender ok
RCPT TO:<valdis@localhost>
250 2.1.5 <valdis@localhost>... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself
There are exactly zero RFC822 headers in this DATA package.
.
250 2.0.0 x099QqgX005006 Message accepted for delivery
QUIT
221 2.0.0 turing-police.cc.vt.edu closing connection

And Sendmail Does The Right Thing - takes the whole thing, even lacking rfc822
headers (which it doesn't need for delivery because the RCPT TO told it
everything it needed to know).  It fixes up Date: and Message-id: (possibly
debatable for port 25), adds the required Received: and Return-Path: headers,
and hands the whole thing to procmail, which stashes it into an nmh file in my
inbox folder.

[~] scan +inbox last:3
 65122  *    M 12:54 +00     19k IBM Announcement   IBM United States Announcement Letters
 65131  *   *  18:15 +01      4k Matthias Waehlisc  Re: BGP Experiment <<It's well-known: A fork of Quagga. Cheers matth
 65141 U     ? 04:26 -05      0k To:                 <<There are exactly zero RFC822 headers in this DATA package. 
[~] cat Mail/inbox/65141
Date: Wed, 9 Jan 2019 04:26:52 -0500

There are exactly zero RFC822 headers in this DATA package.

---

From: "John Levine" <[email-masked]>
Date: 9 Jan 2019 09:35:28 -0500

In article <[email-masked]> you write:
>-=-=-=-=-=-

My mail server doesn't support or understand any media types at all.  MIME and
media types are all handled in the MUA.

Please review RFC 5598, the Internet Mail Architecture.  The mail
server that advertises EHLO keywords is an MTA.  MIME types are
handled in the MUA.

Also, practically, every modern MUA handles HTML now.  If you want to use
markdown, adjust your MUA so it translates your markdown into text/html
and sends that.  There are dozens of markdown->html translators so this
is just splicing together code that already exists.

R's,
John

---

From: Viruthagiri Thirumavalavan <[email-masked]>
Date: Wed, 9 Jan 2019 20:24:19 +0530

Ok Thanks John.

I'll take a look.

On Wed, Jan 9, 2019 at 8:05 PM John Levine <[email-masked]> wrote:

> In article <
> [email-masked]> you
> write:
> >-=-=-=-=-=-
> >

-- 
Best Regards,

---

From: Brandon Long <[email-masked]>
Date: Wed, 9 Jan 2019 14:05:49 -0800

I think the distinction isn't as strict as John puts it, though that
doesn't mean the proposal is correct either.

Today's MTAs often have to do a lot of message body handling for
spam/av/malware/phishing, especially if they want to be able to reject bad
messages at SMTP time to prevent backscatter.

I'd also point out that there are multiple SMTP extensions for negotiating
the format of the SMTP data (SMTPUTF8, 8BITMIME, BINARY), though those
typically apply to the full format of the data portion and not just
specific parts of it.

That said, late transformations or transformations in hops have been
offered in the past, and now interact poorly with the digital signatures
present in DKIM/ARC/SMIME, and so should be avoided.  Downgrading also
failed in UTF8SMTP, as another example.

Frankly, I don't see text/markdown replacing text/plain or text/html as the
primary message content any time soon, it doesn't really solve a problem
people have.  Obviously there should be a mime type for it, but most MUAs
provide a WYSIWYG interface based on HTML these days, and have for years,
they're not chomping at the bit to hand craft formatted responses.

Brandon

---

From: Gene Hightower <[email-masked]>
Date: Wed, 9 Jan 2019 15:08:43 -0800

On 09/01/2019 06.35, John Levine wrote:

> My mail server doesn't support or understand any media types at all.
> MIME and media types are all handled in the MUA.

I agree, except for BINARYMIME: the one MIME type advertised in the
greeting stage of ESMTP.

Does your MTA use a mail store based on mbox files?  Many still do;
that means no BINARYMIME.  Do you only support DATA for sending mail
bodies (no BDAT); that also means no BINARYMIME.

Can you imagine the bandwidth saved (and carbon emissions, storage,
etc.) if we had universal support for BINARYMIME?  No more encoding
photos and videos as base64 text.

As for text/markdown as a MIME type: it would be nice to have
something lighter than HTML for simple formatted text.  But we already
have text/enriched defined in RFC 1896 and that seems to have zero
adoption.

HTML as a “standard” is a dog's dinner.

---

From: John C Klensin <[email-masked]>
Date: Wed, 09 Jan 2019 18:26:58 -0500

--On Wednesday, January 9, 2019 15:08 -0800 Gene Hightower
<[email-masked]> wrote:

>...

>> My mail server doesn't support or understand any media types
>> at all. MIME and media types are all handled in the MUA.
> 
> I agree, except for BINARYMIME: the one MIME type advertised
> in the greeting stage of ESMTP.

Even that is dubious.  What BINARYMIME does an SMTP extension is
to enable the use of those media types; it does not require or
specify them.

>...
> something lighter than HTML for simple formatted text.  But we
> already have text/enriched defined in RFC 1896 and that seems
> to have zero adoption.

Actually, fairly heavily adopted in the early days, just largely
irrelevant, as a little markup but not much, since text/html was
specified and took over

---

From: Ned Freed <[email-masked]>
Date: Thu, 10 Jan 2019 08:03:09 -0800 (PST)

> I agree, except for BINARYMIME: the one MIME type advertised in the

Binary is a content transfer encoding, not a media type.

> that means no BINARYMIME.

Not true - transcoding is always a possibility, and transcoding requires
no understanding of specific media types.

This was designed into MIME from the start.

> Do you only support DATA for sending mail

Again, transcoding address this.

> photos and videos as base64 text.

Transcoding can also be used to upconvert to binary. These days you have to
stay away from transcoding after submission and before final delivery in order
not to break signatures, but nothing prevents it from being used on, say,
submission.

But the availability of the technology doesn't mean people will use it. The
asessment of most seems to be that the benefits don't outweigh the costs.

> HTML as a “standard” is a dog's dinner.

Even if it were true, it doesn't make it any less popular.

				Ned

---

From: Gene Hightower <[email-masked]>
Date: Thu, 10 Jan 2019 09:43:03 -0800

> Binary is a content transfer encoding, not a media type.

Yes, correction: transfer encoding, not media type.

Let me try to restate my point using a few more words.

The area where MTA (the “mail server” from above) software needs to
announce support for a protocol feature to enable features at the
level of the message body is CHUNKING and BINARYMIME, RFC 3030.

If many of the leading figures in the email world can engage in a
lively discussion about a proposed MARKDOWN extension to SMTP, I
thought I'd try to raise some interest in BINARYMIME.

On 10/01/2019 08.03, Ned Freed wrote:

> But the availability of the technology doesn't mean people will use
> it. The asessment of most seems to be that the benefits don't
> outweigh the costs.

First step is CHUNKING; small benefit, modest cost.  Involves MTAs.

Next step is BINARYMIME; much larger benefit, much larger cost.
Involves storage formats (no mbox) and IMAP software and MUAs; the
whole email ecosystem.

We know CHUNKING is doable, Microsoft and Google both do it.
(I also do it.)

Question is, is BINARYMIME doable?  Or a lost cause?

---

From: John Bucy <[email-masked]>
Date: Thu, 10 Jan 2019 11:05:07 -0800

> Question is, is BINARYMIME doable?  Or a lost cause?

As I see it, the problem is a message getting stuck "in the middle" where
the next hop doesn't support the extension e.g. user has a .forward file.
The most obvious options to me are:
1. downgrade the message from binary to base64. dmarc made this infeasible
at least for senders that use it but I'm hopeful that arc can fix that. Not
sure about s/mime.
2. An MTA could not accept an extension for a particular recipient if it
isn't sure that there is no next hop or the next hop supports it (i.e. fail
RCPT) so that this can be pushed all the way back to submission. I think we
identified the correct extended status for this for (at least) binarymime
and smtputf8 in a previous thread.

With respect to deploying extensions, sites with millions of mailboxes
- want to gradually deploy any new feature rather than knifeswitch to
everyone at once
- may have a long tail of mailboxes that take a long time to upgrade (e.g.
gateways to a legacy system) and it would be nice for the 99% to get the
benefit sooner

cheers
john

---

From: "John Levine" <[email-masked]>
Date: 10 Jan 2019 14:15:11 -0500

>Question is, is BINARYMIME doable?  Or a lost cause?

After 18 years, I think the answer should be obvious.

At modern network speeds, base64 encoding is good enough for the file
that people attach to mail messsages.  If you have something big
enough that BINARYMIME would make a difference, sensible people use
something like dropbox and send a link.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "John R Levine" <[email-masked]>
Date: 10 Jan 2019 14:48:19 -0500

> the next hop doesn't support the extension e.g. user has a .forward file.

For systems that have separate MUA and MTA, the MUA has no way to tell 
whether a downstream recipient will support BINARYMIME, and like you said, 
trying to recode on the fly causes more problems than it solves.

I see that Gmail has a 25M message size limit, which allows an 18M 
attached file encoded in base64.  If you supported BINARYMIME so the 
attached files could be 24M, would anyone care?

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John Bucy <[email-masked]>
Date: Thu, 10 Jan 2019 11:55:53 -0800

On Thu, Jan 10, 2019 at 11:48 AM John R Levine <[email-masked]> wrote:

> whether a downstream recipient will support BINARYMIME

The MSA could call ahead/cut-through, doesn't exim do that? That might also
allow for the mua to throw an error ui at the user if they fat-fingered the
recipient rather than getting a bounce back later.

cheers
john

---

From: [email-masked]
Date: Thu, 10 Jan 2019 15:07:44 -0500

> recipient rather than getting a bounce back later.

No.

Consider this reply.  I'm in Comcast cable territory, which means that I can
only do outbound port 25 to Comcast/Xfinity servers.  So my only realistic
way to get this mail out is to 587 it to Google's submission servers.

Now how do I "call ahead" for the cc: that's going to John Levine? There's
no way for me to contact mx1.taugh.com and ask "Hey, I'm about to hand
an email to Google, do you support BINARYMIME?" 

And if I *did* make that connection, why should I not just *send the message*?

Hint: Find out what "direct to MX spam" is, and why Comcast blocks outbound
port 25.

---

From: "John R Levine" <[email-masked]>
Date: 10 Jan 2019 15:48:00 -0500

> recipient rather than getting a bounce back later.

It is my impression that some do, but most don't.  There isn't an SMTP 
error code for "downstream recipient doesn't accept BINARYMIME" so we'd be 
talking about a great deal of duct tape to make that work.

Rather than do that, I'd rather do different duct tape that stores the 
giant blob on an external server via soemthing like drive, box, or 
dropbox, and puts a message/external-body part into the message.  See RFC 
2017.

Regards,

---

From: "John Levine" <[email-masked]>
Date: 10 Jan 2019 16:07:27 -0500

In article <[email-masked]> you write:
>Now how do I "call ahead" for the cc: that's going to John Levine?

The theory is that you submit the whole message to Google and it
probes the recipients before it accepts the message, but now you have
the added issue of how to report back that receipient A can handle it
but recipient B cannot.

I'd really rather the duct tape be applied to make external-body work
better.

R's,
John

---

From: [email-masked]
Date: Thu, 10 Jan 2019 16:49:23 -0500

> the added issue of how to report back that receipient A can handle it

That will teach me to reply before caffeine. ;)  I was discussing the MUA-MSA
interaction. And you'd think that somebody like me would know that stuff :)

You can receive a RCPT TO, and probe before returning a 250/550. You can cheat
a bit if you turn on PIPELINE, then you can accept up to the DATA, and do
probes in parallel with accepting the next RCPT before feeding back all the 2xx
Recipient OK and 5xx Won't Work replies.

That of course just kicks the can back up the road, as the new problem is "OK,
so knowing that, what does the MUA do *now*?"

This is the point where the software engineer looks at the e-mail from the project
manager that shows the ship date, and says "Screw it" and puts in code to
just base64 the damned thing :)

> I'd really rather the duct tape be applied to make external-body work

Agreed.  The last time I looked at this (admittedly quite some time ago), the
biggest missing parts were an RFC-standard way to denote an expiration time
("this URL good until <datestamp>") and a secure way to pass an indication of
what credentials are needed to access the object.

---

From: "John R Levine" <[email-masked]>
Date: 10 Jan 2019 17:09:53 -0500

On Thu, 10 Jan 2019, [email-masked] wrote:
> ("this URL good until <datestamp>")

Wouldn't that be the expiration= parameter?  See RFC 1521, section 7.3.3.

> and a secure way to pass an indication of what credentials are needed to 
> access the object.

That seems rather un-mail-ish.  The usual model is that if I can read the 
message, I can read anything in the message.  If someone else can peek at 
the message in transit or whatever, they can read whatever's in it.  If 
the message is supposed to be secret, encrypt it with S/MIME or PGP, or 
maybe just encrypt the external-body part.  To make the URL hard to guess, 
make it long and pseudo-random, but most URLs from Drive, Dropbox, et al. 
are anyway.

Regards,

---

From: Дилян Палаузов <[email-masked]>
Date: Thu, 10 Jan 2019 23:43:48 +0000

Hello,

> the added issue of how to report back that receipient A can handle it

PRDR (per recipient delivery response)?  Just like LMTP-after-data-per-recipient acceptance/rejection, but for SMTP,
possibly adding extra DSN codes.  PRDR solves other also problems, like stating “the first recipient accepts the
message, and the second recipient does not” (e.g. the second recipient has different spam rules).

Last time PRDR was discussed here was 2014.  Since 2013 it is integrated in Exim and back in March 2014 it was not
enabled by default.  Today in Exim enable_prdr still defaults to False in the documentation, but the
src/configure.default sets it to true.

Why isn’t enable_prdr in Exim enabled nowadays by default?

Regards
  Дилян

PRDR specification → http://www.eric-a-hall.com/specs/draft-hall-prdr-00.txt

---

From: John Bucy <[email-masked]>
Date: Thu, 10 Jan 2019 17:37:24 -0800

On Thu, Jan 10, 2019 at 3:44 PM Дилян Палаузов <[email-masked]>
wrote:

> PRDR (per recipient delivery response)?  Just like
> LMTP-after-data-per-recipient acceptance/rejection, but for SMTP,
I would like to see PRDR happen but I don't think it's actually required in
this case; you know it's binary (or eai) at MAIL time so RCPT could fail
for that.

---

From: John Bucy <[email-masked]>
Date: Thu, 10 Jan 2019 17:40:39 -0800

> I'd really rather the duct tape be applied to make external-body work

Getting the blobs out of band may well be better for large messages. Even
if binary isn't the most compelling case for this, I would like to see a
better way to incrementally roll out new features that require end-to-end
support that is less prone to extended messages leaking out to everyone
else that isn't ready yet.

---

From: Keith Moore <[email-masked]>
Date: Thu, 10 Jan 2019 20:52:09 -0500

No.

---

From: Gilles Chehade <[email-masked]>
Date: Fri, 11 Jan 2019 14:55:01 +0100

> Recipient OK and 5xx Won't Work replies.

Do we rename SMTP to MTP while at it ? :-)

-- 
Gilles Chehade						       @poolpOrg

---

From: Gene Hightower <[email-masked]>
Date: Fri, 11 Jan 2019 12:51:03 -0800

> After 18 years, I think the answer should be obvious.

Yes: we need just a few more years before the old guard dies off and
makes way for generational change.

I'm keeping the faith!

On 10/01/2019 11.15, John Levine wrote:

> At modern network speeds, base64 encoding is good enough for the
> file that people attach to mail messsages.

I pay per mebibyte for my (Google Fi) data plan.

> If you have something big enough that BINARYMIME would make a
> difference,

Gmail says “SIZE 157286400” — 150 mebibytes.  I, too, set that same
limit for my mail.

> sensible people use something like dropbox and send a link.

That should tell you that the SMTP protocol fails compared to HTTP
which supports binary transfer.

---

From: Gene Hightower <[email-masked]>
Date: Fri, 11 Jan 2019 13:00:15 -0800

On 10/01/2019 11.05, John Bucy wrote:

> As I see it, the problem is a message getting stuck "in the middle"
> where the next hop doesn't support the extension e.g. user has a
> .forward file.

> The most obvious options to me are:

Why not support BINARYMIME if the MTA is doing a local delivery for
that mailbox and it's local storage format can support it.

I think re-coding in the submission agent isn't too bad since very
often that's the step that does the DKIM signing, so no signature
breakage.

For any other situation, skip the BINARYMIME.

My second point: why not start with CHUNKING?  Absolutely no
compatibility problems with this necessary first step.

---

From: "John R Levine" <[email-masked]>
Date: 11 Jan 2019 16:11:57 -0500

> that mailbox and it's local storage format can support it.

Because mail is complicated, and hacks that support some but not all 
deliveries make life miserable for everyone.  I cannot easily tell you 
what mail sent to my system will be delivered to local mailboxes, what 
will be consumed by programs which may or may not handle binarymime, and 
what will be forwarded to other systems.  (Most mail goes through procmail 
which decides what to do with it.)  I also cannot tell you what MUAs my 
users use and which ones handle binarymime.

For any non-trivial mail system, that level of complexity is quite 
typical.

Regards,

---

From: Gene Hightower <[email-masked]>
Date: Fri, 11 Jan 2019 13:22:21 -0800

On 10/01/2019 12.48, John R Levine wrote:

> Rather than do that, I'd rather do different duct tape that stores
> the giant blob on an external server via soemthing like drive, box,
> or dropbox, and puts a message/external-body part into the message.
> See RFC 2017.

Or the MTA could sore it.  Make it available via some URL that can be
forwarded.  No need for support in the MUA if the MTA can do it.

The MTA has to cache the message in non-volatile storage anyway.  With
the price of storage now, holding it for 24 hours (or more?  36?)
might well be doable.

---

From: "John R Levine" <[email-masked]>
Date: 11 Jan 2019 16:24:30 -0500

> forwarded.  No need for support in the MUA if the MTA can do it.

Yeah, I think I've seen MTAs that do that.

Regards,

---

From: Gene Hightower <[email-masked]>
Date: Fri, 11 Jan 2019 13:38:24 -0800

> For any non-trivial mail system, that level of complexity is quite

Yes, for the complicated case, it's complicated.

But how many people use a webmail service, access their mail via the
web interface and delivery is direct, one hop?  Maybe it's 99.x% of
email users?  The 0.y% of power users with procmail rules and strange
forwarding and all that keep using DATA with base64.

---

From: "John R Levine" <[email-masked]>
Date: 11 Jan 2019 16:48:31 -0500

> forwarding and all that keep using DATA with base64.

If you think that the internals of a system like gmail or yahoo are 
simple, you don't know much about webmail.  They don't have procmail but 
they provide very complex per-user processing.

Regards,

---

From: [email-masked]
Date: Fri, 11 Jan 2019 16:52:48 -0500

> forwarding and all that keep using DATA with base64.

The problem arises when Joe Sixpack sends Fred Poweruser an email,
and Joe's MTA has to know what Fred's system does.

And when you're running at scale, even 4 9's worth of "worked right" means
a very high potential number of trouble tickets...

---

From: Gene Hightower <[email-masked]>
Date: Fri, 11 Jan 2019 13:58:40 -0800

On 11/01/2019 13.48, John R Levine wrote:

> simple, you don't know much about webmail.

Did I say simple?

> They don't have procmail but they provide very complex per-user
> processing.

But it's all within their own systems, it's one administrative domain.

That makes it *doable*.

What makes it hard is if it's multi-hop and the next hop is run by
some external entity.

Or if the mail will be displayed by some random MUA that may or may
not support BINARYMIME.

Again: can we start with CHUNKING?  Does anyone *object* to CHUNKING?

---

From: Ned Freed <[email-masked]>
Date: Sun, 13 Jan 2019 09:11:55 -0800 (PST)

> Yeah, I think I've seen MTAs that do that.

They do. At submission, security gateway, and final delivery.

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 13 Jan 2019 13:40:24 -0500

> They do. At submission, security gateway, and final delivery.

I see that Thunderbird will store attached files on Box or similar 
services. although it just edits the link into the text of the message 
rather than making it a separate MIME part.

Regards,

---

From: Ned Freed <[email-masked]>
Date: Mon, 14 Jan 2019 08:18:28 -0800 (PST)

> rather than making it a separate MIME part.

You will also find cases where such links are themselves replaced with a
redirection through some sort of filter, in order to prevent the attack
where a message is sent with a reference to something innocuous that
gets by filter checks, but is subsequently replaced with malware.

IME these sorts of filter mechanisms work poorly, but good luck getting
rid of them.

				Ned

---

From: John Bucy <[email-masked]>
Date: Wed, 16 Jan 2019 10:20:23 -0800

> Again: can we start with CHUNKING?  Does anyone *object* to CHUNKING?

CHUNKING works today. Hotmail/Exchange/Office365 was the only thing that I
was aware of that advertised it when I implemented it for Gmail ~2013. IIRC
Exim added it a few years ago. Postfix added it in 2018, not sure if it's
enabled by default.

cheers
john

---

From: "John R Levine" <[email-masked]>
Date: 16 Jan 2019 13:50:46 -0500

>> Again: can we start with CHUNKING?  Does anyone *object* to CHUNKING?

I don't object to it, but since it doesn't solve any real problem, why 
bother?

Regards,

---

From: Jeremy Harris <[email-masked]>
Date: Wed, 16 Jan 2019 19:15:18 +0000

> I don't object to it, but since it doesn't solve any real problem, why

Efficiency.  Not having to do dot-stuffing means that sendfile etc.
can be used.
-- 
Cheers,
  Jeremy

---

From: Gene Hightower <[email-masked]>
Date: Wed, 16 Jan 2019 11:18:44 -0800

On 16/01/2019 10.20, John Bucy wrote:

> CHUNKING works today. Hotmail/Exchange/Office365 was the only thing
> that I was aware of that advertised it when I implemented it for
> Gmail ~2013.

Yes: I, too, implement CHUNKING on my mail system (~Oct 2016).  Of
course, I also advertise BINARYMIME.  No takers just yet.

> IIRC Exim added it a few years ago. Postfix added it in 2018, not
> sure if it's enabled by default.

It's in the latest Postfix release 3.3.2 from November 24, 2018.  It
seems to be enabled by default.  As this version moves into popular
distros I expect to see more CHUNKING in the wild.

It's been in EXIM but might have been disable in a lot of configs due
a bug or two.  Not sure what the current status is.

All this gives me (some small) hope.

But, of course, CHUNKING is the easy fist step to BINARYMIME.  There
are much more difficult issues to resolve before we can move to binary
email.

---

From: Claus Assmann <[email-masked]>
Date: Wed, 16 Jan 2019 11:21:18 -0800

> Efficiency.  Not having to do dot-stuffing means that sendfile etc.

If you store the message in "line format" also in the queue,
there's no need to do any conversion.

---

From: Jeremy Harris <[email-masked]>
Date: Wed, 16 Jan 2019 19:36:00 +0000

> there's no need to do any conversion.

Sorry.  I used "dot-stuffing" as shorthand for
"line consisting of a single dot means end of data-phase".
-- 
Cheers,
  Jeremy

---

From: Gene Hightower <[email-masked]>
Date: Wed, 16 Jan 2019 12:02:56 -0800

> Efficiency.  Not having to do dot-stuffing means that sendfile etc.

Can you use sendfile() after STARTTLS?

---

From: Jeremy Harris <[email-masked]>
Date: Wed, 16 Jan 2019 20:14:14 +0000

> Can you use sendfile() after STARTTLS?

I'm not aware of the common TLS libraries providing an equivalent
interface yet, but once the symmetric-crypto portion is established
in the kernel I'd think it possible.  Perhaps five years from now?
-- 
Cheers,
  Jeremy

---

From: "John Levine" <[email-masked]>
Date: 16 Jan 2019 15:59:27 -0500

>Efficiency.  Not having to do dot-stuffing means that sendfile etc.

On my system, and I think most Unix systems, the line endings are changed
from \n to \r\n on the fly, so you still can't use sendfile.

---

From: "John Levine" <[email-masked]>
Date: 16 Jan 2019 16:16:22 -0500

>in the kernel I'd think it possible.  Perhaps five years from now?

Even if someone puts some of the crypto glop in the kernel, the
benefit of sendfile() is minimal.

The point of sendfile is that the kernel can send network data
directly from the disk buffer.  That's never going to happen if it has
to be encrypted first.  If it has to read the buffer, encrypt it, and
put it back, that's just like copying it while encrypting.

Either way, it seems like a pretty minor optimization.  My MTA
routinely has a hundred connections going at once and it's totally I/O
bound.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Wed, 16 Jan 2019 18:17:19 -0800 (PST)

> Exim added it a few years ago. Postfix added it in 2018, not sure if it's

We (Sun/Oracle) implemented CHUNKING support in 2006. We have not
noticed a significant performance improvement, but our DATA pipeline
is pretty heavily optimized.

A quick check of my home logs over the past year or so shows 37% of outgoing
messages were sent with chunking and 49% were received with it. Not exactly
rare.

The usual problem with having CHUNKING enabled by default is when someone
installs a broken SMTP security proxy that doesn't understand CHUNKING and
doesn't know how to filter the CHUNKING extension out of the EHLO response. And
then complains about things not working.

Such garbage is getting rarer over time but is still all too common.

As for BINARYMIME, we implemented that for SUBMIT in 2012 due to a couple of
customer requests. As I explained previously, our view is that BINARYMIME makes
sense for SUBMIT because it is far more common for it to be
bandwidth-constained than SMTP transfers, and transcoding is acceptable at
submit time.

We don't have any real way of knowing to what extent it's used, but given the
relative paucity of problem reports my suspiciion is "not much". I'm not
entirely sure why - a SUBMIT client support should not be that difficult.

				Ned

---

From: Gene Hightower <[email-masked]>
Date: Wed, 16 Jan 2019 19:31:14 -0800

On 16/01/2019 18.17, Ned Freed wrote:

> As for BINARYMIME, we implemented that for SUBMIT in 2012 due to a
> couple of customer requests.

Microsoft, for their hotmail.com service anyway, does support
BINARYMIME for received mail.  And, as far as I can tell, it works.

I send a photo with "Content-Type: image/png; charset=binary" and view
it using their web interface and I see my photo.  Sent it all one big
fat BDAT.

Strangely, when Microsoft sends mail they don't seem to use BINARYMIME
even though I offer the extension.

Microsoft, so it seems, have overcome the problems that seem to plague
other email software and service providers with regard to BINARYMIME.

---

From: Paul Smith <[email-masked]>
Date: Thu, 17 Jan 2019 09:24:53 +0000

> even though I offer the extension.
To be honest, I'd think BINARYMIME would be easy to implement on a final 
delivery SMTP server (as you KNOW what's downstream from you, and that 
they won't need to recode the message).

However, on the outgoing part of a submission server (or on an 
intermediate MTA), it's a lot more tricky, as you don't know what you 
may be sending to and whether it'll need recoding.

So, that's quite possibly why Microsoft accept it on incoming, but don't 
use it on outgoing. Seems sensible to me.

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: Steve Atkins <[email-masked]>
Date: Thu, 17 Jan 2019 09:49:20 +0000

> On Jan 17, 2019, at 9:24 AM, Paul Smith <[email-masked]> wrote:
> 
> To be honest, I'd think BINARYMIME would be easy to implement on a final delivery SMTP server (as you KNOW what's downstream from you, and that they won't need to recode the message).
> 
> However, on the outgoing part of a submission server (or on an intermediate MTA), it's a lot more tricky, as you don't know what you may be sending to and whether it'll need recoding.
> 
> So, that's quite possibly why Microsoft accept it on incoming, but don't use it on outgoing. Seems sensible to me.

The time I've been bitten by this in the wild was when Inbound mail was delivered to Exchange, which then attempted to forward it to the final recipient. The final recipient had didn't speak binarymime, Exchange had no idea what to do in that case so just dropped the message on the floor.

I fixed it by modifying the final recipient to accept binarymime, but I'm not sure what I'd have done if I hadn't been able to do that and it really showed how that part of the protocol is kind of broken.

Cheers,
  Steve

---

From: Ned Freed <[email-masked]>
Date: Thu, 17 Jan 2019 04:29:52 -0800 (PST)

> even though I offer the extension.

Likely because they transcoded it to 8bit almost immediately. That's what we
do. There's really no choice because pretty much everyone attaches signatures
these days - specifically DKIM - and once you do that transcoding is no longer
an option. 

Delaying signing or resigning in the event your initial choice to transcode
turns out to be incorrect is pretty unappealing implementation-wise. And even
if we tried to accomodate the minescule number of SMTP BINARYMIME servers out
there, this would result in an increase in delivery failures (for those that
don't/can't transcode), message content (DSN delivery failures when the DSN is
also binary), and DKIM signature breakage.

This is not a difficult call to make.

> other email software and service providers with regard to BINARYMIME.

Clearly not, as you yourself have noted.

					Ned

---

From: Gene Hightower <[email-masked]>
Date: Thu, 17 Jan 2019 04:55:05 -0800

> Clearly not, as you yourself have noted.

Microsoft supports BINARYMIME for received mail on port 25.

This something that, I hear, is very difficult for others to do.  I'm
told it's very complicated.  Somehow they have figured out how to make
this work.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 17 Jan 2019 14:06:28 +0100

> told it's very complicated.  Somehow they have figured out how to make

The difficult bit is what to do if you receive a message that's signed and 
that depends on binarymime, need to forward it, and the next-hop receiver 
doesn't support binarymime. Didn't someone just say that Microsoft gets 
that case wrong?

The binarymime spec itself isn't very complicated.

Arnt

---

From: Ned Freed <[email-masked]>
Date: Thu, 17 Jan 2019 05:28:11 -0800 (PST)

> told it's very complicated.  Somehow they have figured out how to make

They have not figured out how to make it work, because it's not possible to
make it work in general. Transcoding destroys DKIM signatures, period. So they
either:

(1) Reject messages sent with BINARYMIME and signatures, either in all cases
    or in cases where they know they are going to forward, leading to
    unncessary failures, or,

(2) Accept such messages and trash the signatures, leading to unnecessary
     failures, and,

(3) Deal with dynamic forwarding cases either by not having any or
    falling back to (1) or (2), leading to unnecessary failures.

My guess is that the number of BINARYMIME messages they get via SMTP is
so tiny they haven't had to deal with the problems. And of course since
the people attempting to send them BINARYMIME encounter problems they
probably give up on the extension.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Thu, 17 Jan 2019 05:36:39 -0800 (PST)

> The binarymime spec itself isn't very complicated.

Conceptually, maybe, but some of the boundary conditions in a transcoder
are ... interesting.

Add message/global to the mix with nested encodings and it's even more
fun.

				Ned

---

From: Gene Hightower <[email-masked]>
Date: Thu, 17 Jan 2019 21:37:57 -0800

On 17/01/2019 05.28, Ned Freed wrote:

> They have not figured out how to make it work, because it's not
> possible to make it work in general. Transcoding destroys DKIM
> signatures, period.

This mailing list destroys DKIM signatures.

So: presumably transcoding from 8BITMIME to 7 bit has all the same
problems, no?  But somehow 8BITMIME became common enough for everyone
to just assume 8 bit transmission now.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 18 Jan 2019 08:15:16 +0100

> to just assume 8 bit transmission now.

It didn't have the same problems at the time when it became common.

Just-send-8 became common around 1990, 8BITMIME was formalised a few years 
later. Spam and signatures arrived on the scene later.

Arnt

---

From: Ned Freed <[email-masked]>
Date: Fri, 18 Jan 2019 07:28:50 -0800 (PST)

> This mailing list destroys DKIM signatures.

And doing so causes serious problems. Significant effort is underway to address
this in the DMARC WG.

> problems, no?

If it were happening now, yes, absolutely.

> But somehow 8BITMIME became common enough for everyone

Because the transition happened literally decades before DKIM existed.

				Ned

---

From: Brandon Long <[email-masked]>
Date: Wed, 30 Jan 2019 16:02:58 -0800

> probably give up on the extension.

Um, Exchange trashes signatures on forwarding even without BINARYMIME.
They've been
trying to fix this for years, and only recently made it sorta work for very
specific forwarding
situations.  They re-write multiple headers to be written in their style,
including ones in the
"body" section from a DKIM perspective.

Brandon

---

From: Gene Hightower <[email-masked]>
Date: Wed, 30 Jan 2019 20:14:34 -0800

> Um, Exchange trashes signatures on forwarding even without BINARYMIME. 

I was talking about their hotmail service.  That's what I've tested.
Don't know about exchange.