Subject: [ietf-smtp] CHUNKING and PIPELINING

From: Jeremy Harris <[email-masked]>
Date: Sat, 6 Mar 2021 23:09:13 +0000

Is is permitted to pipeline a QUIT after a BDAT LAST?

RFC 3030 (SMTP Service Extensions
           for Transmission of Large
           and Binary MIME Messages)
does not say whether or not BDAT LAST is a synch point per
RFC 2929 (SMTP Service Extension for Command Pipelining).

2920 says (3.1) :-
"The actual transfer of message content is explicitly allowed to be
    the first "command" in a group."

-- 
Cheers,
   Jeremy

---

From: Ned Freed <[email-masked]>
Date: Sat, 06 Mar 2021 16:58:47 -0800 (PST)

> Is is permitted to pipeline a QUIT after a BDAT LAST?

RFC 5321 section 4.1.1.10 is quite clear that QUIT can be issued at
any time. The state of the connection doesn't matter. 

So yes, a QUIT can be pipelined after BDAT LAST. It doesn't matter
if the BDAT succeeds or fails.

> RFC 2929 (SMTP Service Extension for Command Pipelining).

That would only matter if there were restrictions on when QUIT can
be set.

>     the first "command" in a group."

				Ned

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 7 Mar 2021 15:14:10 -0500

> On Mar 6, 2021, at 7:58 PM, Ned Freed <[email-masked]> wrote:
> 

Are you sure about that?  For example, PIPELINING of QUIT with
"DATA" would often not have the intended effect. :-)

With PIPELINING, the client needs to wait for all the server replies
after EHLO, DATA, VRFY, EXPN, TURN, QUIT, and NOOP:

	https://tools.ietf.org/html/rfc2920#section-3.1

before sending further commands.  So the question is where "BDAT LAST"
fits into the picture.

> be sent.

But there are such restrictions, it must not be sent pipelined with
any of the list above (admittedly sending QUIT again after QUIT is
rather redundant).

What Jeremy did not mention is that he's seeing interoperability
issues (reportedly with Google and IIRC Yahoo) when pipelining
"BDAT nnnn LAST<CRLF>QUIT<CRLF>".

Did Google, et. al. not implement PIPELINING for BDAT correctly?
It is not (to me) entirely obvious from the spec.  The PIPELING
semantics should have been spelled out more clearly in 3030.

-- 
	Viktor.

---

From: Jeremy Harris <[email-masked]>
Date: Sun, 7 Mar 2021 21:37:41 +0000

> "BDAT nnnn LAST<CRLF>QUIT<CRLF>".

Specifically, both of the two large providers I've noted giving
problems consistently drop the connection without giving a response
for the BDAT LAST.o

Google is fine with a QUIT pipelined after data-ending-in-dot
(tranditional non-CHUNKING); I've not had a chance to evaluate
yahoodns.net yet.  Both are fine with non-pipelined QUIT and
BDAT (LAST).
-- 
Cheers,
   Jeremy

---

From: Ned Freed <[email-masked]>
Date: Sun, 07 Mar 2021 16:19:34 -0800 (PST)

> yahoodns.net yet.  Both are fine with non-pipelined QUIT and

If so, that's deeply broken, never mind the legality of following a BDAT LAST
with a QUIT. Nothing in the pipeling specification says it is OK for a
later command to in effect be evaluated out of order.

There's even text in RFC 5321 section 4.1.1.10 saying that you can't close the
connection prematurely.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Sun, 07 Mar 2021 20:31:32 -0800 (PST)

> "DATA" would often not have the intended effect. :-)

Sorry, I left out an important restriction: The connection must be in a state
to accept commands. So DATA, AUTH, TURN, and QUIT do require a check. (Although
in the case of QUIT it really doesn't matter - if the command, in violation of
the specification, fails, what, exactly, are you supposed to do? Send it again
and hope it works this time?)

However, if the command in question always leaves the connection in a state to
accept commands, there should be no problem pipelining a command that the
specification say can be sent at any time.

> before sending further commands.

First, I'm not seeing a MUST NOT in there. I also note that the data itself
following a succesful DATA command is not specified as ending a command group,
and RFC 3030 clearly indicates that BDAT without the LAST can be pipelined.

Second, it also says that the reason for this is there's a change of state the
client has to accommodate. Since QUIT is specifically required to be accepted
in any command state, other requirements insure that it will work.

I don't think any conclusion other than BDAT LAST can be pipelined with QUIT is
reasonable. It's OK to pipeline it with a RSET as well, for the same reasons.

> So the question is where "BDAT LAST" fits into the picture.

BDAT, irrespective of the use of LAST, includes the data as part of the command
itself. After it's done, succeed or fail, you're back to accepting commands.

Now, this does leave open the question of how to handle malformed BDAT
commands. This is tricky: For example, what are you supposed to do if
you get:

   BDAT garbage

or better yet:

   BDAT 2456 something-other-than-last

There's also the question of what to do if you have support for BDAT but
it's disabled and you didn't offer it in the EHLO response.

> any of the list above (admittedly sending QUIT again after QUIT is

OK, so let's suppose that it is reasonable to conclude from the language that
BDAT LAST/QUIT pipelining isn't allowed. The question then is given all the
other requirements on clients and servers, e.g. must process commands in order,
must push out responses to group ending commands, must not flush the input
buffer, and so on, is it possible for a compliant server and a client that's
compliant in every other way to screw up?

If there is on, I'm not seeing it.

> semantics should have been spelled out more clearly in 3030.

And the handling of invalid BDAT (especially the second example I gave) as
well. But I think common sense is all you need to see that this should be OK
and there's something wrong in Google-Yahoo-land.

Has anyone tried pipelining RSET after BDAT LAST, or QUIT after the trailing
dot on a regular DATA on these servers? (Our client doesn't attempt any of this
because we can't tell right away whether or not there's another message we want
to send on this connection, so we don't know what command we're going to send
next and we don't want to wait to find out.)

				Ned

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 8 Mar 2021 03:57:58 -0200

> On Mar 8, 2021, at 2:31 AM, Ned Freed <[email-masked]> wrote:
> 

Oops (I guess that was clear from context), I meant:

	BDAT nnnn LAST<CRLF>
	...<nnnn> bytes of data...
	QUIT<CRLF>

> and there's something wrong in Google-Yahoo-land.

That's certainly what I'd like to be the case, but I am not sure the
specification succeeds in making it clear.  The only example shows
QUIT sent separately:

   https://tools.ietf.org/html/rfc3030#section-4.2

   R: <wait for connection on TCP port
   S: <open connection to server>
   R: 220 cnri.reston.va.us SMTP service ready
   S: EHLO ymir.claremont.edu
   R: 250-cnri.reston.va.us says hello
   R: 250-PIPELINING
   R: 250-BINARYMIME
   R: 250 CHUNKING
   S: MAIL FROM:<[email-masked]> BODY=BINARYMIME
   S: RCPT TO:<[email-masked]>
   S: RCPT TO:<[email-masked]>
   R: 250 <[email-masked]>... Sender and BINARYMIME ok
   R: 250 <[email-masked]>... Recipient ok
   R: 250 <[email-masked]>... Recipient ok
   S: BDAT 100000
   S: (First 10000 octets of canonical MIME message data)
   S: BDAT 324
   S: (Remaining 324 octets of canonical MIME message data)
   S: BDAT 0 LAST
   R: 250 100000 octets received
   R: 250 324 octets received
   R: 250 Message OK, 100324 octets received
   S: QUIT
   R: 221 Goodbye

> next and we don't want to wait to find out.)

With plain old "DATA" (Postfix support CHUNKING only in the server role), Postfix
acting as a client always pipelines ".<CRLF>QUIT<CRLF>" and has no trouble with
Google or Yahoo.

[ Postfix only caches connections when there's a sufficient backlog
  for the same destination, so under "normal" conditions we know to
  close the connection right away. ]

So it seems it is specifically "BDAT LAST" + (ultimately) QUIT that
these providers are allergic to.

I take it there's at least agreement on this list that pipelining of
these is not definitely proscribed.  And perhaps even expected?  If
so, perhaps (unless they're already here) time to reach out to Google
et. al., though in the meantime a conservative implementation might
need to not enable pipelining after BDAT by default.

-- 
	Viktor.

---

From: Gene Hightower <[email-masked]>
Date: Mon, 8 Mar 2021 16:45:21 -0800

I can pipeline QUIT after BDAT nnnn LAST to Google's servers and it
seems to work. Works with Microsoft/Outlook, too.

Y! (Oath/AIM/AOL) doesn't support CHUNKING.

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 9 Mar 2021 18:47:18 -0200

> On Mar 8, 2021, at 10:45 PM, Gene Hightower <[email-masked]> wrote:
> 

Thanks, I just tried it, and it worked for me as well.
Jeremy, was there perhaps a bug in your code?

[slightly obfuscated transcript]:

< 220 mx.google.com ESMTP m18si8702635qkn.9 - gsmtp
> EHLO amnesiac.example
< 250-mx.google.com at your service, [192.0.2.1]
< 250-SIZE 157286400
< 250-8BITMIME
< 250-STARTTLS
< 250-ENHANCEDSTATUSCODES
< 250-PIPELINING
< 250-CHUNKING
< 250 SMTPUTF8
> MAIL FROM:<[email-masked]>
> RCPT TO:<[email-masked]>
< 250 2.1.0 OK m18si8702635qkn.9 - gsmtp
< 250 2.1.5 OK m18si8702635qkn.9 - gsmtp
> BDAT 129
> BDAT 0 LAST
> QUIT
< 250 2.0.0 OK m18si8702635qkn.9 - gsmtp
< 250 2.0.0 OK m18si8702635qkn.9 - gsmtp
< 221 2.0.0 closing connection m18si8702635qkn.9 - gsmtp

A tcpdump confirms that pipelining took place:

25 > 34153: Flags [P.], length 53: SMTP: 220 mx.google.com ESMTP e16si7561792qts.371 - gsmtp
34153 > 25: Flags [P.], length 26: SMTP: EHLO amnesiac.example
25 > 34153: Flags [P.], length 168: SMTP: 250-mx.google.com at your service, [192.0.2.1]
34153 > 25: Flags [P.], length 64: SMTP: MAIL FROM:<[email-masked]>
25 > 34153: Flags [P.], length 42: SMTP: 250 2.1.0 OK e16si7561792qts.371 - gsmtp
25 > 34153: Flags [P.], length 42: SMTP: 250 2.1.5 OK e16si7561792qts.371 - gsmtp
34153 > 25: Flags [P.], length 158: SMTP: BDAT 129
25 > 34153: Flags [P.], length 42: SMTP: 250 2.0.0 OK e16si7561792qts.371 - gsmtp
25 > 34153: Flags [P.], length 42: SMTP: 250 2.0.0 OK e16si7561792qts.371 - gsmtp
25 > 34153: Flags [P.], length 58: SMTP: 221 2.0.0 closing connection e16si7561792qts.371 - gsmtp

-- 
	Viktor.

---

From: Jeremy Harris <[email-masked]>
Date: Wed, 10 Mar 2021 21:04:16 +0000

> Jeremy, was there perhaps a bug in your code?

As it turns out, no - or at least not the obvious part.
Some further testing has found that the syndrome
(at least with Google's SMTP servers) is down to my pipelining all of

(SMTP) MAIL, RCPT, BDAT LAST, QUIT, (TLS) close-alert, (TCP) FIN.

The TLS alert and TCP FIN go in the last of the flight of TCP
segments carrying the SMTP commands and data.

Google was actually sending TCP FIN without sending an SMTP response
for the MAIL command, much less the BDAT (which is what I had assumed).

I had thought that sending a TLS close-alert had semantics
"no further data sourced from here, but the other direction
is unaffected" - just like the TCP FIN.  Re-reading RFC 8446
(TLS 1.3 spec) I still think so:  Section 6 "Alert Protocol"  :-

    The
    "close_notify" alert is used to indicate orderly closure of one
    direction of the connection.  Upon receiving such an alert, the TLS
    implementation SHOULD indicate end-of-data to the application.

If I separate out the SMTP commands from the TLS & TCP shutdowns,
reaping the SMTP responses in between, all is well.  If I send
the TLS shutdown with the SMTP commands but not with a TCP FIN,
it is not ok.

Perhaps I need to raise the question on the tls mailinglist.
-- 
Cheers,
   Jeremy

---

From: Ned Freed <[email-masked]>
Date: Wed, 10 Mar 2021 13:18:44 -0800 (PST)

Wow. It honestly hadn't occurred to me to think about TLS and pipelining, but
it obviously warrants some consideration.

While I agree with your analysis, checking on the TLS list sounds like
a really good idea.

				Ned

> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 11 Mar 2021 00:17:00 -0200

> On Mar 10, 2021, at 7:04 PM, Jeremy Harris <[email-masked]> wrote:
> 

My first read of RFC3030 was that one's expected to pause after
all the RCPT commands and before the first BDAT, to check
whether at least one recipient had been accepted.  But this
appears to not be the case.

So indeed you can pipeling the entire MAIL transaction from "MAIL"
to "QUIT" (if the number of BDAT segments is small enough).

What's worse (or better depending on your perspective) is that
you can even pipeline MULTIPLE mail transactions, nothing there
to stop the next "MAIL" command being sent right after "BDAT LAST".

At some point a 3030bis clarifying these issues would be welcome
IMHO.

> for the MAIL command, much less the BDAT (which is what I had assumed).

Here, I'm not surprised.  It is not prudent to start to tear
down the TLS connection before reading all the server replies,
and hence sending a "close_notify" right after a pipelined
"QUIT" is unwise.

>   implementation SHOULD indicate end-of-data to the application.

That text is rather new in RFC8446, and prior versions of TLS (1.0—1.2)
do not support half-close.  And even if they did, many applications are
simply not prepared to handle this.

> the TLS shutdown with the SMTP commands but not with a TCP FIN,

That should be your implementation, only do the TLS close_notify
once you're neither writing nor reading any SMTP chatter.

-- 
	Viktor.

---

From: Jeremy Harris <[email-masked]>
Date: Thu, 11 Mar 2021 09:52:47 +0000

On 11/03/2021 02:17, Viktor Dukhovni wrote:
> do not support half-close.
>  And even if they did, many applications are

https://tools.ietf.org/html/rfc5246#page-29

   The other party MUST respond with a close_notify
    alert of its own and close down the connection immediately,
    discarding any pending writes.

So Google, were it thinking to follow the TLS 1.2 interpretation
(which it shouldn't have; it was a 1.3 session) violated that in
sending a TCP FIN without first sending a TLS close alert.

But, at least for TLS <= 1.2 de jure, I have to agree with you
on not being eager with the TLS close.

I still think I'll call them on it.
-- 
Cheers,
   Jeremy

---

From: Tony Finch <[email-masked]>
Date: Sat, 13 Mar 2021 00:30:21 +0000

> At some point a 3030bis clarifying these issues would be welcome

This reminds me of 2007 when I was writing various speculative drafts
http://fanf2.user.srcf.net/hermes/doc/qsmtp/ about reducing SMTP
submission latency, and the question of freely pipelining multiple
transactions came up back then. I don't think there was enough BDAT in the
wild for it to be a pressing concern, so the ideas never went anywhere.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  https://dotat.at/
South German Bight, Humber: Southwesterly veering westerly 6 to gale 8,
increasing severe gale 9 for a time. Moderate or rough, occasionally very
rough later. Squally showers. Good, occasionally poor.

---

From: "John Levine" <[email-masked]>
Date: 12 Mar 2021 22:00:42 -0500

>to stop the next "MAIL" command being sent right after "BDAT LAST".

Feeling that I wanted to keep up with the kewl kids, I added BDAT to
my SMTP server and looked at the logs.

The largest single users of BDAT are Gmail and Outlook, but I see a lot
of other smaller senders like sailthru.

There's also an impressive amount of BDAT from spambots.  Some of them
even pipeline multiple messages.  My input process has been dying after the
second message's BDAT.  I can't tell whether the input is malformed or
I have some bug, but since it hasn't yet affected any real mail I'll
wait and see if I can collect more data.

I also see some MTAs send a bunch of roughly 8K chunks, while Outlook
sends one big blob.

R's,
John

---

From: Gene Hightower <[email-masked]>
Date: Fri, 12 Mar 2021 22:41:44 -0800

> my SMTP server and looked at the logs.

Oh the slippery slope: what's next? BINARYMIME? :-)

---

From: "John R Levine" <[email-masked]>
Date: 13 Mar 2021 13:52:05 -0500

> Oh the slippery slope: what's next? BINARYMIME? :-)

Funny you sould ask.  My MTA runs on a BSD system where the line endings 
are \n, so it turns \r\n into \n on the way in and reverses is on the way 
out.  For BINARYMIME I'd have to parse the body on the fly and only strip 
\r when it's not in a binary body part.  That is more code than I feel 
like writing this year.

I see that Outlook does BINARYMIME but Gmail doesn't.  I'm with Gmail 
here.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Gene Hightower <[email-masked]>
Date: Sat, 13 Mar 2021 11:27:48 -0800

here.

Microsoft supports BINARYMIME about ½ way; you can send binary message
content to them, but they never seem to use it for outgoing mail.

Apparently Apple shipped a version of Postfix with BINARYMIME support
with “the old macos server” I have been told, but I have not found the
source code for that version anyplace.