Subject: draft-ietf-fax-esmtp-conneg-08.txt
From: Hiroshi Tamura <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked]
Date: Thu, 26 Jun 2003 08:52:26 +0900 (JST)
List-Id: <ietf-smtp.imc.org>

To SMTP experts,

FAX WG did the WG Last Call of the I-D below and it was complete.

http://www.ietf.org/internet-drafts/draft-ietf-fax-esmtp-conneg-08.txt

At San Francisco's meeting, our WG agreed to ask for comments
from SMTP experts after the WG LC.
In fact, last year, you had a good review for the previous I-D.

If there are issues to say, please comment.

Best Regards,
--
Hiroshi Tamura, Co-chair of IETF-FAX WG

Date: Sat, 21 Jun 2003 09:31:31 -0400
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: ambiguities in RFC 2821 regarding end of mail data
Sender: [email-masked]

It seems to me that we are working with two different ideas of the 
"end of mail data indication".

I would argue that it is a line containing only a period.

Others take the position that it is CRLF.CRLF

Those two are different, as someone writing a parser may discover. 
The ambiguity between the two motivated my questions at the start of 
this thread.

RFC 2821 does not seem to recognize the difference.  Rather RFC 2821 
confuses the issue.  This sentence from section 4.1.1.4 can not be true:

"The mail data is terminated by a line containing only a period, that 
is, the character sequence "<CRLF>.<CRLF>" (see section 4.5.2)."

because the character sequence "<CRLF>.<CRLF>" IS NOT a line 
containing only a period.  Mail data may be terminated by one or the 
other but not both, because they are different and mutually exclusive.

We write "CRLF.CRLF" because that is an easy way to show that we 
intend a line containing only a period.  But CRLF.CRLF emphatically is 
not a line containing only a period.  A line -- one line -- cannot 
contain two CRLF sequences.

Rich

Date: Sat, 21 Jun 2003 07:39:36 -0400
From: Keith Moore <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: ambiguities in RFC 2821 regarding end of mail data
Sender: [email-masked]

On Fri, 20 Jun 2003 18:10:22 -0400
"Richard O. Hammer" <[email-masked]> wrote:

> 
> If Vladis and Keith are right (that the entire CRLF.CRLF should be 
> dropped), then the final "line" of data would typically have no CRLF. 
>   It would consist of a set of bytes ending without a CRLF.
> 
> Do you think that is right?

what you might see is ... cr lr cr lf . cr lf

in other words, the last line of the message really does end in cr lf,
and is followed by cr lf . cr lf.

if I'm writing code to send a message, I certanly don't want to trust
the message to end in cr lf, and I don't want to check either.  
I want to say

send ("data\r\n")
get (response)
send (message)
send ("\r\n.\r\n")
get (response)

though I'd be curious if anyone has tried to measure how many messages
received don't end in CR LF.

Date: 20 Jun 2003 23:18:29 -0000
Mail-Followup-To: [email-masked]
Automatic-Legal-Notices: See http://cr.yp.to/mailcopyright.html.
From: "D. J. Bernstein" <[email-masked]>
To: [email-masked]
Subject: Re: ambiguities in RFC 2821 regarding end of mail data
Sender: [email-masked]

Richard O. Hammer writes:
> Do you think that is right?

No. The final line is always complete. Partial final lines can't be
transmitted through SMTP. See http://cr.yp.to/smtp/message.html.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

Date: Fri, 20 Jun 2003 18:48:32 -0400
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: Re: ambiguities in RFC 2821 regarding end of mail data
Sender: [email-masked]

Keith Moore wrote:

>>The second ambiguity relates to the first, and it concerns the 
>>question: When I receive a period alone in the first line of mail data
>>should I consider that the end of the data?
> 
> you mean, you receive the sequence
> 
> 44 41 54 41 0D 0A 2E 0D 0A 
> 
> ?

Yes, I suppose that would be the sequence which I am asking about. 
But the way we do our parsing of data I do not see it all together 
that way, since the first OD OA are consumed by a routine which reads 
command lines (which are recognized as lines because they end with OD 
OA).  When DATA is recognized in a command line then the InputStream 
is handed to another parsing routine, specialized for reading data, 
which starts reading where the command-line parser left off, after the 
first OD OA.

> no it's not the end of mail data, since the CR LF after DATA
> cannot be part of the end-of-data marker.  OTOH it's also an SMTP
> protocol error,

This exchange has stimulated me to do the homework which I probably 
should have done before I started.  I have just seen this text in RFC 
2821, section 4.1.1.4, second paragraph:

"The mail data is terminated by a line containing only a period, that 
is, the character sequence "<CRLF>.<CRLF>" (see section 4.5.2).  This 
is the end of mail data indication.  Note that the first <CRLF> of 
this terminating sequence is also the <CRLF> that ends the final line 
of the data (message text) or, if there was no data, ends the DATA 
command itself."

I note two things from this.  First, "if there was no data, ends the 
DATA command itself", allows the possibility that there may be no 
data.  And it seems to answer my second question: a period alone in 
the first line of data ends the data.

Second, turning back now to the first question which I asked, notice 
that the first sentence of that paragraph from RFC 2821 contradicts 
itself.  The character sequence "<CRLF>.<CRLF>" is not "a line 
containing only a period".  It a CRLF -- put there to show that we 
have really reached the end of the prior line -- followed by a line 
containing only a period.  So this sentence gives us two possible "end 
of mail data indication"s.   One is "<CRLF>.<CRLF>" and the other is 
".<CRLF>" alone in a line.  They are not the same.  But I think the 
later is more consistent with other material in this RFC.

Rich

Date: Fri, 20 Jun 2003 18:10:22 -0400
From: "Richard O. Hammer" <[email-masked]>
CC: [email-masked]
Subject: Re: ambiguities in RFC 2821 regarding end of mail data
Sender: [email-masked]

If Vladis and Keith are right (that the entire CRLF.CRLF should be 
dropped), then the final "line" of data would typically have no CRLF. 
  It would consist of a set of bytes ending without a CRLF.

Do you think that is right?

Rich

To: "Richard O. Hammer" <[email-masked]>
Cc: [email-masked]
Subject: Re: ambiguities in RFC 2821 regarding end of mail data 
From: [email-masked]
Date: Fri, 20 Jun 2003 17:35:39 -0400
Sender: [email-masked]

--==_Exmh_131023789P
Content-Type: text/plain; charset=us-ascii

On Fri, 20 Jun 2003 16:50:17 EDT, "Richard O. Hammer" <[email-masked]>  said:

> The first ambiguity concerns this question:  When I receive this 
> sequence CRLF.CRLF, should the data which I capture include the first 
> CRLF or not?

No.

> But the RFC leaves room for confusion.  This phrase from section 3.3
> might be interpreted both ways, "...the SMTP server ... considers all
> succeeding lines up to but not including the end of mail data
> indicator to be the message text."  What is "the end of mail data
> indicator"?  Section 4.1.1.4 calls the whole CRLF.CRLF the "end of
> mail data indication".

Right.  Nothing confusing there.

> The second ambiguity relates to the first, and it concerns the 
> question: When I receive a period alone in the first line of mail data 
> should I consider that the end of the data?

Well... unless you're using SMTP to pass non 822/2822 mail, you're going
to be dropping it on the floor ANYHOW because if that's the end of the
data, there's not much to deliver (you need *at least* a CRLF pair to
mark the end-of-822 headers and start-of-body).

> I believe that I should consider the sequence .CRLF to be the end of 
> the data if those are the first three bytes received by the data 
> parsing routine.

Unless you're doing non-822 mail, what you've received at that point is a
broken set of RFC822 headers consisting of the single octet '.'

--==_Exmh_131023789P
Content-Type: application/pgp-signature

--==_Exmh_131023789P--

Date: Fri, 20 Jun 2003 17:06:43 -0400
From: Keith Moore <[email-masked]>
To: "Richard O. Hammer" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: ambiguities in RFC 2821 regarding end of mail data
Sender: [email-masked]

> I am writing code to parse incoming SMTP bytes, and I have encountered
> two ways in which RFC 2821 seems ambiguous regarding handling of the 
> end of mail data indicator.  I hope that this list is the appropriate 
> place for me to make this report.
> 
> The first ambiguity concerns this question:  When I receive this 
> sequence CRLF.CRLF, should the data which I capture include the first 
> CRLF or not?

strictly speaking, no.  though you'll find many MTAs that do.

> The second ambiguity relates to the first, and it concerns the 
> question: When I receive a period alone in the first line of mail data
> should I consider that the end of the data?

you mean, you receive the sequence

44 41 54 41 0D 0A 2E 0D 0A 

?

no it's not the end of mail data, since the CR LF after DATA
cannot be part of the end-of-data marker.  OTOH it's also an SMTP
protocol error, since if the sender really were trying to transmit
a first line consisting of a single '.' then it would be expected
to double it.  given that it wouldn't be a valid message anyway,
I wouldn't try too hard to handle this case - just make sure your
MTA doesn't crash or introduce a security hole when it happens.
if your MTA waits for the rest of the message while the sending
MTA waits for your response to the end of the message and one of
you eventually times out, and you discard the message, the right
thing happens anyway - it just takes longer.  but since it shouldn't
occur very often (like maybe ever) then there's no point in optimizing
it.

Keith

Date: Fri, 20 Jun 2003 16:50:17 -0400
From: "Richard O. Hammer" <[email-masked]>
To: [email-masked]
Subject: ambiguities in RFC 2821 regarding end of mail data
Sender: [email-masked]

I am writing code to parse incoming SMTP bytes, and I have encountered 
two ways in which RFC 2821 seems ambiguous regarding handling of the 
end of mail data indicator.  I hope that this list is the appropriate 
place for me to make this report.

The first ambiguity concerns this question:  When I receive this 
sequence CRLF.CRLF, should the data which I capture include the first 
CRLF or not?

I believe that the first CRLF should be part of the data, according to
my reading of RFC 2821, sections 2.3.7, 3.3, and 4.1.1.4.  That CRLF
is part of the last line of data, since, as I read 2.3.7 you don't 
have a "line" of data unless it is terminated by CRLF.

But the RFC leaves room for confusion.  This phrase from section 3.3
might be interpreted both ways, "...the SMTP server ... considers all
succeeding lines up to but not including the end of mail data
indicator to be the message text."  What is "the end of mail data
indicator"?  Section 4.1.1.4 calls the whole CRLF.CRLF the "end of
mail data indication".

I noticed this ambiguity because the MTA with which I am familiar, 
james.apache.org, uses a parsing routine which presently does not 
return the first CRLF as part of the mail data.  In conformity with 
one of the ways that the RFC can be understood, it checks for the 
whole end of mail data indicator (CRLF.CRLF) and sends EOF without 
returning any of that indicator including the first CRLF.

The second ambiguity relates to the first, and it concerns the 
question: When I receive a period alone in the first line of mail data 
should I consider that the end of the data?

I believe that I should consider the sequence .CRLF to be the end of 
the data if those are the first three bytes received by the data 
parsing routine.  But again the RFC seems unclear to me.  The parsing 
routine presently used by the MTA with which I am familiar does not 
treat a period alone in the first line as the end of data; it waits 
for the sequence CRLF.CRLF which could only be in the second line or 
later.

Rich Hammer
Hillsborough, N.C.

Date: 	Thu, 19 Jun 2003 16:24:32 +0300
From: Matti Aarnio <[email-masked]>
To: "Vaudreuil, Greg M (Greg)" <[email-masked]>
Cc: [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
Sender: [email-masked]

On Wed, Jun 18, 2003 at 09:29:59AM -0500, Vaudreuil, Greg M (Greg) wrote:
> I hope the below referenced Internet-draft has captured the discussion
> we have been having.  Specific comments would be welcome, preferably in
> time to turn a new revision before the IETF meeting.
> 
> Greg V.

I have two things:

I would like to see section/subsection numbers inside chapter 2,
separating non-pipeline and pipeline behaviour discussion into
separate sections would be usefull in the long run, IMHO.  (Like
there are separate examples, also the text should have sections.)

Due to network level TCP optimizations, the example in 4.2
could be rewritten slightly:

               -------------------------------------

  4.2 Pipelining BINARYMIME 

     The following dialogue illustrates the use of the large message 
     extension to send a BINARYMIME object to two recipients using the 
     CHUNKING and PIPELINING extensions.  This also shows sender side
     TCP buffering effect where data is accumulated, until MTU size
     chunks can be sent, and thus MAIL, RCPTs and first BDAT verb are
     all in one TCP frame:

     R: <wait for connection on TCP port 25> 
     S: <open connection to server> 
     R: 220 example.com SMTP service ready 
     S: EHLO ymir.example2.com 
     R: 250-example.com says hello 
     R: 250-PIPELINING 
     R: 250-BINARYMIME 
     R: 250 CHUNKING 
     S: MAIL FROM:<[email-masked]> BODY=BINARYMIME 
     S: RCPT TO:<[email-masked]> 
     S: RCPT TO:<[email-masked]> 
     S: BDAT 100000    
     S: (First 100000 octets of canonical MIME message data) 
     R: <sometime during that BDAT sending, these responses arrive>
     R: 250 <[email-masked]>... Sender and BINARYMIME ok 
     R: 250 <[email-masked]>... Recipient ok 
     R: 250 <[email-masked]>... Recipient ok 
     S: BDAT 324  
     S: (Remaining 324 octets of canonical MIME message data) 
     S: BDAT 0 LAST 
     S: <sender explicitely PUSHes its outgoing TCP stream
     S:  data in order to achieve a synchronisation point>
     R: 250 100000 octets received 
     R: 250 324 octets received 
     R: 250 Message OK, 100324 octets received 
     S: QUIT 
     R: 221 Goodbye 

               -------------------------------------

-- 
/Matti Aarnio	<[email-masked]>

From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
To: [email-masked]
Subject: RE: BDAT, RFC 3030 protocol clarification?
Date: Wed, 18 Jun 2003 09:29:59 -0500
Sender: [email-masked]

I hope the below referenced Internet-draft has captured the discussion we have been having.  Specific comments would be welcome, preferably in time to turn a new revision before the IETF meeting.

Greg V.

	Title		: SMTP Service Extensions for Transmission of Large and 
                          Binary MIME Messages
	Author(s)	: G. Vaudreuil
	Filename	: draft-vaudreuil-esmtp-binary3-00.txt
	Pages		: 13
	Date		: 2003-6-17

Date: Mon, 16 Jun 2003 20:54:06 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
Sender: [email-masked]

> > yes.  but clients are still not supposed to send BDAT unless the server 
> > has returned 2xx on MAIL and at least one RCPT command.
> 
> The problem is that this requires a mandatory pipe drain just before sending
> the DATA/BDAT command. Given that we can otherwise get a short e-mail down
> to a 3 packet handshake (less if a keep-alive connection is re-used), one
> outbound packet of MAIL/RCPT/(DATA/BDAT)/. and one return packet of a string
> of 250's, requiring a drain would double the number of RTTs involved.

okay, I reread RFC 2920 - I had mis-remembered how it handled the DATA
synchronization problem.  The client has to drain once after EHLO (2 payload
packets) to make sure that the server actually supports PIPELINING and/or
CHUNKING, and once after the BDAT LAST (2 more) to make sure that the 
message was accepted.

> I think the problem here is that for a sufficiently large BDAT body
> (multiple megabytes and up), you have to choose between drinking from the
> fire hose so you stay connected so you can send the client the responses,
> and just doing the FIN thing to tear the connection down and drain the hose
> (and losing the responses in the process).

I'm tempted to suggest that clients could limit the TCP window size until
they're sure they want to see the message, since it's easier to adjust
the TCP window size (at least on some platforms) than the BDAT chunk size.

> I suppose adding language of the form "A client SHOULD drop back from
> pipelining mode when retrying a connection that failed without returning a
> status code" would be too much? (Incidentally, I've encountered mail clients
> behind b0rked firewalls that broke PMTU Discovery - the symptom was that
> pipelining would fail, but it would work when not pipelining...)

I think this is along the same lines is "a server MAY terminate an SMTP
connection if the client keeps sending BDAT even though previous BDAT
commands resulted in errors..." - it is not something that the standard should
require, but it is implementation advice that might be useful.

Keith

Date: Mon, 16 Jun 2003 17:03:34 -0700 (PDT)
From: [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
To: [email-masked]
Cc: Keith Moore <[email-masked]>, "Vaudreuil, Greg M (Greg)" <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> On Mon, 16 Jun 2003 13:23:37 EDT, Keith Moore said:

> > yes.  but clients are still not supposed to send BDAT unless the server
> > has returned 2xx on MAIL and at least one RCPT command.

> The problem is that this requires a mandatory pipe drain just before sending
> the DATA/BDAT command.

Exactly.

> Given that we can otherwise get a short e-mail down to
> a 3 packet handshake (less if a keep-alive connection is re-used), one outbound
> packet of MAIL/RCPT/(DATA/BDAT)/.

Not with DATA -- you have to check to see if the state change was
accepted before you can send the content.

IMO this is the real win of BDAT over DATA.

> and one return packet of a string of 250's,
> requiring a drain would double the number of RTTs involved.

Yep.

> > pipelining clients have an obligation to read server BDAT responses and
> > act on them in a timely fashion, just not to block waiting on those
> > responses.

> I think the problem here is that for a sufficiently large BDAT body (multiple
> megabytes and up), you have to choose between drinking from the fire hose so
> you stay connected so you can send the client the responses, and just doing the
> FIN thing to tear the connection down and drain the hose (and losing the
> responses in the process).

Right again. And you may want to do this even if the MAIL FROM and a RCPT
TO succeeded but you want to fail the BDAT for some other reason.

> I suppose adding language of the form "A client SHOULD drop back from pipelining
> mode when retrying a connection that failed without returning a status code"
> would be too much?

Having written fallback code to handle the connection closing after
sending EHLO, I fear that this may cause as many problems as it solves...

> (Incidentally, I've encountered mail clients behind b0rked
> firewalls that broke PMTU Discovery - the symptom was that pipelining would
> fail, but it would work when not pipelining...)

Yes, I've seen that. Had one just the other day, in fact. I've also seen
instances where pipelining worked and non-pipelined commands failed. (Firewall
nonsense of a different sort.)

				Ned

To: Keith Moore <[email-masked]>
Cc: "Vaudreuil, Greg M (Greg)" <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification? 
From: [email-masked]
Date: Mon, 16 Jun 2003 18:13:42 -0400
Sender: [email-masked]

--==_Exmh_-228734434P
Content-Type: text/plain; charset=us-ascii

On Mon, 16 Jun 2003 13:23:37 EDT, Keith Moore said:

> yes.  but clients are still not supposed to send BDAT unless the server 
> has returned 2xx on MAIL and at least one RCPT command.

The problem is that this requires a mandatory pipe drain just before sending
the DATA/BDAT command.  Given that we can otherwise get a short e-mail down to
a 3 packet handshake (less if a keep-alive connection is re-used), one outbound
packet of MAIL/RCPT/(DATA/BDAT)/. and one return packet of a string of 250's,
requiring a drain would double the number of RTTs involved.

> pipelining clients have an obligation to read server BDAT responses and 
> act on them in a timely fashion, just not to block waiting on those 
> responses.

I think the problem here is that for a sufficiently large BDAT body (multiple
megabytes and up), you have to choose between drinking from the fire hose so
you stay connected so you can send the client the responses, and just doing the
FIN thing to tear the connection down and drain the hose (and losing the
responses in the process).

I suppose adding language of the form "A client SHOULD drop back from pipelining
mode when retrying a connection that failed without returning a status code"
would be too much? (Incidentally, I've encountered mail clients behind b0rked
firewalls that broke PMTU Discovery - the symptom was that pipelining would
fail, but it would work when not pipelining...)

-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

--==_Exmh_-228734434P
Content-Type: application/pgp-signature

--==_Exmh_-228734434P--

Date: Mon, 16 Jun 2003 13:23:37 -0400
Subject: Re: BDAT, RFC 3030 protocol clarification?
Cc: Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked]
To: "Vaudreuil, Greg M (Greg)" <[email-masked]>
From: Keith Moore <[email-masked]>
Sender: [email-masked]

On Monday, June 16, 2003, at 12:31  PM, Vaudreuil, Greg M (Greg) wrote:

> Do I hear an emerging consensus to clarify in the harmony with the 
> pipelining standard?  I see three points to clarify:
>
> 	1) Servers must accept the state-change into receiving BDAT command 
> and associated message data even with no valid RCPT TO recipients or 
> problems with the MAIL FROM command.

yes.  but clients are still not supposed to send BDAT unless the server 
has returned 2xx on MAIL and at least one RCPT command.

> 	2) Servers must send a 4XX or 5XX in response to a BDAT if it can't 
> accept for any reason, incluing having no valid recipients.

must seems a bit too restrictive - it could be read as requiring 
servers to check for any possible failure before returning a BDAT 
response. if we said SHOULD instead of must it would allow servers to 
delay checking for various failures.

> 	3) Servers MAY choose to terminate the TCP connection rather than 
> drink (then expell) a big gulp from a pipelining client.

not in general.  this is  a specific kludge for the case where the 
server returned 5xx to MAIL or to all RCPTs, and the client sent BDAT 
anyway, and the server returned 5xx to that too.  we don't want to give 
servers license to close connections for BDAT under other conditions.

> Clients must interpret the returned status codes and do the right 
> thing (not send that message again, or delay).

pipelining clients have an obligation to read server BDAT responses and 
act on them in a timely fashion, just not to block waiting on those 
responses.

Date: Mon, 16 Jun 2003 09:50:27 -0700 (PDT)
From: [email-masked]
Subject: RE: BDAT, RFC 3030 protocol clarification?
To: "Vaudreuil, Greg M (Greg)" <[email-masked]>
Cc: [email-masked], Keith Moore <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> Do I hear an emerging consensus to clarify in the harmony with the pipelining standard?  I see three points to clarify:

> 	1) Servers must accept the state-change into receiving BDAT command and
>	associated message data even with no valid RCPT TO recipients or
>	problems with the MAIL FROM command.

Correct.

> 	2) Servers must send a 4XX or 5XX in response to a BDAT if it can't
>	accept for any reason, incluing having no valid recipients.

Correct.

> 	3) Servers MAY choose to terminate the TCP connection rather than drink
>	(then expell) a big gulp from a pipelining client.

Correct.

>	 Clients must interpret the returned status codes and do the right
	 thing (not send that message again, or delay).

This presupposes the client got a status code to interpret. It is possible
that the client will get an transmit error and never see the status. In this
case the client has to assume a temporary failure; assuming a permanent failure
in the event of a disconnect without a status would be bad.

				Ned

Date: 	Mon, 16 Jun 2003 19:57:05 +0300
From: Matti Aarnio <[email-masked]>
To: "Vaudreuil, Greg M (Greg)" <[email-masked]>
Cc: [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
Sender: [email-masked]

On Mon, Jun 16, 2003 at 11:31:24AM -0500, Vaudreuil, Greg M (Greg) wrote:
> Do I hear an emerging consensus to clarify in the harmony with the
> pipelining standard?  I see three points to clarify:
> 
>   1) Servers must accept the state-change into receiving BDAT
>	command and associated message data even with no valid RCPT TO
>	recipients or problems with the MAIL FROM command.
>   2) Servers must send a 4XX or 5XX in response to a BDAT if it
> 	can't accept for any reason, incluing having no valid recipients.
>   3) Servers MAY choose to terminate the TCP connection rather than
>	drink (then expell) a big gulp from a pipelining client.  Clients
> 	must interpret the returned status codes and do the right thing
> 	(not send that message again, or delay).

Anyway, perhaps:
    3b) Servers must very carefully flush out all responses before
        closing the connection in case they decide to terminate the
        incoming TCP connection.

(my experience is that if (obvious to us) minute things like these
are not specified, some people will not do them correctly...)

> Greg V.
-- 
/Matti Aarnio	<[email-masked]>

Date: Mon, 16 Jun 2003 12:56:49 -0400
From: Lawrence Greenfield <[email-masked]>
To: Keith Moore <[email-masked]>, [email-masked], <[email-masked]>
Cc: [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
Sender: [email-masked]

   From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
   Date: Mon, 16 Jun 2003 11:31:24 -0500

   Do I hear an emerging consensus to clarify in the harmony with the
   pipelining standard?  I see three points to clarify:

	   1) Servers must accept the state-change into receiving BDAT
	   command and associated message data even with no valid RCPT
	   TO recipients or problems with the MAIL FROM command.

Yes.

           2) Servers must send a 4XX or 5XX in response to a BDAT if
	   it can't accept for any reason, incluing having no valid
	   recipients.

Yes.

	   3) Servers MAY choose to terminate the TCP connection
	   rather than drink (then expell) a big gulp from a
	   pipelining client.  Clients must interpret the returned
	   status codes and do the right thing (not send that
	   message again, or delay).

I'm not sure how this is possible. A client might very well receive an
error from the write() system call (or equivalent) before doing any
reading from the socket.

Are you saying that the client must drain its input after receiving
an error from write(), and act on those SMTP responses? Ugh.

I think the correct thing is to tell servers that they must consume
the data. After all, RFC 3030 is pretty specific about this:

   A 250 response MUST be sent to each successful BDAT data block within
   a mail transaction.  If a failure occurs after a BDAT command is
   received, the receiver-SMTP MUST accept and discard the associated
   message data before sending the appropriate 5XX or 4XX code.  If a
   5XX or 4XX code is received by the sender-SMTP in response to a BDAT
   chunk, the transaction should be considered failed and the sender-
   SMTP MUST NOT send any additional BDAT segments.  If the receiver-
   SMTP has declared support for command pipelining [PIPE], the receiver
   SMTP MUST be prepared to accept and discard additional BDAT chunks
   already in the pipeline after the failed BDAT.

Larry

From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
To: [email-masked], Keith Moore <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: RE: BDAT, RFC 3030 protocol clarification?
Date: Mon, 16 Jun 2003 11:31:24 -0500
Sender: [email-masked]

Do I hear an emerging consensus to clarify in the harmony with the pipelining standard?  I see three points to clarify:

	1) Servers must accept the state-change into receiving BDAT command and associated message data even with no valid RCPT TO recipients or problems with the MAIL FROM command.
	2) Servers must send a 4XX or 5XX in response to a BDAT if it can't accept for any reason, incluing having no valid recipients.
	3) Servers MAY choose to terminate the TCP connection rather than drink (then expell) a big gulp from a pipelining client.  Clients must interpret the returned status codes and do the right thing (not send that message again, or delay).

Greg V.

Date: Sat, 14 Jun 2003 13:43:04 -0700 (PDT)
From: [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
To: Keith Moore <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> > > I suggest that if a BDAT-capable server receives BDAT, it SHOULD either
> > > receive and discard the data, or else send its response and drop the
> > > connection. Shouldn't need saying, but I guess it does not hurt.
> >
> > I concur, but for a different reason: I don't like the implication that
> > a successful MAIL FROM and at least one successful RCPT TO means BDAT
> > will also be successful. It is entirely reasonable for a transaction to
> > success all the way up to the BDAT but fail at that point. In such a case
> > the implementation must be prepared to deal with the data attached to the
> > BDAT whether it likes it or not.

> I agree that there are circumstances in which this is the right thing
> for the server to do.

> > The rule should therefore be that a server which supports BDAT MUST honor the
> > state change implied by the command and deal with the attached data in some
> > way. Small amounts of attached data SHOULD be dealt with by accepting it and
> > discarding it. Large amounts MAY be dealt with by returning an error and
> > shutting down the connection.

> I don't have a problem with that, but we should be clear that "accepting"
> BDAT data that is going to be discarded means returning a 4xx or 5xx response
> (which, hopefully, will eventually be seen by the client).

Agreed.

				Ned

Date: Sat, 14 Jun 2003 13:58:26 -0400
From: Keith Moore <[email-masked]>
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
Sender: [email-masked]

> > I suggest that if a BDAT-capable server receives BDAT, it SHOULD either
> > receive and discard the data, or else send its response and drop the
> > connection. Shouldn't need saying, but I guess it does not hurt.
> 
> I concur, but for a different reason: I don't like the implication that
> a successful MAIL FROM and at least one successful RCPT TO means BDAT
> will also be successful. It is entirely reasonable for a transaction to
> success all the way up to the BDAT but fail at that point. In such a case
> the implementation must be prepared to deal with the data attached to the
> BDAT whether it likes it or not.

I agree that there are circumstances in which this is the right thing
for the server to do.

> The rule should therefore be that a server which supports BDAT MUST honor the
> state change implied by the command and deal with the attached data in some
> way. Small amounts of attached data SHOULD be dealt with by accepting it and
> discarding it. Large amounts MAY be dealt with by returning an error and 
> shutting down the connection.

I don't have a problem with that, but we should be clear that "accepting"
BDAT data that is going to be discarded means returning a 4xx or 5xx response
(which, hopefully, will eventually be seen by the client).

Date: Sat, 14 Jun 2003 10:24:50 -0700 (PDT)
From: [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
To: Arnt Gulbrandsen <[email-masked]>
Cc: Keith Moore <[email-masked]>, "Vaudreuil, Greg M (Greg)" <[email-masked]>, Matti Aarnio <[email-masked]>, [email-masked]
Sender: [email-masked]

> Keith Moore writes:
> > my understanding was  that the client should not send BDAT commands
> > unless it had received a 2xx response from MAIL and at least one of
> > the RCPT commands.

> Well, 1854 says sending DATA is okay, 3030 says BDAT and DATA are the
> same, and AFAIK nothing says that in this respect the two are
> different. IMO the successor to 3030 should say so.

The key difference is that DATA implements a two-stage state change while
BDAT does it in a single stage. So, although the semantics of the command
may be the same, the protocol interaction is very different.

> > failure to do so could cause synchronization problems if the server
> > then said 5xx in response to BDAT because it wasn't going to accept
> > the message anyway.

> > IMHO if a server is ignoring data in the TCP buffer when it claims to
> > support PIPELINING, that server is broken and we shouldn't try to fix
> > it in the spec. but if a server complains when a client sends BDAT
> > for no valid recipients,
> > I don't blame the server for that.

> I suggest that if a BDAT-capable server receives BDAT, it SHOULD either
> receive and discard the data, or else send its response and drop the
> connection. Shouldn't need saying, but I guess it does not hurt.

I concur, but for a different reason: I don't like the implication that
a successful MAIL FROM and at least one successful RCPT TO means BDAT
will also be successful. It is entirely reasonable for a transaction to
success all the way up to the BDAT but fail at that point. In such a case
the implementation must be prepared to deal with the data attached to the
BDAT whether it likes it or not.

The rule should therefore be that a server which supports BDAT MUST honor the
state change implied by the command and deal with the attached data in some
way. Small amounts of attached data SHOULD be dealt with by accepting it and
discarding it. Large amounts MAY be dealt with by returning an error and 
shutting down the connection.

				Ned

From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, Keith Moore <[email-masked]>
Cc: "Vaudreuil, Greg M (Greg)" <[email-masked]>, Matti Aarnio <[email-masked]>, [email-masked]
Subject: RE: BDAT, RFC 3030 protocol clarification?
Date: Fri, 13 Jun 2003 09:14:40 -0500
Sender: [email-masked]

Thanks... you beat me to this observation.  After sending, a nibble of doubt led me to revisit RFC 2920, the standard version of pipelining.  That document does not require that there be a valid recipient before sending data, and advises the server to accept and discard the data.

Based on this, I change my position to resolve what is now clearly an internal contradiction in favor of the clarifying that when pipelining with BDAT, the server is required to accept BDAT and all associated data even if there are no valid recipients.

Greg V.

-----Original Message-----
From: Arnt Gulbrandsen [mailto:[email-masked]]
Sent: Friday, June 13, 2003 3:50 AM
To: Keith Moore
Cc: Vaudreuil, Greg M (Greg); Matti Aarnio; [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?

Keith Moore writes:
> my understanding was  that the client should not send BDAT commands 
> unless it had received a 2xx response from MAIL and at least one of 
> the RCPT commands.

Well, 1854 says sending DATA is okay, 3030 says BDAT and DATA are the 
same, and AFAIK nothing says that in this respect the two are 
different. IMO the successor to 3030 should say so.

> failure to do so could cause synchronization problems if the server  
> then said 5xx in response to BDAT because it wasn't going to accept 
> the message anyway.
>
> IMHO if a server is ignoring data in the TCP buffer when it claims to 
> support PIPELINING, that server is broken and we shouldn't try to fix 
> it in the spec. but if a server complains when a client sends BDAT 
> for no valid recipients,
> I don't blame the server for that.

I suggest that if a BDAT-capable server receives BDAT, it SHOULD either 
receive and discard the data, or else send its response and drop the 
connection. Shouldn't need saying, but I guess it does not hurt.

--Arnt

From: Arnt Gulbrandsen <[email-masked]>
To: Keith Moore <[email-masked]>
Subject: Re: BDAT, RFC 3030 protocol clarification?
Cc: "Vaudreuil, Greg M (Greg)" <[email-masked]>, Matti Aarnio <[email-masked]>, [email-masked]
Date: Fri, 13 Jun 2003 10:50:24 +0200
Sender: [email-masked]

Keith Moore writes:
> my understanding was  that the client should not send BDAT commands 
> unless it had received a 2xx response from MAIL and at least one of 
> the RCPT commands.

Well, 1854 says sending DATA is okay, 3030 says BDAT and DATA are the 
same, and AFAIK nothing says that in this respect the two are 
different. IMO the successor to 3030 should say so.

> failure to do so could cause synchronization problems if the server  
> then said 5xx in response to BDAT because it wasn't going to accept 
> the message anyway.
>
> IMHO if a server is ignoring data in the TCP buffer when it claims to 
> support PIPELINING, that server is broken and we shouldn't try to fix 
> it in the spec. but if a server complains when a client sends BDAT 
> for no valid recipients,
> I don't blame the server for that.

I suggest that if a BDAT-capable server receives BDAT, it SHOULD either 
receive and discard the data, or else send its response and drop the 
connection. Shouldn't need saying, but I guess it does not hurt.

--Arnt

Date: Thu, 12 Jun 2003 22:36:11 -0400
From: Keith Moore <[email-masked]>
To: "Vaudreuil, Greg M (Greg)" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
Sender: [email-masked]

one more thing - if a client sends BDAT when the server hasn't sent
2xx to MAIL and at least one RCPT, it's perfectly reasonable for the
server to abort the TCP connection, since even if the client has
additional messages to send it's not likely to resync anyway.

I suppose a server that wanted to be kind could accept NNk bytes of BDAT
(discarding the data and returning 5xx to each) before closing the
connection.

I think the requirement to check MAIL/RCPT responses is on the client
but an implementation note for the server about the potential synchronizaiton
problem also seems appropriate.

Date: Thu, 12 Jun 2003 22:29:56 -0400
From: Keith Moore <[email-masked]>
To: "Vaudreuil, Greg M (Greg)" <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: BDAT, RFC 3030 protocol clarification?
Sender: [email-masked]

my understanding was  that the client should not send BDAT commands unless 
it had received a 2xx response from MAIL and at least one of the RCPT
commands. failure to do so could cause synchronization problems if the server 
then said 5xx in response to BDAT because it wasn't going to accept the
message anyway.

IMHO if a server is ignoring data in the TCP buffer when it claims to support
PIPELINING, that server is broken and we shouldn't try to fix it in the spec.
but if a server complains when a client sends BDAT for no valid recipients,
I don't blame the server for that.  

this isn't quite as severe as a pipeline "stop" but it does mean that the 
pipeline stalls if the client hasn't seen the response to MAIL and at least
one good response to RCPT before it wants to start sending data.

especially in these days of rampant spam, it's more important for a server to
be able to refuse traffic which it wouldn't forward/relay anyway than it is
for the client to be able to avoid a pipeline stall.

Keith

> I am preparing a revision to RFC 3030 in anticipation of draft-standard
> status.
> 
> The below reported interoperability failure seems to result from a
> difference in interpretation of the following paragraph.  This paragraph,
> and the example in section 4.2, seems to, but may not clearly state, that a
> pipelining "stop" is required after the sending of the "envelope" and the
> beginning of the "body" sending portions of the SMTP transaction.
> 
> ----Quoted from section 2 of RFC 3030 ---
> 
> After all MAIL and RCPT responses are collected and processed, the message
> is sent using a series of BDAT commands.  The BDAT command takes one
> required argument, the exact length of the data segment in octets.  The
> message data is sent immediately after the trailing <CR> <LF> of the BDAT
> command line.  Once the receiver-SMTP receives the specified number of
> octets, it will return a 250 reply code.
> 
> ------
> 
> Matti Aarnio and others have questioned the need, and in the case below, did
> not implement, the pipeline stop. The stop costs an extra round-trip, but
> eliminating it may continue interoperability problems with servers that
> assume otherwise.  
> 
> I would like opinions on how to clarify this... do we clarify that a stop is
> a suggestion when sending large messages or do we clarify that this is a
> required full stop?
> 
> I hum for clarifying the stop as mandatory as the most conservative
> interpretation and one that promotes maximum interoperability with the
> installed base.

From: "Vaudreuil, Greg M (Greg)" <[email-masked]>
To: Matti Aarnio <[email-masked]>
Cc: [email-masked]
Subject: BDAT, RFC 3030 protocol clarification?
Date: Thu, 12 Jun 2003 20:53:37 -0500
Sender: [email-masked]

I am preparing a revision to RFC 3030 in anticipation of draft-standard status.

The below reported interoperability failure seems to result from a difference in interpretation of the following paragraph.  This paragraph, and the example in section 4.2, seems to, but may not clearly state, that a pipelining "stop" is required after the sending of the "envelope" and the beginning of the "body" sending portions of the SMTP transaction.

----Quoted from section 2 of RFC 3030 ---

After all MAIL and RCPT responses are collected and processed, the message is sent using a series of BDAT commands.  The BDAT command takes one required argument, the exact length of the data segment in octets.  The message data is sent immediately after the trailing <CR> <LF> of the BDAT command line.  Once the receiver-SMTP receives the specified number of octets, it will return a 250 reply code.

------

Matti Aarnio and others have questioned the need, and in the case below, did not implement, the pipeline stop. The stop costs an extra round-trip, but eliminating it may continue interoperability problems with servers that assume otherwise.  

I would like opinions on how to clarify this... do we clarify that a stop is a suggestion when sending large messages or do we clarify that this is a required full stop?

I hum for clarifying the stop as mandatory as the most conservative interpretation and one that promotes maximum interoperability with the installed base.

Comments?

Greg V.

-----Original Message-----
From: Matti Aarnio [mailto:[email-masked]]
Sent: Monday, February 17, 2003 6:44 PM
To: [email-masked]
Subject: MS Exchange 2000 broke RFC 3030 badly...

I write at this forum, as apparently MS people read this, and
me being a non-customer to them, I have no direct access routes
to send bug reports.

I have implemented ESMTP  CHUNKING / BDAT since 1997,  and now
finally some major implementations do appear -- but sadly first
of them is broken.   The end result is that even if Microsoft
fixes things this week, bug containing versions will be running
for years...     (Shall RFC 3030 be thus considered failure due
to widely installed buggy implementation ?)

In RFC 3030 two ESMTP extensions are defined:
 - CHUNKING
 - BINARYMIME

The buggy implementation is of CHUNKING's BDAT verb processing.

Said verb runs like this:

  BDAT nnnn [LAST] CRLF
  < nnnn bytes of message header+content data with CRLF line ends >

The server shall always consume the message content associated
with BDAT verb.  (Ok, RFC 3030 text isn't explicitely clear at this,
which has lead to a number of implementation bugs, including
the one presently being discussed.)

With simultaneous PIPELINING facility/compability declared by the
remote MTA server, a smtp-client sending in optimized pipeline mode
sends MAIL FROM, one or more of RCPT TO, and also BDAT+message content
all in one TCP data push.

How   MS Exchange 2000   as deployed at Hotmail.COM now does work is:

 - If   MAIL FROM   and   RCPT TO (one or more) are received
   successfully,  BDAT will be accepted successfully.

 - If for some reason all RCPT TOs are rejected, treatment of BDAT
   is broken.  Associated data is not consumed (discarded), and
   the result is unpredictable number of "500 Unrecognized command"
   messages, as the message content ended up in SMTP command processing.

   Following UNIX bash-script shows what happens.

(echo "EHLO foo";sleep 1;
echo -e "MAIL FROM:<>\nRCPT TO:<[email-masked]>\nBDAT 9 LAST\n1\n2\n3\n";
sleep 2;
echo -e "MAIL FROM:<>\nRCPT TO:<[email-masked]>\nBDAT 9 LAST\n1\n2\n3\n";
sleep 3) | telnet mx1.hotmail.com smtp

   Doing network protocol dump, one can see that from MAIL FROM to
   BDAT content data all are sent with single TCP data push, as is
   the nature of smtp-client in fully developed PIPELINING mode.

   Here are the received responses:

220 mc1-f8.law16.hotmail.com Microsoft ESMTP MAIL Service, Version: 5.0.2195.5600 ready at  Mon, 17 Feb 2003 16:23:42 -0800
   <<-  EHLO foo
250-mc1-f8.law16.hotmail.com (02.01.00.0007) Hello [62.240.94.4]
250-SIZE 4278190
250-PIPELINING
250-8bitmime
250-BINARYMIME
250-CHUNKING
250-VRFY
250-AUTH LOGIN
250-AUTH=LOGIN
250-X-HMAUTH
250 OK
   <<- MAIL FROM:<>
250 <>....Sender OK
   <<- RCPT TO:<[email-masked]>
550 Requested action not taken: mailbox unavailable
   <<- BDAT 9 LAST (+ data)
503 Need Rcpt command.
500 Unrecognized command
500 Unrecognized command
500 Unrecognized command
   <<- MAIL FROM:<>
503 Sender already specified
   <<- RCPT TO:<[email-masked]>
550 Requested action not taken: mailbox unavailable
   <<- BDAT 9 LAST (+ data)
503 Need Rcpt command.
500 Unrecognized command
500 Unrecognized command
500 Unrecognized command

   Fully pipelined smtp client code does not expect
   any of those "500 Unrecognized command" lines, and
   goes out of protocol sync.

--
/Matti Aarnio   <[email-masked]>