Subject: Re: per-recipient response twisters
From: Claus Assmann <[email-masked]>
To: [email-masked]
Date: Mon, 26 Feb 2007 20:20:18 -0800
List-Id: <ietf-smtp.imc.org>

On Mon, Feb 26, 2007, Eric A. Hall wrote:

> Consider a scenario where the post-data response block looks like this:

>   353 detailed responses follow
>   250 [email-masked] may accept the message
>   550 fighter@example will not accept the message
>   421 my disk disappeared... please try again later
....
> Actually it seems that SMTP clients should generally treat the
> per-recipient response codes as deferred RCPT TO responses, and to
> essentially reuse the existing logic for failures in particular. This has

Yes, the logic must be identical to the handling of the case without
"deferred RCPT responses". Anything else would cause an additional
implementation hurdle (and it would probably be hard to explain
why you want different semantics).

> the benefit of simplifying implementation logic a bit and seems to cover
> most of the corner cases. Does anyone disagree with this?

Not me.

Date: Mon, 26 Feb 2007 17:03:47 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: per-recipient response twisters
Sender: [email-masked]

I have been working on rewriting the deferral draft to accommodate some of
our discussions and have gotten into some hairy corners.

Consider a scenario where the post-data response block looks like this:

  353 detailed responses follow
  250 [email-masked] may accept the message
  550 fighter@example will not accept the message
  421 my disk disappeared... please try again later

The final response code determines whether or not the message is
accepted--if so (2xx) then the client releases the message from its queue
and generates whatever notifications are needed for failed recipient, but
if the final response indicates that transfer has failed entirely, the
client has to requeue the message according to traditional rules.

So, what does that mean for [email-masked] and the 550 response? Does
the client requeue the message, and discard the recipient list? Or does it
go ahead and process the failed recipient, and then requeue the message
absent the failures?

This same problem also appears in regular SMTP somewhat, in that a RCPT TO
address may be rejected during envelope exchange, and the entire message
being rejected. Although it does not appear to be written in stone
anywhere, it seems that the correct course of action is to process the
failed recipients as soon as you know about them, and then requeue the
message absent the failed recipients.

Actually it seems that SMTP clients should generally treat the
per-recipient response codes as deferred RCPT TO responses, and to
essentially reuse the existing logic for failures in particular. This has
the benefit of simplifying implementation logic a bit and seems to cover
most of the corner cases. Does anyone disagree with this?

More generally, it also seems that this whole mechanism is a deferred RCPT
response mechanism, rather than a per-recipient data response mechanism. I
will likely change the extension acronym back to DRR accordingly.

It also seems that this is why I had the 352 response code in the original
RCPT handling--the model is deferring a response to RCPT, and 352 says
"wait for an answer" as appropriate. I'm not going to put it back in, but
this answers the question why it was in there in the first place.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 15 Feb 2007 12:09:24 -0500
From: "Eric A. Hall" <[email-masked]>
To: Tony Finch <[email-masked]>
CC: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/15/2007 4:04 AM, Tony Finch wrote:
> On Wed, 14 Feb 2007, Eric A. Hall wrote:
>> If the write fails then the client wouldn't have received it.
> 
> And the server is supposed to drop the message *after* it has committed to
> handling it?

That's what duplicate avoidance means--dropping duplicates.

But maybe we are making more of an issue out of this than it really is.

If duplicate avoidance really is "a problem" and not an annoyance, then
some kind of checkpoint mechanism with deliberate client acknowledgments
to the server acknowledgment would be the way to go. It's heavy but
appropriate for something that is really and truly a problem that needs to
be solved.

If it's just an annoyance, then something as simple as "detect errors" as
suggested is probably overkill.

Either way, it's a problem for SMTP and not a problem for this extension,
so it ought to be dealt with separately.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 15 Feb 2007 09:04:44 +0000
From: Tony Finch <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Tony Finch <[email-masked]>, [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On Wed, 14 Feb 2007, Eric A. Hall wrote:
>
> If the write fails then the client wouldn't have received it.

And the server is supposed to drop the message *after* it has committed to
handling it? This worries me...

Requiring the server to send individual replies separately still doesn't
improve its ability to detect errors. It can do the same thing whether or
not it has pipelined output after a transaction's final reply. (I admit
it's probably simpler to implement without pipelining.)

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOGGER: NORTHWESTERLY BACKING SOUTHERLY 4 OR 5, INCREASING 6 OR 7. MODERATE,
INCREASING ROUGH LATER. FAIR. GOOD.

Date: Wed, 14 Feb 2007 18:23:29 -0500
From: "Eric A. Hall" <[email-masked]>
To: Tony Finch <[email-masked]>
CC: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/14/2007 6:21 PM, Tony Finch wrote:
> On Wed, 14 Feb 2007, Eric A. Hall wrote:
>> Generally speaking though, I think the bis-bis update to SMTP
>> probably ought to implement something like "final reply ought to have
>> its own write() if possible" as a way for receivers to detect that
>> client has lost connection to the server.
> 
> How is that supposed to help? If the client receives the reply then
> goes idle, the server will hand around until its SMTP timeout ends.

If the write fails then the client wouldn't have received it.

> If the client has dropped off the net then the server's write will be
> buffered in the kernel and TCP retries will continue until the server's
> SMTP timeout ends.

Yeah this and other things are problematic. OTOH it's better than the
nothing that we have now.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 14 Feb 2007 23:21:00 +0000
From: Tony Finch <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On Wed, 14 Feb 2007, Eric A. Hall wrote:
>
> Generally speaking though, I think the bis-bis update to SMTP probably
> ought to implement something like "final reply ought to have its own
> write() if possible" as a way for receivers to detect that client has lost
> connection to the server.

How is that supposed to help? If the client receives the reply then goes
idle, the server will hand around until its SMTP timeout ends. If the
client has dropped off the net then the server's write will be buffered in
the kernel and TCP retries will continue until the server's SMTP timeout
ends. The server won't be able to tell the difference between the two
scenarios because it can't find out how much data is unacknowledged. The
same applies if the client has pipelined the next message's envelope and
the server pipelines all its replies or splits them as you suggest: in the
event of failure the TCP stack will buffer and amalgamate the replies.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FORTIES: WESTERLY BACKING SOUTHERLY 5 TO 7, OCCASIONALLY GALE 8. ROUGH OR VERY
ROUGH. RAIN AT FIRST. MODERATE OR GOOD.

To: [email-masked]
Cc: [email-masked]
Subject: Re: When to acknowledge messages for per-recipient data reply / deferred rcpt reply
From: [email-masked]
Date: Wed, 14 Feb 2007 16:15:20 -0500
Sender: [email-masked]

--==_Exmh_1171487720_31555P
Content-Type: text/plain; charset=us-ascii

On Tue, 13 Feb 2007 10:57:13 CST, Grant Taylor said:

> I think that (valid) recipients should be acknowledged as soon as 
> (reasonably) possible.  I'm thinking about the number of sending systems 
> that I've had to deal with that have had failed connections part way 
> through transmission of messages.  If a recipient is acknowledged as 
> soon as (reasonably) possible rather than waiting until all recipients 
> have been processed it is more likely that sending systems will receive 
> the acknowledgment prior to failed transmission, thus having fewer 
> recipients that it needs to re-send the message for.

Do you have any firm statistics regarding what percent of "failed" were due
to actual TCP-level communications failures due to backhoe fade and dead
routers, against what percent was due to a time-out condition while the
receiving host went off and did things like virus/spam scan?

I'd expect that any production-scale system that actually allows for different
results for different recipients will want to make exactly 1 pass through the
body, accumulating N results in parallel, rather than make N passes through the
message.  As a result, all the replies will likely become ready fairly close
temporally to each other.  So there's no real expectation that for recipients
A, B, C, D we'll see A's reply much sooner than the others, unless A's
default policy is "accept with no checking at all" and B-D actually do some
checking.

I'm not sure I'd want to write the specs if we do the obvious alternative
of returning results out-of-order with respect to the ordering of RCPT TOs
originally sent - so if D is the "no checking" case, we return D A B C.

--==_Exmh_1171487720_31555P
Content-Type: application/pgp-signature

--==_Exmh_1171487720_31555P--

Date: Wed, 14 Feb 2007 13:10:20 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/12/2007 12:26 AM, Claus Assmann wrote:
> So the current protocol suggestion looks like this:
> 
> * Client initiates by sending MAIL with an extension.
> * RCPT returns normal reply codes (no 3yz)
> * DATA and the last BDAT return (in order):
>   either: a single {2,4,5}yz reply code to indicate the
>       overall transaction status,
>   or: 353 to indicate the start of server responses,
>       followed by:
>      - one reply code per recipient which got a 2yz reply.
>      - one end of mail reply code (as in the current ESMTP model,
>        which is the same as the "final" reply code in
>        draft-hall-deferrals).
> 
> Eric: could you update your draft to reflect this status?

Yeah, will do this coming weekend probably. I lost the source document and
was trying to get by with small edits to the text file, but I'll need to
rewrite to accommodate all the changes (minor and major).

> (and please change the name so it's more specific, i.e., it should
> have "RCPT" somewhere in it's name and hopefully a nice abbreviation,
> e.g., DRR, RRAD, ...)

I'll probably name the extension PRDR

> On Tue, Feb 06, 2007, Eric A. Hall wrote:

>> 1) the hard time limits
> 
> I would increase the time for the first reply after end of mail
> data from 1 minute to at least 5 minutes or even 10 minutes (to
> match the current (RFC2821) value).

okay

>> 2) duplicate avoidance technology does not seem to have been resolved to
>> everyone's satisfaction yet
> 
> Checkpointing is "out" as it has (security) problems between untrusted
> parties. So I think we "just" live with what we have?

I'll word some of it more strongly

I'm not going to introduce any new rules or behaviors over existing SMTP.
The best place to fix those problems is in SMTP generally, and I don't
want to risk coming up with a slightly different set of rules.

Generally speaking though, I think the bis-bis update to SMTP probably
ought to implement something like "final reply ought to have its own
write() if possible" as a way for receivers to detect that client has lost
connection to the server.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 14 Feb 2007 17:46:44 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: checkpointing (was: another per-recipient data reply proposal)
Sender: [email-masked]

On Wed, 14 Feb 2007, Claus Assmann wrote:
>
> As long as checkpointing is not a requirement for "per-recipient
> data reply" but just mentioned as one possible solution to the
> problem of duplicated mails, I'm fine with it. Requiring checkpointing
> would keep people from implementing "per-recipient data reply" due
> to the complexity of a possible checkpointing implementation.

Yes, I agree.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
CROMARTY FORTH TYNE: WESTERLY BACKING SOUTHERLY 5 OR 6, INCREASING 7. SLIGHT
OR MODERATE INCREASING ROUGH. MAINLY FAIR. GOOD.

Date: Wed, 14 Feb 2007 09:16:22 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: checkpointing (was: another per-recipient data reply proposal)
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Mon, Feb 12, 2007, Tony Finch wrote:

> On Sun, 11 Feb 2007, Claus Assmann wrote:

> > Checkpointing is "out" as it has (security) problems between untrusted
> > parties.

> I don't think it has significant problems if you only use it for duplicate
> elimination, i.e. without data checkpointing. Do you have any omissions to

As long as checkpointing is not a requirement for "per-recipient
data reply" but just mentioned as one possible solution to the
problem of duplicated mails, I'm fine with it. Requiring checkpointing
would keep people from implementing "per-recipient data reply" due
to the complexity of a possible checkpointing implementation.

The RFC should simply state that the problem mentioned in RFC 1047
requires that senders and clients at least properly obey the timeouts.

Date: Tue, 13 Feb 2007 10:57:13 -0600
From: Grant Taylor <[email-masked]>
Reply-To: [email-masked]
To: [email-masked]
Subject: When to acknowledge messages for per-recipient data reply / deferred rcpt reply
Sender: [email-masked]

After reading many archived messages on per-recipient data reply / 
deferred rcpt reply I have come up with a thought regarding when to 
acknowledge recipients / messages.

I think that (valid) recipients should be acknowledged as soon as 
(reasonably) possible.  I'm thinking about the number of sending systems 
that I've had to deal with that have had failed connections part way 
through transmission of messages.  If a recipient is acknowledged as 
soon as (reasonably) possible rather than waiting until all recipients 
have been processed it is more likely that sending systems will receive 
the acknowledgment prior to failed transmission, thus having fewer 
recipients that it needs to re-send the message for.  The only reason 
that I can come up with as to why not do this is for a performance / 
efficiency reason(s) out weighing the desire to have a robust protocol 
that will still partially work for some recipients in the event of a 
communications failure.

Grant. . . .

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 13 Feb 2007 10:34:37 -0500
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On Sun, Feb 11, 2007 at 09:26:52PM -0800, Claus Assmann wrote:
> 
> Eric: could you update your draft to reflect this status?
> (and please change the name so it's more specific, i.e., it should
> have "RCPT" somewhere in it's name and hopefully a nice abbreviation,
> e.g., DRR, RRAD, ...)

I guess if we're on about the name, it would be nice if it had "-smtp-"
up front as well.  (So that when it is announced on the internet-drafts
list, it stands out for anyone looking for that.)

Yours,
-mm-

Date: Mon, 12 Feb 2007 13:32:36 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On Sun, 11 Feb 2007, Claus Assmann wrote:
>
> Checkpointing is "out" as it has (security) problems between untrusted
> parties.

I don't think it has significant problems if you only use it for duplicate
elimination, i.e. without data checkpointing. Do you have any omissions to
point out in my draft?
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-rfc1845bis.html

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SHANNON: NORTHWEST 7 TO SEVERE GALE 9, BACKING SOUTH 5 TO 7. HIGH, DECREASING
VERY ROUGH FOR A TIME. RAIN OR SHOWERS. MODERATE OR GOOD.

Date: Sun, 11 Feb 2007 21:26:52 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

So the current protocol suggestion looks like this:

* Client initiates by sending MAIL with an extension.
* RCPT returns normal reply codes (no 3yz)
* DATA and the last BDAT return (in order):
  either: a single {2,4,5}yz reply code to indicate the
      overall transaction status,
  or: 353 to indicate the start of server responses,
      followed by:
     - one reply code per recipient which got a 2yz reply.
     - one end of mail reply code (as in the current ESMTP model,
       which is the same as the "final" reply code in
       draft-hall-deferrals).

Eric: could you update your draft to reflect this status?
(and please change the name so it's more specific, i.e., it should
have "RCPT" somewhere in it's name and hopefully a nice abbreviation,
e.g., DRR, RRAD, ...)

On Tue, Feb 06, 2007, Eric A. Hall wrote:

[about open problems]

> 1) the hard time limits

I would increase the time for the first reply after end of mail
data from 1 minute to at least 5 minutes or even 10 minutes (to
match the current (RFC2821) value). Otherwise there is little chance
to get the single {2,4,5}yz reply code as content analysis probably
takes longer in many cases.

> 2) duplicate avoidance technology does not seem to have been resolved to
> everyone's satisfaction yet

Checkpointing is "out" as it has (security) problems between untrusted
parties. So I think we "just" live with what we have?

Date: Wed, 7 Feb 2007 10:28:29 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Wed, Feb 07, 2007, Tony Finch wrote:

[perform a single "scan" of the message]
> [[...]] The model would then be that the MTA
> scans, decides whether to accept for each recipient, commits if necessary,
> then returns all the replies at once. This is why I had been arguing for
> using exactly the LMTP style of final reply.

If you have "all the replies" then you can also generate a "final
reply" (its the minimum of all reply codes, I do that currently in
my test sink) and transmit it together with the others. This makes
error handling in the server _much_ simpler.

[vs. perform individual "scans" of the message per recipient]
> [[...]] If you are going to do time-consuming things after data,
> especially in a way that increases with envelope size, then LMTP becomes
> painful: you want to trickle out the replies as they become available, but
> you don't want to have to repeatedly commit the same message. As Claus
> pointed out, Eric's draft allows a single commit point after the
> per-recipient replies and before an overall final reply, which fixes the
> problem.

This means the second model (final reply code) is more general and
doesn't have the drawbacks of the first one (LMTP). Moreover, it
is trivial to "embed" the LMTP model into the second by adding the
final reply code (see above). So I don't see a reason to go with
the LMTP model.

Date: Wed, 07 Feb 2007 10:11:33 -0500
From: "Eric A. Hall" <[email-masked]>
To: Tony Finch <[email-masked]>
CC: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/7/2007 5:06 AM, Tony Finch wrote:

> There is a question related to a security consideration: if you do
> per-recipient content scanning instead of per-message content scanning
> then it's much easier for an attacker to use up your CPU.

This is a valid concern. OTOH, systems that already perform per-user
scanning by limiting SMTP transfers to one recipient already have to deal
with the same relative load burden, so it's not entirely new.

In the general sense, I think it's an admin problem more than anything, in
that admins need to know how to manage their load. "If per-recipient
scanning breaks your mail, don't do per-recipient scanning" is the same
sort of sentiment as "if database driven web content breaks your web
server, don't do database driven web content"

As far as that goes, actually, people who are already doing after-dot
content scanning (either globally or one-recipient) already know that it
takes some planning and resource management to balance volume with load.
"hmm, should I buy another server, or eliminate some filters?" These are
admin decisions, and to that extent, this is really the wrong place to ask
if it is worth it, since it clearly is worth it to so many already.

Having said all that, MTA developers that expose this feature directly
will also have options that they can use to limit how much trouble their
users get into. EG, an MTA can limit certain types of scanning to global
filters (ie, only doing ~Bayes and string-matching filters in the
per-recipient loop and do everything else globally). MTAs can also adopt
certain kinds of attack-avoidance features, such as enforcing limited
RCPTs per connect. The extent to which an MTA developer my want to provide
this kind of hand-holding (or not) is out of scope naturally, but we
should ack that it's an option that exists for many of them.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Wed, 7 Feb 2007 10:06:31 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On Mon, 5 Feb 2007, Claus Assmann wrote:
>
> As I wrote before: it would be really nice to get some feedback
> from people who are familiar with MTAs to comment about the (code)
> complexity etc.

I've been thinking a little about how to add this to Exim.

There is a question related to a security consideration: if you do
per-recipient content scanning instead of per-message content scanning
then it's much easier for an attacker to use up your CPU. Because of that
I had been thinking that MTAs would continue to scan messages only once,
as at present, and that it would take negligible effort to make the
per-recipient accept/deny decisions. The model would then be that the MTA
scans, decides whether to accept for each recipient, commits if necessary,
then returns all the replies at once. This is why I had been arguing for
using exactly the LMTP style of final reply.

However it has become clear from the discussion that people are
considering per-recipient scanning. I'm not entirely convinced of the
wisdom of this, but Exim's experience of feature creep tells me that
postmasters will want it. The biggest use-cases I can see are
per-recipient statistical classifiers and SpamAssassin's per-user
configurability. If you are going to do time-consuming things after data,
especially in a way that increases with envelope size, then LMTP becomes
painful: you want to trickle out the replies as they become available, but
you don't want to have to repeatedly commit the same message. As Claus
pointed out, Eric's draft allows a single commit point after the
per-recipient replies and before an overall final reply, which fixes the
problem.

What do others think about the trade-off between the CPU usage security
issue and the desire for per-recipient scanning?

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
LUNDY FASTNET: NORTHEAST VEERING EAST 4 INCREASING 5 TO 7. SLIGHT OR MODERATE
BECOMING ROUGH OR VERY ROUGH. RAIN AT TIMES, OCCASIONALLY SLEET LATER.
MODERATE OR GOOD.

Date: Tue, 6 Feb 2007 21:12:33 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: show of hands: which per-recipient data reply proposal?
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--X1bOJ3K7DJ5YkBrT
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-02-06 10:49:59 -0800, Claus Assmann wrote:
> * Client initiates by sending MAIL with an extension.
> * RCPT returns normal reply codes (no 3yz)
> * DATA and the last BDAT return (in order):
>   1. 353 to indicate the start of server responses
>     1.a. possible optimization: a single {2,4,5}yz reply code
>          to indicate the overall transaction status in
>          which case the next two steps are skipped.
>   2. one reply code per recipient which got a 2yz reply.
>   3. one end of mail reply code (as in the current ESMTP model,
>     which is the same as the "final" reply code in
>     draft-hall-deferrals).
>=20
>=20
> So we still have to decide between:
>=20
> (I) get rid of step 1.: 353 (and increase the timeout to the
>     first deferred RCPT response),
> (II) or use 353 with the optimization 1.a.
>=20
>=20
> Here's my vote: (II).

(II)++.

I notice that the optimization is just normal SMTP.

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--X1bOJ3K7DJ5YkBrT
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--X1bOJ3K7DJ5YkBrT--

Date: Tue, 6 Feb 2007 10:49:59 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: show of hands: which per-recipient data reply proposal?
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

[Resend latest version to hopefully get some "rough consensus" via
a simple "vote"]

* Client initiates by sending MAIL with an extension.
* RCPT returns normal reply codes (no 3yz)
* DATA and the last BDAT return (in order):
  1. 353 to indicate the start of server responses
    1.a. possible optimization: a single {2,4,5}yz reply code
         to indicate the overall transaction status in
         which case the next two steps are skipped.
  2. one reply code per recipient which got a 2yz reply.
  3. one end of mail reply code (as in the current ESMTP model,
    which is the same as the "final" reply code in
    draft-hall-deferrals).

So we still have to decide between:

(I) get rid of step 1.: 353 (and increase the timeout to the
    first deferred RCPT response),
(II) or use 353 with the optimization 1.a.

Here's my vote: (II).

Date: Tue, 06 Feb 2007 13:32:48 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/6/2007 1:00 PM, Claus Assmann wrote:

> Ok, then we are now at the point where we "only" argue about the
> "first reply after data"? That is, your current proposal looks like
> this:
> 
> * Client initiates by sending MAIL with an extension.
> * RCPT returns normal reply codes (no 3yz)
> * DATA and the last BDAT return (in order):
>   1. 353 to indicate the start of server responses
>     1.a. possible optimization: a single {2,4,5}yz reply code
>          to indicate the overall transaction status.
>   2. one reply code per recipient which got a 2yz reply.
>   3. one end of mail reply code (as in the current ESMTP model,
>     which is the same as the "final" reply code in
>     draft-hall-deferrals).

Yeah

> So we still have to decide between:
> 
> (I) get rid of step 1.: 353 (and increase the timeout to the
>     first deferred RCPT response),
> (II) or use 353 with the optimization 1.a.

and keep it as MAY. If we agree on that then the two issues that seem to
still be open:

1) the hard time limits

2) duplicate avoidance technology does not seem to have been resolved to
everyone's satisfaction yet

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Tue, 6 Feb 2007 10:00:31 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Tue, Feb 06, 2007, Eric A. Hall wrote:

> > Does your pseudo code look correct to you?
> 
> I didn't run it through lint, sorry

Lint wouldn't have helped :-)

However, this little bug demonstrated my point about the complexity.

> Section 6.2 says the following:
> 
> |     If the message is accepted or refused by every recipient, the 353
> |     response code and the subsequent responses MAY be substituted with
> |     a single response code which reflects the global outcome.
> 
> although that could probably be clarified. I think the particulars would
> be up to the implementation. Here are some scenarios:

Keep it simple unless there is a good reason for complexity.  If
we want to keep this optimization, then MAY should be changed to
SHOULD (not MUST as the timeout is too short to always make a
decision).

> I thought we had eliminated deferred RCPT so that only the 25x recipients
> were processed. If an implementation wants to manage some of those

Ok, then we are now at the point where we "only" argue about the
"first reply after data"? That is, your current proposal looks like
this:

* Client initiates by sending MAIL with an extension.
* RCPT returns normal reply codes (no 3yz)
* DATA and the last BDAT return (in order):
  1. 353 to indicate the start of server responses
    1.a. possible optimization: a single {2,4,5}yz reply code
         to indicate the overall transaction status.
  2. one reply code per recipient which got a 2yz reply.
  3. one end of mail reply code (as in the current ESMTP model,
    which is the same as the "final" reply code in
    draft-hall-deferrals).

So we still have to decide between:

(I) get rid of step 1.: 353 (and increase the timeout to the
    first deferred RCPT response),
(II) or use 353 with the optimization 1.a.

(the other alternative:
(III) use 353 without the optimization 1.a.
doesn't look very useful to me).

Date: Tue, 06 Feb 2007 12:31:54 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/5/2007 10:52 PM, Claus Assmann wrote:

>>   begin_inspection {
>>      if global_inspection == 1 {
>>         if global_return == 1 {
>>            print 550;
>>            exit;
>>         }

        ; tell the client we are starting per-recipient filters

        print 353;

>>      foreach $recipient {
>>         if recipient_return == 1 {
>>            print 550;
>>         } else {
>>            print 250;
>>         }
>>      print 250;
>>   }
> 
> [I removed the comments and empty lines...]
> 
> Does your pseudo code look correct to you?

I didn't run it through lint, sorry

> It misses the "first response after end of data" (353) before the
> "foreach" part...  or was this code supposed to implement some other
> model?

That was supposed to follow the comment line that I've reinserted

> Moreover, it misses some optimizations (unless I misunderstood
> your draft):
> 1. if all RCPTs return 250: send 250
> 2. if all RCPTs return 550: send 550
> 3. otherwise: send 353
>      and then the individual RCPT replies
>      and then 250.

Section 6.2 says the following:

|     If the message is accepted or refused by every recipient, the 353
|     response code and the subsequent responses MAY be substituted with
|     a single response code which reflects the global outcome.

although that could probably be clarified. I think the particulars would
be up to the implementation. Here are some scenarios:

1) MTA implements some kind of global filtering such as virus scanning,
and issues 550 instead of per-recipient codes.

2) MTA knows that some recipients are exempt from filtering (eg, abuse@
needs to get viruses, in case spewing users are being reported), and if
the MTA knows that all of the recipients are in the safe-list then return
a 250.

3) MTA temporarily stores individual recipient response codes in array and
compresses them into a single response (assuming that a timer does not
expire first).

There are other possibilities here, but I don't want to argue about what I
consider implementation details. The point is that it is optional, and to
the extent that it is optional then implementations can do what they like
to use it or not. I can improve the wording if you think it's too vague.

> PS: there are more complexities caused by your proposal, e.g.,
> foreach $recipient
> isn't as simple as it looks: you need a list of "ok" and "deferred"
> RCPTs and for the latter you need "new status code" too.

I thought we had eliminated deferred RCPT so that only the 25x recipients
were processed. If an implementation wants to manage some of those
differently (such as with checking the all-mail list if it's available)
then that's up to the implementation.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Mon, 5 Feb 2007 19:52:49 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Mon, Feb 05, 2007, Eric A. Hall wrote:

[first response after end of data]

> It provides a start point for the client's response timer, thereby
> preventing the client from timing out sooner than it should.

The client starts its timer after the last I/O operation it did.
Whether the next input is the "first response after end of data"
or the first RCPT reply doesn't matter as long as you properly
specify the maximum timeouts.

> I'd like to know what is difficult about the 353 response. If part of it
> is too difficult for the value it provides we can look at stripping out
> those parts of it. Some pseudo-code as I see it:
> 
>   begin_inspection {
>      if global_inspection == 1 {
>         if global_return == 1 {
>            print 550;
>            exit;
>         }
>      foreach $recipient {
>         if recipient_return == 1 {
>            print 550;
>         } else {
>            print 250;
>         }
>      print 250;
>   }

[I removed the comments and empty lines...]

Does your pseudo code look correct to you?

It misses the "first response after end of data" (353) before the
"foreach" part...  or was this code supposed to implement some other
model?

Moreover, it misses some optimizations (unless I misunderstood
your draft):
1. if all RCPTs return 250: send 250
2. if all RCPTs return 550: send 550
3. otherwise: send 353
     and then the individual RCPT replies
     and then 250.

As I wrote before: it would be really nice to get some feedback
from people who are familiar with MTAs to comment about the (code)
complexity etc.

PS: there are more complexities caused by your proposal, e.g.,
foreach $recipient
isn't as simple as it looks: you need a list of "ok" and "deferred"
RCPTs and for the latter you need "new status code" too. For the
LMTP model it's much simpler: the MTA keeps only the "ok" RCPTs (as
it has to do anyway) and it can override those values after the
data phase (as all of them were "ok", no need to distinguish between
"ok" and "deferred"). This is just one example of the "subtle"
problems introduced into existing code by the more complex proposal.

Date: Mon, 05 Feb 2007 10:14:31 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/4/2007 10:50 PM, Claus Assmann wrote:
> On Sun, Feb 04, 2007, Eric A. Hall wrote:

>> See the discussion on reliability timers earlier--a start code is going to
>> be important. Also see the last post by me in re global filters--this is
> 
> Your draft requires a first response after 1 minute and then each
> RCPT command within 10 minutes. I don't see why that first response
> is important wrt timeouts, but I'll look at that discussion again.

It provides a start point for the client's response timer, thereby
preventing the client from timing out sooner than it should.

>> the right place to issue such a response. Frank's is also correct that
>> there is some efficiency and reliability if we can avoid enumerating long
>> lists of response codes. For all those reasons I'd like to ask you to try
>> using it and see if it's not too hard.

> I understand that your proposal is the most complete and offers all
> kind of optimizations. As a consequence of that it also the most
> complex and hence most complicated to implement. As I wrote before:
> that may prevent other MTA authors from implementing it and it may
> introduce implementation errors. Remember: it is _S_imple Mail
> Transport Protocol :-)

protocols should be as simple as possible, but no simpler :)

I'd like to know what is difficult about the 353 response. If part of it
is too difficult for the value it provides we can look at stripping out
those parts of it. Some pseudo-code as I see it:

  begin_inspection {

     ; are global filters enabled?

     if global_inspection == 1 {

        ; check for viruses, etc

        if global_return == 1 {

           ; found a problem

           print 550;
           exit;
        }

     ; tell the client we are starting per-recipient filters

     foreach $recipient {

        ; check the content against recipient rules

        if recipient_return == 1 {

           ; found a problem

           print 550;

        } else {

           ; no problem

           print 250;

        }

     ; accept responsibility for the message

     print 250;

  }

That doesn't include the timers or the detailed operations of course. Are
the timers the problem? Is the process too hard to integrate with existing
functions and operations?

I'm all for simplicity here but there is so much value provided by the
head-of-list response code that I really feel that it needs to be kept.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Mon, 5 Feb 2007 08:25:35 +0000
From: Tony Finch <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Tony Finch <[email-masked]>, [email-masked]
Subject: Re: Fwd: I-D ACTION:draft-hall-deferrals-00.txt
Sender: [email-masked]

On Sat, 3 Feb 2007, Eric A. Hall wrote:
>
> A robust mechanism only requires the server to know that the client
> received the entire list of responses: if so then the client is not at
> obvious risk of resending; if not the server can refuse to accept
> responsibility on the grounds that the session was lost. So all we need is
> a way for the client to send SOME KIND OF DATA, after the server has sent
> the full list.

That just moves the problem; it doesn't solve it. With your suggestion, if
the client thinks it sent the next command but the server didn't receive
it then the message gets lost or duplicated. Remember that TCP APIs
generally don't allow an endpoint to discover on connection failure which
of the data that was passed to the stack was acked by the other end. TCP
is not reliable in the event of loss of connectivity.

> My current design is somewhat thinner than that--if the server is able to
> write() the final response code without error, then it should be safe to
> assume that the client received the full list as well as the final
> response and the mail belongs to the server now.

Absolutely not! This is exactly the RFC 1047 problem!

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOLE: NORTHEAST VEERING SOUTHEAST 3 OR 4. MODERATE OR ROUGH. FAIR. MODERATE OR
GOOD.

Date: Sun, 4 Feb 2007 19:50:14 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Sun, Feb 04, 2007, Eric A. Hall wrote:

> This is basically the DEFERRALS model with two changes. One is that you've
> dropped the 352 response to RCPT, which I'm not married to. The other

Initially I had that listed as an optimization, but then dropped
it from the mail as I didn't want to complicate matters.

> change is that you've dropped the 353 response code to signal the start of
> deferred RCPT responses, which is a bit of a problem.

> See the discussion on reliability timers earlier--a start code is going to
> be important. Also see the last post by me in re global filters--this is

Your draft requires a first response after 1 minute and then each
RCPT command within 10 minutes. I don't see why that first response
is important wrt timeouts, but I'll look at that discussion again.

Anyway, we could write an additional RFC that defines a generic
"stay tuned" reply code[*]: if the server isn't done with the command
yet but the timeout is almost reached, it can send a "stay tuned"
reply code telling the client to reset the timeout (obviously there
must be some limit to the overall timeout, e.g., 10 times the
default). The implementation of that RFC could be made a requirement
for "DRR".

> the right place to issue such a response. Frank's is also correct that
> there is some efficiency and reliability if we can avoid enumerating long
> lists of response codes. For all those reasons I'd like to ask you to try
> using it and see if it's not too hard.

I implemented your draft first (didn't I write that before?), and
it required significant code changes because of all the "optimizations".
Then I tried the LMTP model and the only (but significant) problem
I ran into is the missing ("final") reply to the data which makes
error handling more complicated than your proposal. That's why I
suggested a "compromise" to get the best (IMHO) from both models.

I understand that your proposal is the most complete and offers all
kind of optimizations. As a consequence of that it also the most
complex and hence most complicated to implement. As I wrote before:
that may prevent other MTA authors from implementing it and it may
introduce implementation errors. Remember: it is _S_imple Mail
Transport Protocol :-)

As I wrote before: maybe others can look at the implementation
effort in other MTAs for the various proposals and give some statement
wrt complexity and error handling?

Footnote:[*]: that's what we implemented in milter to deal with
this kind of timeouts during processing of the mail body.

Date: Sun, 04 Feb 2007 22:05:36 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: another per-recipient data reply proposal
Sender: [email-masked]

On 2/4/2007 8:07 PM, Claus Assmann wrote:
> Here's another proposal based on my implementation experiences:
> 
> * Client initiates by sending MAIL with an extension.
> * RCPT returns normal reply codes.
> * DATA and the last BDAT return (in order):
>   - one reply code per recipient which got a 2yz reply.
>   - one end of mail reply code (as in the current ESMTP model,
>     which is the same as the "final" reply code in
>     draft-hall-deferrals).
> 
> This is the approach the causes the least changes to existing code
> (at least in the MTAs that I maintain) and does not introduce complex
> error handling. Maybe others can take a look at exim, postfix,
> qmail, etc?

This is basically the DEFERRALS model with two changes. One is that you've
dropped the 352 response to RCPT, which I'm not married to. The other
change is that you've dropped the 353 response code to signal the start of
deferred RCPT responses, which is a bit of a problem.

See the discussion on reliability timers earlier--a start code is going to
be important. Also see the last post by me in re global filters--this is
the right place to issue such a response. Frank's is also correct that
there is some efficiency and reliability if we can avoid enumerating long
lists of response codes. For all those reasons I'd like to ask you to try
using it and see if it's not too hard.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sun, 04 Feb 2007 21:54:36 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: Overview of per-recipient data reply proposals
Sender: [email-masked]

On 2/4/2007 6:01 PM, Peter J. Holzer wrote:
> On 2007-02-04 16:11:51 -0500, Eric A. Hall wrote:

>> Many of us are using tools that inspect the message body before we decide
>> to bother with delivery, so this is an important filter.
> 
> I'm not sure I understand. All of the proposals allow content-inspection
> according to per-recipient rules before delivery.

There are two points where content inspection currently occurs in the
wild: one is inside the MTA, and the other is per-recipient analysis that
is usually handled during delivery agent handling (with DSNs being
generated on failure).

Models that only allow for content inspection in the delivery function
basically sacrifice the global inspection option. Yes it's possible to use
global rules and require the delivery process to fail every recipient, but
that's not very subtle nor is it particularly attractive.

> point you make is that the deferral draft doesn't allow delivery to
> start before the final reply, while with the LMTP model the message can
> be delivered to a recipient as soon as content scanning for that
> particular recipient is done.

Well yeah but allowing the data to be analyzed before the delivery
function is called obviates the need for that entirely.

Consider a spam-trap address that is third in the list of recipients--if
you only allow analysis at the delivery stage then it's basically too late
to do anything with it. Same with viruses. By combining delivery with
inspection you've basically said that we need to confirm successful
processing for each user individually, which is a huge waste.

Okay. This is handled in the DEFERRALS draft with the ability to simply
reject the message outright (in lieu of the "here come the deferred RCPT
responses" code), prior to delivery function being called, and without
enumerating each of the recipients.

|  6.2.    The 353 Response Code
|
|    If the message is accepted or refused by every recipient, the 353
|    response code and the subsequent responses MAY be substituted with
|    a single response code which reflects the global outcome. For
|    example, if all of the recipients accepted the email message, then
|    a single 250 response code MAY be returned in lieu of the 353
|    response code, the per-recipient response codes, and the final
|    response code. See section 6.4 for considerations about these
|    final response codes.

I don't know about the other proposals (simply surfacing LMTP would not,
but I haven't studied the others), and I think it would be a useful
data-point for comparison purposes.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: another per-recipient data reply proposal
Date:  Mon, 05 Feb 2007 03:30:51 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 21
Sender: [email-masked]

Claus Assmann wrote:

> Here's another proposal based on my implementation experiences:

> * Client initiates by sending MAIL with an extension.
> * RCPT returns normal reply codes.
> * DATA and the last BDAT return (in order):
>   - one reply code per recipient which got a 2yz reply.
>   - one end of mail reply code (as in the current ESMTP model,
>     which is the same as the "final" reply code in
>     draft-hall-deferrals).

How about limiting it to a kind of "SREJ" (selective reject) ?

If all 2yz accept the mail just send a final 2yz.  And if all
rejected it just send a final 5yz.  If some rejected it send
corresponding "SREJs" indicating "who" that is (like 551, only
different, "why" as extended code), terminated by a final 2yz.

Frank

Date: Sun, 4 Feb 2007 17:07:13 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: another per-recipient data reply proposal (was: Overview ...)
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

Here's another proposal based on my implementation experiences:

* Client initiates by sending MAIL with an extension.
* RCPT returns normal reply codes.
* DATA and the last BDAT return (in order):
  - one reply code per recipient which got a 2yz reply.
  - one end of mail reply code (as in the current ESMTP model,
    which is the same as the "final" reply code in
    draft-hall-deferrals).

This is the approach the causes the least changes to existing code
(at least in the MTAs that I maintain) and does not introduce complex
error handling. Maybe others can take a look at exim, postfix,
qmail, etc?

Date: Sun, 4 Feb 2007 16:52:39 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: end of mail data response? (was: draft-hall-deferrals-00.txt)
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Sun, Feb 04, 2007, Peter J. Holzer wrote:

> (Also, I'm not sure how expensive fsync() really is if there are lots of
> fsync() requests - AFAIK order doesn't have to be maintained, so the OS
> can optimize quite a lot there - but does it?).

On a "normal" FS fsync() is very expensive. For sendmail 8 there
is even an option:

# checkpoint queue runs after every N successful deliveries
#O CheckpointInterval=10

to reduce the number of fsync() operations, however, this may cause
repeated deliveries if the MTA crashes between "checkpoints".

You can try some FS tests yourself or you can read about my
performance results at
http://www.sendmail.org/~ca/email/sm-X/design-2005-05-05/main/main.html
(or check the postfix docs/source, there is probably something similar
wrt fsync()).

> > I've implemented "DRR" in my SMTP server, and I noticed that the
> > LMTP model, i.e., no explicit "end of mail data" reply, introduces
> > some problems:
> > 
> > - it requires special code to deal with the fact that a command
> >   doesn't have a reply.
> 
> Why doesn't it have a reply? It has one reply for every remaining
> recipient, and since there must be at least one (otherwise DATA returns
> a 503 reply) there is at least one reply.

That's not a reply for "end of mail data", it is for some RCPT.  In
some pseudo code (error handling omitted for simplicity) you could
do this after you received all mail data:

if "DRR" is active: call subroutine to give RCPT status.
commit mail data (all transaction information).
give "end of mail data" reply.

If you don't have a "end of mail data" reply, then the changes are
more "intrusive", which increases the complexity and hence may cause
implementation errors. Let's keep it simple.  After I implemented
the original draft and the LMTP model, I came to the conclusion
that this little modification of the LMTP model is the easiest to
implement and has the simplest error handling.  Try it yourself in
your favorite MTA :-)

> "end of mail data" sounds to me like it is sent before the individual
> rcpt replies. But from your reasoning I think you want it after the
> individual replies, like the "final response code" in Eric's draft.
> Right?

Correct. The "order" is preserved: first RCPTs, then "end of mail
data".

Date: Mon, 5 Feb 2007 00:08:12 +0000
From: Tony Finch <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Sender: [email-masked]

On Sun, 4 Feb 2007, Eric A. Hall wrote:
>
> Now, given that we are working in a pure SMTP model, it becomes pretty
> important to do things "the SMTP way" and while it is possible to always
> return 2xx at the envelope, it's probably not accurate if in fact you
> expect to respond with failure codes to some of those recipients later on.
> The SMTP way here is to be up-front about the staggered model; you know
> that some of them will fail, and unless the recipient is absolutely known
> not to fail (eg postmaster@, abuse@, or a lawyer that has to keep copies
> of all correspondence, or whatever), the right way to respond to the
> envelope address is to issue a 3xx, so that the client knows that the
> recipient may yet be refused. This is just common courtesy, and it also
> follows the same kind of design as other places in the protocol

At the moment 250 a reply to RCPT is provisional because the server can
still decide to reject after the DATA, so I don't see how 3yz is more like
the SMTP way.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FISHER: NORTHWEST 5 TO 7, OCCASIONALLY GALE 8 AT FIRST, DECREASING 4 LATER.
ROUGH OCCASIONALLY VERY ROUGH AT FIRST. MAINLY FAIR. GOOD.

Date: Mon, 5 Feb 2007 00:01:48 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Overview of per-recipient data reply proposals
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--ILuaRSyQpoVaJ1HG
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-02-04 16:11:51 -0500, Eric A. Hall wrote:
> On 2/4/2007 12:31 PM, Peter J. Holzer wrote:
> > I'll try to provide an overview of various proposals I am aware of which
> > allow a server to return different reply codes for different recipients
> > after receiving the data. They are in roughly chronological order.
>=20
> I'd add another slice to your list, which is whether content inspection is
> separate from delivery or not.
>=20
> Many of us are using tools that inspect the message body before we decide
> to bother with delivery, so this is an important filter.

I'm not sure I understand. All of the proposals allow content-inspection
according to per-recipient rules before delivery. That doesn't seem to
be a distinguishing feature.=20

If you are referring to your mail <[email-masked]>, the
point you make is that the deferral draft doesn't allow delivery to
start before the final reply, while with the LMTP model the message can
be delivered to a recipient as soon as content scanning for that
particular recipient is done.

BTW, the term "delivery" is a bit slippery here. Unlike an LMTP-server
which delivers directly into a user's mailbox, an SMTP server does not
generally do that. It may forward the mail to another SMTP server. So I
prefer to say that the server "accepts" or "commits" the message, which
means that now the server is responsible for further delivery, but
omits the details how and when that delivery is done. In many cases the
message (plus recipient(s)) will be written to a queue and the next
delivery attempt will start sometime later.

The information when the server accepts a message is in the list
already. There should probably also be performance and implementation
considerations.

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--ILuaRSyQpoVaJ1HG
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--ILuaRSyQpoVaJ1HG--

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Advancing RFC2821 to Draft Standard -- outline of work
Date:  Sun, 04 Feb 2007 22:57:59 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 36
Cc: Dave Crocker <[email-masked]>
Sender: [email-masked]

Dave Crocker wrote:

> Are you guys saying that current implementations do not conform
> to the spec or are you saying that the spec should be changed to
> tell people to do things differently?

Some interpret the spec. as saying "I can always accept mail and
bounce it later if I don't like it".  Others intrepret is as "I
can't accept mail unless I'm almost sure that I can (and will)
deliver it, or if I'm almost sure that a later bounce (NDR) won't
hit innocent bystanders."

Another group misinterprets it as "dropping suspicious mails is
better than bouncing them".  That's wrong if they're not _very_
sure that the mail is bad.

The other two interpretations are covered by "the originator as
indicated in the reverse path".  The first group simply believes
that the reverse path is okay.  The 2nd group takes into account
that this "indication" can be forged.  As it's the case in most
mails today.

The first group claims that they MUST send NDRs to follow the
spec., the second group says that receivers SHOULD reject mails
at their border MX to follow the spec.

The latter isn't explicitly stated in 2821.  It's a consequence
of an interpretation of "originator as indicated in the reverse
path", plus the observation that this indication is generally
bogus at (or behind) an MX.

draft-hutzler-spamops tried to address this issue, but only for
MDAs, typically behind the MX when it's too late.

Frank

Date: Sun, 4 Feb 2007 21:32:55 +0000
From: "Peter Bowyer" <[email-masked]>
To: [email-masked]
Subject: Re: First vs. last -- Need and interest
Sender: [email-masked]

On 04/02/07, Claus Assmann <[email-masked]> wrote:
>
> Summary: IMHO this is a fairly important feature. It will help those
> MTAs which implement it to avoid the problems of the current
> "workarounds".

This is a regularly-requested feature for Exim, also. New adopters
understand the advantage of per-recipient preferences for
envelope-level filtering (DNSBLs, HELO checks, rDNS
existance/consistency, SPF, etc) and want to extend this to
content-level per-recipient filtering preferences.

> For others you can simply only accept one RCPT[*]
> per transaction, especially from "unknown" sources.

> PS: [*] obviously you can enhance this by accepting multiple
> RCPTs if they have the same filtering requirements.

Yes, this type of workaround is documented on exim.org[1], promoted on
exim-users and presumably adopted by some proportion of the userbase.
For smaller sites, the workaround probably has negligible impact, but
it doesn't scale to large ESP/ISPs.

A solution is therefore needed.

Peter

[1] http://www.exim.org/eximwiki/ExiscanExamples

-- 
Peter Bowyer
Email: [email-masked]

Date: Sun, 4 Feb 2007 13:16:54 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: First vs. last -- Need and interest
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

I just remembered the other workaround I've seen: the MTA accepts
the mail, "splits" it according to the RCPT filtering requirements,
and then bounces those mails that are rejected by the filter(s).
As it was indicated in a different thread (about RFC2821), this is
a really bad idea.

BTW: some of those requests for "per RCPT filtering" came from
commercial vendors who use the sendmail MTA in their product, so
they most likely got those requests from their customers.

Summary: IMHO this is a fairly important feature. It will help those
MTAs which implement it to avoid the problems of the current
"workarounds". For others you can simply only accept one RCPT[*]
per transaction, especially from "unknown" sources.

PS: [*] obviously you can enhance this by accepting multiple
RCPTs if they have the same filtering requirements.

Date: Sun, 04 Feb 2007 16:11:51 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: Overview of per-recipient data reply proposals
Sender: [email-masked]

On 2/4/2007 12:31 PM, Peter J. Holzer wrote:
> I'll try to provide an overview of various proposals I am aware of which
> allow a server to return different reply codes for different recipients
> after receiving the data. They are in roughly chronological order.

I'd add another slice to your list, which is whether content inspection is
separate from delivery or not.

Many of us are using tools that inspect the message body before we decide
to bother with delivery, so this is an important filter.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sun, 04 Feb 2007 16:10:04 -0500
From: "Eric A. Hall" <[email-masked]>
CC: [email-masked]
Subject: Re: First vs. last -- Need and interest
Sender: [email-masked]

On 2/4/2007 3:09 PM, Dave Crocker wrote:
> offlist.

not quite :)

> Claus Assmann wrote:
>>> I keep wondering whether it isn't just as useful and far simpler to have an 
>>> SMTP option that says "send addresses after content"?
>> No (see an earlier mail about this suggestion).
> 
> I am not finding any postings that relate to the model I am suggesting. 
> Although there are some statements of preference, perhaps, there aren't any 
> explanations that I am finding.

I think he was referring to discussions in ASRG before we moved here.
There are lots of problems with it, but the biggest headache is that there
is a huge amount of mail that does not have any valid recipients, so
transferring the body without recipient confirmation is a waste often as
not. As one anecdotal example, half of the messages I get here on any
given day have envelope recipients that are nothing more than message IDs.
More complicated examples of this are spamtrap addresses, whose purpose
for existence is to prevent me from having to accept mail that includes
them as named recipients.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sun, 04 Feb 2007 12:09:46 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: Re: First vs. last -- Need and interest
Sender: [email-masked]

offlist.

Claus Assmann wrote:
>> I keep wondering whether it isn't just as useful and far simpler to have an 
>> SMTP option that says "send addresses after content"?
> 
> No (see an earlier mail about this suggestion).

I am not finding any postings that relate to the model I am suggesting. 
Although there are some statements of preference, perhaps, there aren't any 
explanations that I am finding.

Eric's proposal retains the initial address processing, and then adds 
post-content responses.  This is more complicated than the model I cited, which 
entirely MOVES the address portion to the end.

Please clarify.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Sun, 4 Feb 2007 11:52:53 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: First vs. last -- Need and interest
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Sun, Feb 04, 2007, Dave Crocker wrote:

> In other words, how clear is it what problems need to be solved? In 
> phrasing it that way, I am really posing questions:  1) What problems are 
> being solved?  2) What is the basis for believing that there is a ready 
> community of adopters for the solution, or rather, that there is a market 
> that agrees that the problems need solving?

1) Per RCPT filtering is a request we (sendmail.org) get fairly
often. As it cannot be done "directly" in ESMTP (yet), people ask
for "wierd" workarounds that are complicated to implement.  Moreover,
I already see some workarounds being implemented at some ISPs (e.g.,
accept only 1 RCPT per transaction) as well as in milters.

2) The number of requests we get and the number of "workaround
proposals".

> I keep wondering whether it isn't just as useful and far simpler to have an 
> SMTP option that says "send addresses after content"?

No (see an earlier mail about this suggestion).

Date: Sun, 04 Feb 2007 11:48:33 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
CC: Frank Ellermann <[email-masked]>, [email-masked]
Subject: Re: Advancing RFC2821 to Draft Standard -- outline of work
Sender: [email-masked]

Are you guys saying that current implementations do not conform to the spec or 
are you saying that the spec should be changed to tell people to do things 
differently?

And given that this all pertains to "SHOULD" labels rather than "MUST" I would 
expect that the delta between spec and reality would need to be quite high, to 
warrant changing the directive.

d/

[email-masked] wrote:
> On Mon, 22 Jan 2007 22:24:27 +0100, Frank Ellermann said:
>> However there's no way that this rest can be a "draft standard"
>> without fixing another major problem in RFC 2821, the design
>> flaw to favour "accept and bounce" instead of "reject".
> 
> Unfortunately, I have to agree with Frank here - today's Internet isn't
> the Internet of RFC821, and we really *should* take a very close look at
> fixing that language even if it *does* mean another iteration at Proposed,
> beecause if we don't fix it this time around, we'll be saddled with it
> for years - RFC6821 is a long way aray.
> 
> Would tagging "accept and bounce" as a SHOULD NOT (with appropriate text
> attached) be doable without resetting back to Proposed?  I know "MUST NOT"
> would cause such a reset, and even I'm not ready to stick *that* label on
> "accept and bounce" - mostly because I've spent too much time working with
> large-scale mail systems where the mailbox *was* valid when we sent the
> far end the '250 OK', but had been race-conditioned into oblivion by the
> time actual final delivery was attempted.

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Sun, 04 Feb 2007 10:20:22 -0800
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: First vs. last -- Need and interest
Sender: [email-masked]

Folks,

Peter J. Holzer wrote:
> I'll try to provide an overview of various proposals 
...

This sort of side-by-side comparison can only be helpful.  However I'm not sure 
whether there is a clear agreement about the goals that will motivate making a 
choice?

In other words, how clear is it what problems need to be solved? In phrasing it 
that way, I am really posing questions:  1) What problems are being solved?  2) 
What is the basis for believing that there is a ready community of adopters for 
the solution, or rather, that there is a market that agrees that the problems 
need solving?

The list discussion seems to have focused on the mechanical issues.  In looking 
for a statement of deeper motivation, one of the more crisp bits seems to be:

      "it has become common practice for different recipients to
      have their own content filters, and a single response code is not
      sufficiently granular to accurately reflect these differences."

from Eric's draft.  The language does not mention additional reasons for needing 
responses that are contingent on the content -- and potentially vary for each 
recipient -- but at least it gives one that is simple, direct and probably 
compelling.  (An other reason includes policies of the receiving MTA operator, 
possibly independent of the expressed input of the addressee.)

When email protocols were being developed, one of the debates was between having 
content precede addresses or addresses precede content.  Each had its benefits 
and drawbacks.  (In protocol development, header-first vs. header-last is a 
classic trade-off.)  I wouldn't characterize the SMTP choice as arbitrary, but 
certainly it was not obviously the right choice.

By having addresses first, the protocol permits withholding transfer of the 
content, if there are no valid addressees.  In a world of limited bandwidth and 
storage, along with any expectation that a substantial number of addresses will 
be invalid, having the addresses come first is an excellent choice.  It is also 
the reason the protocol was made interactive, per addressee, in spite of the 
cumulative hassle of network latencies this creates.

To the extent that storage and bandwidth are cheaper and addresses are typically 
valid, having addresses come after content permits a richer evaluation model by 
the receiving MTA.  In a world of content abuse -- and, yes, per-recipient 
preferences -- the content-first model seems considerably more appealing than 
what we have now.  However, note that we probably lose the ability to have 
rejection before content transfer, except for the most basic sorts of 
rejections, such as being unwilling to take mail from a particular SMTP client host.

The need for communicating richer response information within the session has 
also come up in the MAAWG community, with a new effort to define response codes 
that contain information about policy violations, where I mostly view that term 
as referring to cross-message -- and possibly cross-addressee -- rejections. 
So, rather than saying "This address doesn't exist" the response might be more 
like "Don't bother contacting this host until you clean up your act."

This is not exactly the same requirement as is being discussed on this list, but 
I think it is related.  At last week's MAAWG meeting, the interest in an effort 
to add some SMTP response codes was particularly strong, suggesting that there 
is a ready market for using the result of the work.

As for stylistic choice between LMTP and an SMTP enhancement, my immediate 
concern is that LMTP is only for last-hop processing.

However the need for a richer and finer-grained content-related response model 
is showing up for intermediaries, not just last-hop environments.

LMTP provides responses before content-transfer and after.  That sounds pretty 
appealing, as does using an existing protocol.

Like others, I'm concerned about the added complexity.

I keep wondering whether it isn't just as useful and far simpler to have an SMTP 
option that says "send addresses after content"?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Sun, 4 Feb 2007 19:07:43 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: end of mail data response? (was: draft-hall-deferrals-00.txt)
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--i9LlY+UWpKt15+FH
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-02-04 09:06:07 -0800, Claus Assmann wrote:
> On Sun, Feb 04, 2007, Peter J. Holzer wrote:
> > The more I think about it, the more I'm convinced that committing on a
> > per-recipient basis (as LMTP does) is more robust. I do agree that it is
> > more difficult to implement on the server side, though.
>=20
> What does "committing" mean for you here?

Saving the information that the message should be delivered to the
recipient and telling the client that responsibility for further
delivery has been accepted.

The deferrals draft only commits once for all recipients: Either all or
none get the message. Consequently if the client doesn't receive the
final reply, it has to assume that none of the recipients will receive
it and resend it to all of them.

In LMTP, the server commits for every recipient. If the client receives
only replies for some of the recipients, it has to assume that those for
which it got replies will be delivered and resend only to those for
which it got no replies.

> What exactly do you want
> to commit? Do you want to do like this:
> - is there any RCPT that is accepted?
>   . yes: commit data
>     - for each RCPT:
>       is it accepted?
>       - yes: commit RCPT
>       send reply
>   . no: trivial case
> That's not very efficient as a "commit" (e.g., fsync()) is the most
> expensive I/O operations on conventional hardware.

Yes. It's more expensive but more robust than Eric's draft, I think.

(Also, I'm not sure how expensive fsync() really is if there are lots of
fsync() requests - AFAIK order doesn't have to be maintained, so the OS
can optimize quite a lot there - but does it?).

> I've implemented "DRR" in my SMTP server, and I noticed that the
> LMTP model, i.e., no explicit "end of mail data" reply, introduces
> some problems:
>=20
> - it requires special code to deal with the fact that a command
>   doesn't have a reply.

Why doesn't it have a reply? It has one reply for every remaining
recipient, and since there must be at least one (otherwise DATA returns
a 503 reply) there is at least one reply.

> - the error handling in case of a lost connection is more complicated.
>=20
> Hence I'm strongly in favor of an "extended" LMTP model where
> additionally to the individual RCPT responses a separate "end of
> mail data" response is used.

"end of mail data" sounds to me like it is sent before the individual
rcpt replies. But from your reasoning I think you want it after the
individual replies, like the "final response code" in Eric's draft.
Right?

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--i9LlY+UWpKt15+FH
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--i9LlY+UWpKt15+FH--

Date: Sun, 4 Feb 2007 18:31:19 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Overview of per-recipient data reply proposals (was: Fwd: I-D ACTION:draft-hall-deferrals-00.txt)
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--sdtB3X0nJg68CQEu
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I'll try to provide an overview of various proposals I am aware of which
allow a server to return different reply codes for different recipients
after receiving the data. They are in roughly chronological order.

LMTP (RFC 2033, <URL:http://tools.ietf.org/html/rfc2033>):

    * Client initiates by sending LHLO.

    * RCPT returns normal reply codes.

    * DATA and the last BDAT return one reply code per recipient which
      got a 2xx reply.

    * Message is accepted for a recipient when server sends the data
      reply for that recipient.

draft-varshavchik-data-smtpext
<URL:http://www.courier-mta.org/draft-varshavchik-exdata-smtpext.txt>:

    * Client initiates by sending MAIL FROM with an extension.

    * RCPT returns normal reply codes.

    * DATA returns a single 558 multiline reply which contains nested
      SMTP replies.

    * It isn't clear when the message is accepted by the server: When it
      sends the nested submessage or when it sends the final line of the
      message.

datafirst (David Nicol on the ASRG mailing list,
<URL:http://www1.ietf.org/mail-archive/web/asrg/current/msg12808.html>:

    * Client initiates by sending MAIL FROM with an extension.

    * DATA is sent after MAIL FROM but before the first RCPT TO.

    * RCPT returns normal reply codes.

    * Presumably, each reply to a RCPT command accepts the mail for this
      recipient.

    * This has the advantage of only returning one reply code for each
      recipient (unlike all other proposals, which return 2), but at the
      cost of transferring the data even if there are no valid
      recipients.

recheck Tony Finch on the ASRG mailing list:
<URL:http://www1.ietf.org/mail-archive/web/asrg/current/msg12839.html>:

    * Client initiates by sending MAIL FROM with an extension.

    * RCPT can return 350 in addition to normal reply codes.

    * DATA returns a normal reply, but if there were any 350 replies,
      the transaction isn't finished yet.

    * Client sends a RECHECK command for each recipient which got a 350
      reply. RECHECK commands may be issued in any order.

    * The message is accepted for a particular recipient when the server
      sends the reply to the RECHECK command.

    * The transaction must be finished with RSET.

    * Sort of the opposite of datafirst: This one doubles the number of
      commands as well as the number of replies. But it keeps the "one
      reply per command" model of SMTP.

draft-hall-deferrals:

    * Client initiates by sending MAIL FROM with an extension.

    * RCPT can return 352 in addition to normal reply codes.

    * DATA and the last BDAT return one reply code per recipient which
      got a 352 reply. Additionally, they return a 353 reply code to
      signal that multiple reply codes follow before the individual
      reply codes and a final (2xx/4xx/5xx) reply code to signal whether
      the message has been accepted.

    * The message is accepted for all recipients for which a 2xx reply
      has been sent if the final reply code is also 2xx. Otherwise it
      has been accepted for none.

David Nicol also suggested off-list an interesting variant where DATA is
sent by the client after it receives the first 2xx reply to a RCPT
command. The client can then continue to send additional RCPT commands.

That's it. Have I forgotten anything?

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--sdtB3X0nJg68CQEu
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--sdtB3X0nJg68CQEu--

Date: Sun, 4 Feb 2007 09:06:07 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: end of mail data response? (was: draft-hall-deferrals-00.txt)
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Sun, Feb 04, 2007, Peter J. Holzer wrote:

> I don't think this assumption is safe. Also if you commit the
> transaction only after sending the final reply code, your server could
> crash after sending the reply and before the commit. In this case the
> message will be lost.

That's an obvious violation of RFC (2)821. The server has to commit
the message before it sends the reply.

> The more I think about it, the more I'm convinced that committing on a
> per-recipient basis (as LMTP does) is more robust. I do agree that it is
> more difficult to implement on the server side, though.

What does "committing" mean for you here? What exactly do you want
to commit? Do you want to do like this:
- is there any RCPT that is accepted?
  . yes: commit data
    - for each RCPT:
      is it accepted?
      - yes: commit RCPT
      send reply
  . no: trivial case
That's not very efficient as a "commit" (e.g., fsync()) is the most
expensive I/O operations on conventional hardware.

I've implemented "DRR" in my SMTP server, and I noticed that the
LMTP model, i.e., no explicit "end of mail data" reply, introduces
some problems:

- it requires special code to deal with the fact that a command
  doesn't have a reply.
- the error handling in case of a lost connection is more complicated.

Hence I'm strongly in favor of an "extended" LMTP model where
additionally to the individual RCPT responses a separate "end of
mail data" response is used. This matches the existing model better
(no special case code, no "per RCPT commits"), and makes the error
handling simpler: the server accepts the responsibility for the
mail only if there is positive "end of mail data" reply (just as
it does without this extension), there is no "partial success".

Date: Sun, 4 Feb 2007 07:42:32 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Fri, Feb 02, 2007, Eric A. Hall wrote:

> ahh wait, linear responses are easier for the client but introduce more
> work to the server, since they have to keep track of status for all
> recipients (even the nonexistent ones) across the entire session.

The server needs to keep only track of RCPTs that it accepted:

4.2.  The DATA command
...
   The change is that after the final ".", the server returns one reply
   for each previously successful RCPT command in the mail transaction,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^

Date: Sun, 04 Feb 2007 10:08:07 -0500
From: "Eric A. Hall" <[email-masked]>
CC: [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Sender: [email-masked]

On 2/4/2007 12:06 AM, Eric A. Hall wrote:

> As I said before, there is a significant architectural difference between
> the way that SMTP+LMTP works, and the way that an SMTP-only model would
> work. In the former, the SMTP server simply responds to envelope
> recipients with a flat yes/no binary toggle based on factors such as
> whether or not the recipient exists, if a specified size exceeds quota,
> and so forth. Once the message is received the SMTP server passes the
> "good" recipients to the mailstore via LMTP (the latter only has to deal
> with the good recipients), which may reject recipients based on
> unspecified factors (including content, size/quota problems that were not
> recognized up-front, and so forth). At that point the SMTP server has to
> generate DSNs for any LMTP failures.
> 
> In the pure SMTP model, all of this stuff is compressed into a single
> session, which has some pretty significant ramifications.

I forgot one other part of this, which is that legacy SMTP+LMTP generally
combines scanning and delivery into one process, while a pure SMTP model
basically splits scanning and delivery into two sub-processes.

For example:

        C: DATA
        S: 354 go ahead
        C: <sends data>
        C: <crlf>.<crlf>
       --------------------content scanning starts--------------------
        S: 353 data looks okay but wait for confirmation
        S: 550 5.6.0 <[email-masked]> refuses the content
        S: 250 2.1.5 <[email-masked]> accepts the content
        S: 250 data accepted by some recipients
       ------------------delivery processing starts-------------------

Note that delivery cannot occur until the server accepts responsibility
for the message. However the proposals on the table are intended to allow
for the results of content analysis to be returned before the message has
been accepted. As such, we are basically peeling off the content analysis
sub-part of LMTP into a separate thread, but are not obviating or
otherwise affecting the delivery part.

One related minor point here is that the list gets smaller as you process
through these steps:

     envelope -- rejected, accepted, deferred
     content -- deferred recipients have content scanned
     delivery -- combined list of accepted envelope and content recipients

If you wanted to stick with the pure LMTP model, where analysis was
performed as part of delivery, then you'd need to basically accept
responsibility for the message first, and then enumerate the results of
the delivery process. This would probably require a multi-line 25x response.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sun, 04 Feb 2007 09:55:10 -0500
From: "Eric A. Hall" <[email-masked]>
CC: [email-masked]
Subject: Re: Fwd: I-D ACTION:draft-hall-deferrals-00.txt
Sender: [email-masked]

On 2/4/2007 6:30 AM, Peter J. Holzer wrote:

> In LMTP, the server will send the first 2xx reply after 90 seconds, and
> another reply every 30 seconds. The client will never come close to the
> timeout of 5 minutes. Even if it foolishly doesn't reset the timeout
> after receiving each reply, the message will only be duplicated for the
> last two or three recipients.

The DEFERRALS draft avoids this problem somewhat with the 353 response
code that heads the list of enumerated responses, although the current I-D
does not specify when it should be returned (it could be generated as soon
as the end-of-data mark has been received, or as soon as content scanning
begins [probably the right place for it], or after content scanning has
ended and before the list of recipient codes is returned).

On a broader not, it's entirely feasible to demand stricter conformance
from clients that implement any of these drafts. "Cheating is no longer
allowed, due to problems X and Y."

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sun, 4 Feb 2007 15:55:19 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Fwd: I-D ACTION:draft-hall-deferrals-00.txt
Sender: [email-masked]

--jho1yZJdad60DJr+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-02-03 09:20:18 -0500, Eric A. Hall wrote:
> On 2/3/2007 6:05 AM, Tony Finch wrote:
>=20
> > So the question is whether adding a final response after the per-recipi=
ent
> > resposes helps with this problem.
>=20
> > Whichever design we choose, the client is still in trouble if it loses =
the
> > tail of the server's final per-recipient replies.
>=20
> > The only solution to RFC 1047 is checkpointing.
>=20
> A robust mechanism only requires the server to know that the client
> received the entire list of responses: if so then the client is not at
> obvious risk of resending; if not the server can refuse to accept
> responsibility on the grounds that the session was lost. So all we need is
> a way for the client to send SOME KIND OF DATA, after the server has sent
> the full list.
>=20
> Using a reliable transport like TCP and decent synchronization in the
> protocol, this can be done with the client merely issuing another command,
> such as QUIT.

Not when pipelining is enabled. The client must wait for the 3xx reply
to DATA, but then it can send the message content, the final "." and
QUIT all in one fell swoop.

> My current design is somewhat thinner than that--if the server is able to
> write() the final response code without error, then it should be safe to
> assume that the client received the full list as well as the final
> response and the mail belongs to the server now.

I don't think this assumption is safe. Also if you commit the
transaction only after sending the final reply code, your server could
crash after sending the reply and before the commit. In this case the
message will be lost.

The more I think about it, the more I'm convinced that committing on a
per-recipient basis (as LMTP does) is more robust. I do agree that it is
more difficult to implement on the server side, though.

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--jho1yZJdad60DJr+
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--jho1yZJdad60DJr+--

Date: Sun, 4 Feb 2007 12:30:43 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Fwd: I-D ACTION:draft-hall-deferrals-00.txt
Sender: [email-masked]

--OgqxwSJOaUobr8KG
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-02-03 11:05:58 +0000, Tony Finch wrote:
> On Fri, 2 Feb 2007, Eric A. Hall wrote:
> > On 1/30/2007 2:45 PM, Tony Finch wrote:
> > > My main question about this draft is whether the extra complexity com=
pared
> > > to LMTP is really useful. What is your rationale for it?
> >
> > I'd have to find my brain from 2003 to answer for sure, but I think the
> > motivation came down to the fact that LMTP is essentially designed as a
> > mailstore API and specifically was not designed as a transfer protocol =
(it
> > actually disavows all such purposes).
>=20
> I believe that the rationale and limited purpose for which LMTP was
> designed do not imply that it isn't useful for other purposes. Spam and
> message content scanning were not common in 1996!

I also believe that the rationale given in RFC 2033, that the LMTP
protocol should not be used over WANs because it can aggravate the
situation described in RFC 1047, is faulty. While TCP connections do
sometimes fail because of network errors, I believe the most common
reason for duplicate messages is that processing of the message on the
server side takes longer than the timeout on the client side.

Consider the case where a client has a configured timeout of 5 minutes
(that's shorter than required in RFC 2821, but not uncommon) and
processing on the server takes 1 minute initially plus 30 seconds for
each of the ten recipients.

In SMTP, the server will send back a 2xx reply after 6 minutes. At that
time the client has already quit and will retry later. The message will
be duplicated for all recipients (unless the server detects that
connection is already closed).

In LMTP, the server will send the first 2xx reply after 90 seconds, and
another reply every 30 seconds. The client will never come close to the
timeout of 5 minutes. Even if it foolishly doesn't reset the timeout
after receiving each reply, the message will only be duplicated for the
last two or three recipients.

> > Also, some of the requests and comments have suggested that it doesn't
> > do enough, such as Peter Holzer's suggestion that the mechanism for
> > wrapping the list of responses may not be robust enough to avoid
> > duplicates (and the mechanism I present is already a LOT sturdier than
> > 2033).
>=20
> So the question is whether adding a final response after the per-recipient
> resposes helps with this problem. The authors of RFC 2033 seem to have in
> mind an implementation that performs each recipient's delivery action then
> sends that recipient's final response. We're going to have to be a bit
> more sophisticated. In particular, there's a security consideration: you
> do not want to re-scan the message for each recipient because this makes
> it easier for an attacker to use lots of your CPU. A safer model is to
> scan the message once, then different per-recipient policies depend on the
> result of that scan (e.g. postmaster is much more lax than other
> recipients). This design allows you to send out the final replies as fast
> as possible, e.g. using a single TCP send.

I'd rather depend on the TCP stack using Nagle's algorithm to
concatenate the replies that doing that in the server process. I have in
the past sometimes grossly underestimated the maximum time something
could take. So - purely as an advice to server programmers - I'd suggest
that the server process send out any reply as soon as it is available.
If the server sends the replies fast enough the OS will coalesce them
into a single packet anyway.

> Holzer's suggestion effectively requires this, because he makes the
> server send the overall message status before the per-recipient
> statuses.

No, that was just poor phrasing on my part. "message accepted for some
recipients" should really read "message accepted for zero or more
recipients". It's the individual codes that count.
draft-varshavchik-data-smtpext (which I wasn't aware of) uses a 5xx code
instead of a 2xx code as a "wrapper".

The intention was that the server would send out the initial 250- line
as soon as it has received the data (and possibly performed tests on it
which apply to all recipients) and then send the partial replies as it
performs the tests for each recipient.

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--OgqxwSJOaUobr8KG
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--OgqxwSJOaUobr8KG--

Date: Sun, 04 Feb 2007 00:06:30 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Sender: [email-masked]

On 1/30/2007 7:21 PM, Claus Assmann wrote:

> It avoids replies for those recipients that are accepted without
> having to check the message. Is it worth it? I don't know...
> 
> To make it simpler for people to implement "deferred RCPT reply"
> (DRR) it would be better to simply use the LMTP model as most likely
> several SMTP clients implement this already and hence existing code
> can be reused.

I had to do some long-distance driving today for family stuff, and I spent
some time noodling on this.

As I said before, there is a significant architectural difference between
the way that SMTP+LMTP works, and the way that an SMTP-only model would
work. In the former, the SMTP server simply responds to envelope
recipients with a flat yes/no binary toggle based on factors such as
whether or not the recipient exists, if a specified size exceeds quota,
and so forth. Once the message is received the SMTP server passes the
"good" recipients to the mailstore via LMTP (the latter only has to deal
with the good recipients), which may reject recipients based on
unspecified factors (including content, size/quota problems that were not
recognized up-front, and so forth). At that point the SMTP server has to
generate DSNs for any LMTP failures.

In the pure SMTP model, all of this stuff is compressed into a single
session, which has some pretty significant ramifications. For starters,
the SMTP envelope recipients can no longer be dealt with on a purely
binary basis--the server has to return yes for some, no for some, and a
maybe for others. It doesn't "have" to do this, but if the server is still
going to be passing the recipients that were not rejected to a content
filter (possibly via LMTP or otherwise) then you want to preserve that
back-end model, meaning only passing it addresses that have not already
failed at the envelope level. I think this is pretty obvious from a
programmer perspective.

Now, given that we are working in a pure SMTP model, it becomes pretty
important to do things "the SMTP way" and while it is possible to always
return 2xx at the envelope, it's probably not accurate if in fact you
expect to respond with failure codes to some of those recipients later on.
The SMTP way here is to be up-front about the staggered model; you know
that some of them will fail, and unless the recipient is absolutely known
not to fail (eg postmaster@, abuse@, or a lawyer that has to keep copies
of all correspondence, or whatever), the right way to respond to the
envelope address is to issue a 3xx, so that the client knows that the
recipient may yet be refused. This is just common courtesy, and it also
follows the same kind of design as other places in the protocol

Another important aspect of using 3xx status codes in response to RCPT is
that it provides a way for the server to tell the client that the client's
use of the extension is being recognized. In this regard, it provides a
free ACK to the use of the RCPT TO extension.

All told, I am pretty sure that these are the reasons why I chose to use
3xx in the RCPT TO part. I'm not married to it; that's why it's there.

Incidentally, the compressed model is also why I called this I-D
"INLINE-DSN" the first time through--we are essentially generating DSNs
immediately instead of accepting the message and generating them off-line
later, and the change in model drives a bunch of other things. I prefer
the Deferred RCPT Response name (DRR for short) since it's a bit more
transparent about what's happening, fwiw.

One other thing, I think the reliability issue may be dealt with the
easiest by simply instructing implementers to save the final response code
for its own dedicated write(), and if that is not possible or if there is
any confusion then to go ahead and accept responsibility on the assumption
that the final-response was accepted. That may be too much cross-layer
signaling, but it avoids a lot of unnecessary statefulness that would
otherwise be necessary.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sat, 03 Feb 2007 09:20:18 -0500
From: "Eric A. Hall" <[email-masked]>
To: Tony Finch <[email-masked]>
CC: [email-masked]
Subject: Re: Fwd: I-D ACTION:draft-hall-deferrals-00.txt
Sender: [email-masked]

On 2/3/2007 6:05 AM, Tony Finch wrote:

> So the question is whether adding a final response after the per-recipient
> resposes helps with this problem.

> Whichever design we choose, the client is still in trouble if it loses the
> tail of the server's final per-recipient replies.

> The only solution to RFC 1047 is checkpointing.

A robust mechanism only requires the server to know that the client
received the entire list of responses: if so then the client is not at
obvious risk of resending; if not the server can refuse to accept
responsibility on the grounds that the session was lost. So all we need is
a way for the client to send SOME KIND OF DATA, after the server has sent
the full list.

Using a reliable transport like TCP and decent synchronization in the
protocol, this can be done with the client merely issuing another command,
such as QUIT. This falls apart when clients are broken though, eg close
without QUIT, or client hits a bug and starts spewing commands that it
shouldn't, all of which happen often enough. I don't know if we need to
design for that level of disaster though--it's not really accommodated
anywhere else. It might be worth putting a client command in there
("DRR-DONE") just to avoid any problems that may crop up.

My current design is somewhat thinner than that--if the server is able to
write() the final response code without error, then it should be safe to
assume that the client received the full list as well as the final
response and the mail belongs to the server now. This is about the
simplest possible way to go (other than omitting the final response code
altogether, which is too weak imo).

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Sat, 3 Feb 2007 11:05:58 +0000
From: Tony Finch <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
cc: Tony Finch <[email-masked]>, [email-masked]
Subject: Re: Fwd: I-D ACTION:draft-hall-deferrals-00.txt
Sender: [email-masked]

On Fri, 2 Feb 2007, Eric A. Hall wrote:
> On 1/30/2007 2:45 PM, Tony Finch wrote:
> > My main question about this draft is whether the extra complexity compared
> > to LMTP is really useful. What is your rationale for it?
>
> I'd have to find my brain from 2003 to answer for sure, but I think the
> motivation came down to the fact that LMTP is essentially designed as a
> mailstore API and specifically was not designed as a transfer protocol (it
> actually disavows all such purposes).

I believe that the rationale and limited purpose for which LMTP was
designed do not imply that it isn't useful for other purposes. Spam and
message content scanning were not common in 1996!

> |  Use of LMTP can aggravate the situation described in [DUP-MSGS].

> Also, some of the requests and comments have suggested that it doesn't
> do enough, such as Peter Holzer's suggestion that the mechanism for
> wrapping the list of responses may not be robust enough to avoid
> duplicates (and the mechanism I present is already a LOT sturdier than
> 2033).

So the question is whether adding a final response after the per-recipient
resposes helps with this problem. The authors of RFC 2033 seem to have in
mind an implementation that performs each recipient's delivery action then
sends that recipient's final response. We're going to have to be a bit
more sophisticated. In particular, there's a security consideration: you
do not want to re-scan the message for each recipient because this makes
it easier for an attacker to use lots of your CPU. A safer model is to
scan the message once, then different per-recipient policies depend on the
result of that scan (e.g. postmaster is much more lax than other
recipients). This design allows you to send out the final replies as fast
as possible, e.g. using a single TCP send. Holzer's suggestion effectively
requires this, because he makes the server send the overall message status
before the per-recipient statuses. However you can do the same with
unmodified LMTP final replies.

Whichever design we choose, the client is still in trouble if it loses the
tail of the server's final per-recipient replies. If the server partly or
wholly accepts the message, the client does not know if any of the lost
recipients were rejected and must therefore be sent a bounce. The only
thing the client can do in that situation is to re-send the message to the
lost recipients, risking duplicates. An overall message status only
improves things when all recipients reject the message, and in that
situation it is just a minor optimisation. In fact, if the overall status
is last then the problem is worse! The client does not know whether the
250 replies it did get for the earlier recipients were later rescinded by
the server's final overall response, so the client must re-send to all the
recipients.

So I do not think any kind of whole-message status is an improvement to
LMTP. The only solution to RFC 1047 is checkpointing. I have made some
changes to my draft to accommodate LMTP explicitly, but I haven't
submitted a -01 version because I'm waiting for feedback. In the mean time
the current revision is available at
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-rfc1845bis.html

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
HEBRIDES: SOUTHWESTERLY 5 OR 6, OCCASIONALLY 7 IN NORTH. ROUGH OR VERY ROUGH.
FAIR. GOOD.

Date: Fri, 02 Feb 2007 21:33:04 -0500
From: "Eric A. Hall" <[email-masked]>
To: "Eric A. Hall" <[email-masked]>
CC: [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Sender: [email-masked]

On 2/2/2007 9:30 PM, Eric A. Hall wrote:

> I'm not really married to the 3xx responses to RCPT and if people think a
> straight 1-for-1 linear response would be easier to track

ahh wait, linear responses are easier for the client but introduce more
work to the server, since they have to keep track of status for all
recipients (even the nonexistent ones) across the entire session.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Fri, 02 Feb 2007 21:30:49 -0500
From: "Eric A. Hall" <[email-masked]>
To: [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Sender: [email-masked]

On 1/30/2007 7:21 PM, Claus Assmann wrote:
> On Tue, Jan 30, 2007, Tony Finch wrote:
> 
>> My main question about this draft is whether the extra complexity compared
>> to LMTP is really useful. What is your rationale for it? I have considered
> 
> It avoids replies for those recipients that are accepted without
> having to check the message. Is it worth it? I don't know...

I think I was looking for a way to reduce the size of the post-DATA
response list but I can't really remember. It probably also struck me as
opportune way to deal with some kinds of dictionary attacks, but given the
bilateral nature of this extension that would have been a fairly dumb
thing to design for. It was an off-the-cuff design so I may not have
really thought about it even to that extent, dunno.

I'm not really married to the 3xx responses to RCPT and if people think a
straight 1-for-1 linear response would be easier to track, that's fine
with me. SMTP clients still need to keep track of per-recipient response
codes (RCPT codes if nothing else), and this really only introduces a
minor amount of complexity to that existing requirement, but if it's
annoying to code I have no problems about pulling it out.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Fri, 02 Feb 2007 21:21:33 -0500
From: "Eric A. Hall" <[email-masked]>
To: Tony Finch <[email-masked]>
CC: [email-masked]
Subject: Re: Fwd: I-D ACTION:draft-hall-deferrals-00.txt
Sender: [email-masked]

On 1/30/2007 2:45 PM, Tony Finch wrote:
> My main question about this draft is whether the extra complexity compared
> to LMTP is really useful. What is your rationale for it? I have considered
> writing a draft that simply defines how to turn on RFC 2033 data responses
> in SMTP, without specifying any significant new protocol features.

I'd have to find my brain from 2003 to answer for sure, but I think the
motivation came down to the fact that LMTP is essentially designed as a
mailstore API and specifically was not designed as a transfer protocol (it
actually disavows all such purposes). It may seem like a subtle difference
but I would refer back to the wording in rfc2033 here:

|  Use of LMTP can aggravate the situation described in [DUP-MSGS].  To
|  avoid this synchronization problem, the following requirements are
|  made of implementations:
|
|  A server implementation which is capable of quickly accepting
|  responsibility for delivering or relaying a message to multiple
|  recipients and which is capable of sending any necessary notification
|  messages SHOULD NOT implement the LMTP protocol.
|
|  The LMTP protocol SHOULD NOT be used over wide area networks.

So I'm pretty sure I was thinking along the lines of ~how do we provide
per-recipient response in a way that is actually usable in SMTP. Just for
starters we would need the following:

1) SMTP extension to signify availability

2) command extension to indicate usage

3) some kind of mechanism to contain the list of responses

My I-D pretty much does that, and only that (the additional RCPT response
code of 3xx is probably not NEEDED but it seemed like the correct SMTP way
to deal with some kinds of recipients). Also, some of the requests and
comments have suggested that it doesn't do enough, such as Peter Holzer's
suggestion that the mechanism for wrapping the list of responses may not
be robust enough to avoid duplicates (and the mechanism I present is
already a LOT sturdier than 2033).

So that was my thinking. I don't see that anything is really different on
this go around either. You'll still need to satisfy the above, and by the
time you do everything that's needed to make it work 'the SMTP way' you'd
have something pretty close to this one I imagine.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

Date: Thu, 1 Feb 2007 16:46:10 -0800
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: RFC 2033 data responses for ESMTP (was: draft-hall-deferrals-00.txt)
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Tue, Jan 30, 2007, Tony Finch wrote:

> My main question about this draft is whether the extra complexity compared
> to LMTP is really useful. What is your rationale for it? I have considered
> writing a draft that simply defines how to turn on RFC 2033 data responses
> in SMTP, without specifying any significant new protocol features.

Can you write that draft? It seems there is sufficient interest to
do this.

Date: Thu, 01 Feb 2007 18:59:57 -0500
From: John C Klensin <[email-masked]>
To: [email-masked], Alexey Melnikov <[email-masked]>
cc: Claus Assmann <[email-masked]>, [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Sender: [email-masked]

--On Thursday, 01 February, 2007 14:46 -0800
[email-masked] wrote:

>> > PS: we might use an extension with a version, e.g.,
>> > 220-DRR v1 v2
>> > MAIL From:<...> DRR=v1
>> > where the version numbers correspond to those listed above,
>> > and make one of them (v1?) mandatory to implement. That
>> > might allow for "future enhancements" with "little" effort.
> 
> I really don't think this is necessary.

Let me go a little further.  Unless there is a very clear model
of how extension/enhancements would work, e.g., as to whether
Vn+1 was required to be backward-compatible with Vn, whether a
system that encountered "v99" and didn't recognize it
could/should assume v1 (or v15) and whether it would have to
tell the client it had done that, and so on, our experience is
that this sort of thing is not merely unnecessary, but harmful.
And I don't think there has been nearly enough demonstration of
likely payoff to justify the work it would take to try to define
such an extension mechanism.  

Fewer options yields more interoperable implementations.

     john

From: [email-masked]
Cc: Claus Assmann <[email-masked]>, [email-masked]
Date: Thu, 01 Feb 2007 14:46:26 -0800 (PST)
Subject: Re: draft-hall-deferrals-00.txt
To: Alexey Melnikov <[email-masked]>
Sender: [email-masked]

> Claus Assmann wrote:

> >On Tue, Jan 30, 2007, Tony Finch wrote:
> >
> >
> >>My main question about this draft is whether the extra complexity compared
> >>to LMTP is really useful. What is your rationale for it? I have considered
> >>
> >>
> >It avoids replies for those recipients that are accepted without
> >having to check the message. Is it worth it? I don't know...
> >
> >To make it simpler for people to implement "deferred RCPT reply"
> >(DRR) it would be better to simply use the LMTP model as most likely
> >several SMTP clients implement this already and hence existing code
> >can be reused.
> >
> >Can we get some consensus which way we want to go:
> >1. the already known LMTP model.
> >2. the enhanced model described in draft-hall-deferrals-00.
> >
> >
> I would happily go with the LMTP model. I think this extension is long
> overdue.

Agree with both points.

> >PS: we might use an extension with a version, e.g.,
> >220-DRR v1 v2
> >MAIL From:<...> DRR=v1
> >where the version numbers correspond to those listed above, and
> >make one of them (v1?) mandatory to implement. That might allow for
> >"future enhancements" with "little" effort.
> >
> >

I really don't think this is necessary.

				Ned

Date: Thu, 01 Feb 2007 15:05:39 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Finch <[email-masked]>
CC: "Carl S. Gutekunst" <[email-masked]>, =?ISO-8859-1?Q?Keld_J=F8rn_?= =?ISO-8859-1?Q?Simonsen?= <[email-masked]>, [email-masked]
Subject: Re: greylisting done at end of headers, or end of daya (QUIT) ?
Sender: [email-masked]

Tony Finch wrote:
> On Wed, 31 Jan 2007, Hector Santos wrote:
>> Good Guys are 100% SMTP compliant. Bad Guys are not. Thats the theory and its
>> true.
> 
> IME good guys are frequently not compliant. Broken HELO domains, invalid
> return paths, refusal to accept MAIL FROM:<>, etc. etc.
> 
> Tony.

Good point. I won't say "Frequently" myself, but sure, nothing is 
absolute.

In the market we are addressing, the anonymous one (no info to fall back 
on), we don't know who is "good person or bad person."   When I say 
"good guy" I don't necessarily mean one that is your friend.  Good in 
the sense they run a properly setup SMTP system and make no attempt to 
hide it.  You really don't know if they are "good or bad" people.

The fact is that "bad guys", the one that really aren't your friend, by 
far, are using SMTP non-complaint almost and if not, by choice in some 
form or fashion.  IMV the #1 dearth to SMTP progress is how do we deal 
with the legacy operation or one that behaves as one.  I think the day 
we decide how to deal with legacy software (in a IETF standard fashion), 
is the day we begin to make progress in the AVS market.

But sure, some of the "good guys" do have poorly setup systems, and 
guess what? For us, they fall in the same category as everyone else. 
For me, it is rare exception that I personally have to white list a node 
that for some reason or another has some "bad" state or information they 
can't do anything about.

--
HLS

Date: Thu, 01 Feb 2007 11:05:07 -0800
From: "Carl S. Gutekunst" <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: greylisting done at end of headers, or end of daya (QUIT) ?
Sender: [email-masked]

[email-masked] wrote:
> I stopped having sympathy for non-compliant good guys quite some time ago.
> There's no excuse for it in this day and age, especially for the examples
> you give.
>   

The examples I gave were relevant only to the topic at hand: the costs 
of gray listing. It's a pretty small cost, and a cost I was happy to 
accept for three years, since it was stopping spam. It's not stopping 
any spam for me any more, so now the cost -- even though it was quite 
low -- has become too high.

As far as the more general problem of  Good Guys doing bad things, I 
could spend all day on that; it's what I did for a living for the past 
2+ years. Here's as brief as I can get:

There are a small number of compliance checks that work pretty well for 
almost anyone and have low cost. My favorites include accepting mail 
only at MX hosts, rejecting mail whose bounce address doesn't resolve, 
and rejecting mail from IP addresses that don't have a PTR. Nearly all 
of the Tier 1 ISPs do the same.

There's a much larger set of compliance checks that only work for 
specific user communities. Within those communities they work well, you 
can measure it. Within other communities they'll mostly just give you 
false positives. The trick is figuring out which ones do work, and then 
monitoring them, because what worked yesterday might now work tomorrow.

Specific example: I did some research on the accuracy of several 
anti-spam tools that perform volume sampling, comparing the results with 
known volumes. In the case of DCC -- a tool I used fondly for many years 
-- I noticed that I wasn't actually measuring overall Internet E-mail 
volume; I was sampling the nature of the E-mail lists to which DCC users 
subscribe. I found, for example, that DCC users like surveys and geekish 
mailing lists. But they don't subscribe to sex lists or sweepstakes, and 
they don't buy much stuff online from general merchandisers. It was 
pretty spooky, actually, data that a targeted marking guy would have loved.

The survey folks, the geek lists, and the sex sites send the squeeky 
cleanest most compliant mail you will ever find anywhere. (In the case 
of the sex sites, they have to: they've got the "bicycle shop in China" 
problem big time.) But sweepstakes and online general merchandiser's 
compliance is terrible. A filter that relied heavily on standards 
compliance would work very well for the DCC community. But for another 
community that likes sweepstakes and buys lots of stuff online, that 
filter is going to run into a load of false positives.

<csg>

To: Tony Finch <[email-masked]>
Cc: Hector Santos <[email-masked]>, "Carl S. Gutekunst" <[email-masked]>, =?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <[email-masked]>, [email-masked]
Subject: Re: greylisting done at end of headers, or end of daya (QUIT) ?
From: [email-masked]
Date: Thu, 01 Feb 2007 10:20:37 -0500
Sender: [email-masked]

--==_Exmh_1170343237_13753P
Content-Type: text/plain; charset=us-ascii

On Thu, 01 Feb 2007 11:26:13 GMT, Tony Finch said:
> On Wed, 31 Jan 2007, Hector Santos wrote:
> > Good Guys are 100% SMTP compliant. Bad Guys are not. Thats the theory and its
> > true.
> IME good guys are frequently not compliant. Broken HELO domains, invalid
> return paths, refusal to accept MAIL FROM:<>, etc. etc.

I stopped having sympathy for non-compliant good guys quite some time ago.
There's no excuse for it in this day and age, especially for the examples
you give.

--==_Exmh_1170343237_13753P
Content-Type: application/pgp-signature

--==_Exmh_1170343237_13753P--

To: "Carl S. Gutekunst" <[email-masked]>
Cc: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: greylisting done at end of headers, or end of daya (QUIT) ?
From: [email-masked]
Date: Thu, 01 Feb 2007 10:20:52 -0500
Sender: [email-masked]

--==_Exmh_1170343252_13753P
Content-Type: text/plain; charset=us-ascii

On Wed, 31 Jan 2007 16:51:02 PST, "Carl S. Gutekunst" said:

> Let's say I just bought some software; the vendor is about to E-mail me 
> the license key. Or: I just registered at my health insurance company, 
> and they're sending me a confirmation code via E-mail that I must have 
> for access to my medical records. How am I supposed to white list those? 
> Even if I can guess what domain they're sending from, it's still a 
> manual process.

In both these cases, the mail *should* be retried in a short while *anyhow*,
and make it through on the second attempt (assuming your greylisting works
properly and they have a sane MTA that does retries).  If your greylisting
doesn't work properly, you have nobody to blame but yourself, and if they
have a non-sane MTA, they have nobody to blame but themselves.

And if you are *relying* on the fact that the liscence key or confirm code
will be shipped to you in near-real-time, and can't tolerate a requeue/retry
cycle, you probably need to re-evaluate your use of SMTP as a transport
mechanism.

--==_Exmh_1170343252_13753P
Content-Type: application/pgp-signature

--==_Exmh_1170343252_13753P--

Date: Thu, 01 Feb 2007 12:51:05 +0000
From: Alexey Melnikov <[email-masked]>
To: Claus Assmann <[email-masked]>
CC: [email-masked]
Subject: Re: draft-hall-deferrals-00.txt
Sender: [email-masked]

Claus Assmann wrote:

>On Tue, Jan 30, 2007, Tony Finch wrote:
>  
>
>>My main question about this draft is whether the extra complexity compared
>>to LMTP is really useful. What is your rationale for it? I have considered
>>    
>>
>It avoids replies for those recipients that are accepted without
>having to check the message. Is it worth it? I don't know...
>
>To make it simpler for people to implement "deferred RCPT reply"
>(DRR) it would be better to simply use the LMTP model as most likely
>several SMTP clients implement this already and hence existing code
>can be reused.
>
>Can we get some consensus which way we want to go:
>1. the already known LMTP model.
>2. the enhanced model described in draft-hall-deferrals-00.
>  
>
I would happily go with the LMTP model. I think this extension is long 
overdue.

>PS: we might use an extension with a version, e.g.,
>220-DRR v1 v2
>MAIL From:<...> DRR=v1
>where the version numbers correspond to those listed above, and
>make one of them (v1?) mandatory to implement. That might allow for
>"future enhancements" with "little" effort.
>  
>

Date: Thu, 1 Feb 2007 11:26:13 +0000
From: Tony Finch <[email-masked]>
To: Hector Santos <[email-masked]>
cc: "Carl S. Gutekunst" <[email-masked]>, =?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <[email-masked]>, [email-masked]
Subject: Re: greylisting done at end of headers, or end of daya (QUIT) ?
Sender: [email-masked]

On Wed, 31 Jan 2007, Hector Santos wrote:
>
> Good Guys are 100% SMTP compliant. Bad Guys are not. Thats the theory and its
> true.

IME good guys are frequently not compliant. Broken HELO domains, invalid
return paths, refusal to accept MAIL FROM:<>, etc. etc.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
MALIN: SOUTHWEST VEERING NORTHWEST 5 OR 6, OCCASIONALLY 7 LATER. ROUGH OR VERY
ROUGH. OCCASIONAL DRIZZLE, RAIN LATER. GOOD, OCCASIONALLY MODERATE OR POOR.