Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
From: Hector Santos <[email-masked]>
To: David MacQuigg <[email-masked]>
Cc: SMTP Interest Group <[email-masked]>
Date: Sun, 15 Aug 2010 21:06:06 -0400
List-Id: <ietf-smtp.imc.org>

David MacQuigg wrote:

> I like the original clear and concise wording best (last paragraph of 
> section 6.1).  All three say essentially the same thing about what the 
> receiver MUST do.  

But not strong enough about what is expected of the client as well. It 
could easier add a statement matching the above:

      A sender-SMTP MUST carefully balance these two
      considerations, i.e., using the recommended
      10 minute timeout, or using a non-standard shorter
      timeout against receiver-SMTP who might have a delay
      after the final <CRLF>.<CRLF> and getting the reply
      code.

I mean, it takes two to tango.  When clients follow the "rules," it 
wouldn't become much of an issue. :)  Servers will always do its best 
to make sure the time is minimal.

Keep in mind sometimes its just a matter of using OOPS software, with 
virtual functions and they used a flat 5 minute timeout for each 
state.  I've seen this in various software.  I've seen it again 
recently in a CodeProject.Com SMTP Client C/C++ submission, nice OOPs 
code, but a used a file timeout and single source location for 
catching error timeout exceptions using a single 5 minute values.  I 
pointed out the SMTP compliancy error to the code submitter.

Generally, server software is more robust. SMTP client software is a 
dime a dozen and many just need reminders there are certain guidelines 
to follow.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sun, 15 Aug 2010 08:28:13 -0700
From: David MacQuigg <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

John C Klensin wrote:

> (1) My recommendation would be to retain the timeout, but note
> that server implementations should be aware that some clients
> will ignore the spec for operational reasons and apply a much
> smaller number.  My guess is that a discussion of the
> appropriateness of doing that doesn't belong in an erratum/
> corrigendum to 5321 but in a separate document (if at all).
>
> I note, again fwiw, that I've been trying to get various
> advocates for a ban (or near-ban) on NDNs to write that separate
> document and propose a specific model at regular intervals since
> well before 2821 was completed.
>
>
> (2) How would people feel about moving toward changing the
> "substantive text" to read something like:
>
> 	"To avoid receiving duplicate messages as the result of
> 	timeouts, a receiver-SMTP MUST carefully balance the
> 	time required to respond to the final <CRLF>.<CRLF> end
> 	of data indicator against the desirable goal of
> 	rejecting undeliverable or unacceptable messages at SMTP
> 	time rather than generating NDN messages later. "
>
> I'll have to check whether "NDN messages" is the right reference
> at that point, but it seems to me from the discussion that the
> above should be about the right intent.   We could go even a
> little further and recast the paragraph as something like:
>
> 	"Long delays after the <CRLF>.<CRLF> is received can
> 	result in timeouts and duplicate messages.  Deferring
> 	detailed message analysis until after the SMTP
> 	connection has closed can result in non-delivery
> 	notifications, possibly sent to incorrect addresses.  A
> 	receiver-SMTP MUST carefully balance these two
> 	considerations, i.e., the time required to respond to
> 	the final <CRLF>.<CRLF> end of data indicator and the
> 	desirable goal of rejecting undeliverable or
> 	unacceptable messages at SMTP time."
>   

I like the original clear and concise wording best (last paragraph of 
section 6.1).  All three say essentially the same thing about what the 
receiver MUST do.  The issue of avoiding NDNs is controversial, and 
should be worked out in a separate document, as you suggest.

************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

Date: Sun, 15 Aug 2010 06:17:01 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "Carl S. Gutekunst" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

+1.

I made this note to hopefully correlate the idea that AV vs AS can 
provide different level of overhead at the DATA level.  IMV, AV can 
serve as a system-wide rejection with less per-recipient concerns - no 
one gets the virus, don't leave this to the user to decide is probably 
a better policy. AS is more subjective on a per-recipient basis thus, 
IMV, harder to design for at the DATA level - unless you have a 
system-wide view of it.

All that does have a direct optimization correlation at DATA.

Thanks

-- 
Sincerely

Hector Santos
http://www.santronics.com

Carl S. Gutekunst wrote:
> 
> [Is this too far off topic?]
> 
> Hector Santos wrote:
>> We are at a point where an ISP or rather a MSP (Mail Service Provider) 
>> not having AV software installed could be considered as a liability. I 
>> recall reading (a BNA Legal and Business newsletters) about a proposed 
>> Canadian bill that would make it a requirement otherwise fines and 
>> shutdowns can be imposed in return ISP liabilities will be reduced. I 
>> have not followed up to tell you if it was passed or not.
>>
>> I think there is a difference in the V and S in AVS.  AV is something 
>> that can system wide. AS is more subjective.
> 
> Anti-virus has a considerable set of cost, intrusiveness, risk, and 
> privacy tradeoffs that motivate service providers to give customers some 
> choices. Customers who have a small budget and reasonably savvy users 
> might want just a single filter, perhaps ClamAV. Customers who are very 
> risk adverse would be more likely to pay for using multiple engines. 
> Customers who are extremely risk adverse and less sensitive about 
> privacy (or more trusting of the provider) might want heuristic filters 
> that aggressively flag "suspicious" mail that then requires sandboxing 
> or human inspection for a final verdict.
> 
> And any time you give the users choices, you run the risk of a 
> disposition conflict. Per-recipient status after the dot would fix that.
> 
> <csg>
> 
> 
> 

Date: Sun, 15 Aug 2010 01:05:24 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

[Is this too far off topic?]

Hector Santos wrote:
> We are at a point where an ISP or rather a MSP (Mail Service Provider) 
> not having AV software installed could be considered as a liability. I 
> recall reading (a BNA Legal and Business newsletters) about a proposed 
> Canadian bill that would make it a requirement otherwise fines and 
> shutdowns can be imposed in return ISP liabilities will be reduced. I 
> have not followed up to tell you if it was passed or not.
>
> I think there is a difference in the V and S in AVS.  AV is something 
> that can system wide. AS is more subjective.

Anti-virus has a considerable set of cost, intrusiveness, risk, and 
privacy tradeoffs that motivate service providers to give customers some 
choices. Customers who have a small budget and reasonably savvy users 
might want just a single filter, perhaps ClamAV. Customers who are very 
risk adverse would be more likely to pay for using multiple engines. 
Customers who are extremely risk adverse and less sensitive about 
privacy (or more trusting of the provider) might want heuristic filters 
that aggressively flag "suspicious" mail that then requires sandboxing 
or human inspection for a final verdict.

And any time you give the users choices, you run the risk of a 
disposition conflict. Per-recipient status after the dot would fix that.

<csg>

X-Mail-Sent-By-AEGEE.org-Account: didopalauzov
Date: Sat, 14 Aug 2010 21:58:23 +0200
From: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

Hello,

May I hear other opinions, if what I write is off-topic or not?

I replied in one email to the notes, I wanted to reply.  Not all 
original citations came from David.

On 10.08 he wrote:
 > The only advantage I see in running the spam filter before ACCEPT is 
a few less messages at the bottom of my spam bucket.

On 14.08 he posted:
 > While I think it would be a good thing to send a notice of some sort
when a message is not whitelisted, we cannot do that now because of
the "near ban" on NDNs

This can be avoided by running the spam filter before ACCEPT after 
CRLF.CRLF and SMTP reject the unwanted messages.

 >  ... we do not decide what goes in an individual recipient's 
quarantine folder.  All our Receiver does is tag the message with a spam 
score.  The recipient decides whether to a maintain a separate folder 
with high spam scores (a quarantine), where to set the spam/ham 
threshold, and how much time to spend reviewing messages with a high 
spam score.

If the recipients decides to maintain a separate folder with high spam 
score, this might lead to false positives -- too late read mails, or 
even overseen messages.  Of course, every such delayed reading is caused 
by the users, not by the administrators, but the users might prefer SMTP 
rejections instead of checking the spam folders.

Със здраве
   Дилян

On 14.08.2010 21:25, David MacQuigg wrote:
> Dilyan,
>
> Looks like there is quite a bit of miscommunication here, even some
> responses to stuff I didn't write. Let's continue this discussion in a
> forum where we won't be adding off-topic chatter. I've copied your last
> post to the following forum, and replied to it there.
>
> http://groups.google.com/group/registry_internet/browse_frm/thread/f9115085b6668652/03123968449ee555#03123968449ee555
>
>
> If that link doesn't work, go to
> http://groups.google.com/group/registry_internet and click on the
> discussion "Problems with post-processing and DSNs".
>
> Anyone from this list who is interested is welcome to join us.
>
>
> Дилян Палаузов wrote:
>
> <snip many pages>
>>
>> By the way, in the Russian newspapers you can read job offers for
>> marketing agents. Their job is to send mails using free mail providers
>> to a provided long lists of recipients by copy&paste the list of
>> addresses [= to send spam]. Thus, if the message comes from an A-rated
>> domain, it does not mean the message is ham.
>
> I no longer argue whether it is possible for free mail providers to
> block outgoing spam, I just take advantage of the fact that most of them
> do. None of our A-rated domains send spam. Zero.
>
> -- Dave
>

Date: Sat, 14 Aug 2010 12:25:59 -0700
From: David MacQuigg <[email-masked]>
To: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

Dilyan,

Looks like there is quite a bit of miscommunication here, even some 
responses to stuff I didn't write.  Let's continue this discussion in a 
forum where we won't be adding off-topic chatter.  I've copied your last 
post to the following forum, and replied to it there.

http://groups.google.com/group/registry_internet/browse_frm/thread/f9115085b6668652/03123968449ee555#03123968449ee555

If that link doesn't work, go to 
http://groups.google.com/group/registry_internet and click on the 
discussion "Problems with post-processing and DSNs".

Anyone from this list who is interested is welcome to join us.

Дилян Палаузов wrote:

<snip many pages>
>
> By the way, in the Russian newspapers you can read job offers for 
> marketing agents.  Their job is to send mails using free mail 
> providers to a provided long lists of recipients by copy&paste the 
> list of addresses [= to send spam].  Thus, if the message comes from 
> an A-rated domain, it does not mean the message is ham.

I no longer argue whether it is possible for free mail providers to 
block outgoing spam, I just take advantage of the fact that most of them 
do.  None of our A-rated domains send spam.  Zero.

-- Dave

Date: Sat, 14 Aug 2010 10:38:27 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: IETF SMTP list <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

[email-masked] wrote:

>> So if they've decided it's spam, what's the problem with 
>> throwing it away?
> 
> The same problem as always: False positives. While I think the various
> admonitions we have in various documents that try and discourage 
> silently dropping messages are unrealistic, the fact remains that 
> silent draps create reliability issues. Rejects address this issue and 
> are preferable, as long as they don't create additional issues.
> 
>                 Ned

+1.

As a public note for the unaware, the new RFC 5321 for the first time 
ever, has "cracked opened the door" with new SMTP semantics allowing 
or provides a rationalization for discarding of mail under 
"reasonable" circumstances.

IMV, having new official RFC mail protocols specifically designed for 
this purpose is rational and reasonable, such as RFC 5617 (ASDP) with 
its DKIM=DISCARDABLE policy. As long as the ADSP domain publicly 
exposes this self-asserted policy, I don't think it contributes to 
false positives discarding of accepted but zero false positive proof 
for ADSP violating mail.

However, one implementation question is when a DATA level ADSP based 
(55x) rejection can also serve the same purpose for ADSP domains.

Currently this only seems to be a problem for DKIM resigners ignorant 
of errant or spoofed ADSP protected mail submissions.  Murray's Mail 
List Manager (MLM) draft proposal hopes to provide, IMV, "sound 
protocol consistency" guidelines in this area for DKIM+ADSP compliant 
List Server and SMTP receiver developers to consider.

-- 
Sincerely

Hector Santos
http://www.santronics.com

X-Mail-Sent-By-AEGEE.org-Account: didopalauzov
Date: Sat, 14 Aug 2010 15:58:44 +0200
From: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

Hello,

On 11.08.2010 21:36, David MacQuigg wrote:
> Дилян Палаузов wrote:
>  >
>  > On 10.08.2010 20:18, David MacQuigg wrote:
>  >> My system is too small to draw any conclusions, but my impression is
>  >> that we are overly-concerned about a very small fraction of the
> mailflow
>  >> - those few messages that are legit, but get a false reject from a
>  >> statistical filter. We really don't care about the proper SMTP response
>  >> if it is spam. The only advantage I see in running the spam filter
>  >> before ACCEPT is a few less messages at the bottom of my spam bucket. I
>  >> wish I had some numbers from a larger system.
>  >
>  > A user might have installed redirect for her incoming mails to
> another mail provider. The other provider might SMTP reject incoming
> spam. If there is spam for your user, your server does accept it, then
> sends it to the other mail provider, and that second server SMTP rejects
> it, then your server will have to send DSN for that SPAM (provided that
> you do not want to discard emails). With rejection-preferences similar
> on both servers, the amount of such DSNs will decrease, when both
> servers reject after DATA. You have no influence on what other servers
> do with the spam (smtp rejecting or inserting in spam folders).
>
> We have quite a bit of influence. We insist that our recipients turn off
> all spam filtering, chain-forwarding, SPF checks, whatever might
> conflict with our acting as Receiver for their box67.com email address.
> Typically, they will have a private mailbox known only to our
> Receiver/Forwarder. Failures resulting from SMTP rejects, DSNs, or any
> other problem at the recipient's mailbox are treated the same as a
> mailbox being suddenly offline. Incoming mail to that address is
> rejected until we can resolve the problem with the recipient.

I do not think it is good to insist on such things.  One might want to 
communicate to a sender, that her mailbox does not exist anymore and 
SMTP reject the mails from that sender.  Your service will then 
conclude, that destination mailbox is suddenly offline, but it is not. 
[I used to receive a newsletter from my mobile operator, that contained 
no instructions how to unsubscribe and writing replies has not 
unsubscribed me.  The fastest solution (for me) was to instruct my 
mailbox to SMTP reject mails with that newsletter).

>  >> Would be nice if SMTP had a 'conditional ACCEPT': "Sorry, your
>  >> transmitter ID 'yahoo.com' cannot be verified, Your message has been
>  >> routed to the recipient's quarantine." Imagine the effect that would
>  >> have on all the lame excuses for an invalid or hard-to-verify HELO
> name.
>  >
>  > What do you want to communicate to the sender with this conditional
> ACCEPT: "Dear Sender, your suspicious email has arrived. It might be
> read soon, it might be read late, or it might not be read at all."?
>
> No, the message I suggested had a specific reason for the failure. It
> should also have a link to a webpage with instructions on how to fix the
> failure, like what we have now on REJECT messages:
> http://open-mail.org/3strikes/REJECT.html

The link above shall be send with SMTP permanent reject (not conditional 
accept).

My rewording does not change the meaning of David`s message -- the 
sender wants to know if the message is delivered, or not; and the 
message "Sorry, your transmitter ID 'yahoo.com' cannot be verified, Your 
message has been routed to the recipient's quarantine." does not clarify 
this.

>  > The problem with the spam folders are the false positives. Many users
> do check the spam folders/quarantines not that often as their Inbox and
> with less attention. This might lead to reading a message too late, or
> even overseeing it. I prefer to immediately notify the sender that her
> message was not delivered, including in the SMTP response alternative
> ways to contact the recipient (and spamassassin-evaluation of the mail),
> rather than let the sender hope that her email was properly delivered,
> while there is a risk that the recipient oversees that message.
>
> False rejects are indeed the main problem with modern email systems, but
> this problem isn't solved by rejecting a small fraction of messages with
> the highest spam scores. There are still plenty of messages, almost all
> spam, that get a score low enough to be kept in quarantine, and there is
> always a risk that the recipient will overlook a legitimate message. The
> only good solution to the false reject problem is to offer senders a way
> to avoid the statistical filters entirely. That is what a reputation
> system does. When a message comes from yahoo.com (or any other A-rated
> domain), it goes straight to our recipient's inbox.

Informing the sender, that her message was evaluated as spam and is 
therefore returned, including information how to alternatively contact 
the recipient, is also good solution. Or rather: giving the mailbox 
owners the one [smtp spam reject] or the other [spam folders] option is 
a good solution.

By the way, in the Russian newspapers you can read job offers for 
marketing agents.  Their job is to send mails using free mail providers 
to a provided long lists of recipients by copy&paste the list of 
addresses [= to send spam].  Thus, if the message comes from an A-rated 
domain, it does not mean the message is ham.

In article <[email-masked]> you write:
 >>> Hello,
 >>>
 >>> if rejecting after CRLF.CRLF is better than bouncing, what about
 >>> discussing draft-hall-prdr-00 and doing it more normative? ...
 >>
 >>I never understood why the interest level in this was so low.
 > It seems much more likely in that scenario that rather than 
rejecting, a system would drop it in the second recipient's spam folder, 
or if they're reasonably confident in their filters, just not deliver it 
to the second recipient.

The sender shall be informed, that only the second recipient has not 
received the message.  Just not deliver to the second recipient is not 
good enough.  I hate when mails are discarded:

This year I got a mail from a user, who has not received the mails 
forwarded from my server.  I replied him to check in his spam folder. 
In two weeks the same happened.  In two more weeks, when he wrote me 
again, I called him and figured out, that my answers were discarded and 
have not reached him (they were not in the spam folder, they were 
discarded).

A propos spam folders:  What happens when a spam is sent to a mailing 
list, where everybody can post (so the decision can be made after DATA). 
  Mailing lists do not have spam folders.  You either SMTP reject the 
message, or accept it, and later wrap it and send it to the list 
moderators [causing them manual work], or send it back to the indicated 
sender [not good].  If this message is sent in the same envelope to a 
user with settings, which permit delivering her the message, then there 
is no choice, but to accept the message and bounce it [or accept for the 
first recipient, and TEMP fail for all the rest; when the sender replies 
later, do the same and so on, until no recipients are left -- accept 
only one recipient per delivery try].

 > So if they've decided it's spam, what's the problem with throwing it 
away?

False positive.

Със здраве
   Дилян

Date: Sat, 14 Aug 2010 09:47:18 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

Ray Bellis wrote:

>> That's certainly true, although uncommon in my own experience 
>> because providers make operational choices  to avoid it. As 
>> much as possible, the ability to enable or disable a service is 
>> kept at the same (or higher) granularity than the MX names.
> 
> Yes, that would have been nice.
> 
> Unfortunately, where we had this (at a previous job) we had some 
> customers on the generic "customer of this ISP" domain that were 
> paying for AV, and some that weren't.
> 
> Simply giving them all the service wasn't possible, as the AV engine 
> was licensed on a per-user basis.
> 
> We also had to operate on the basis that AV and anti-spam services
> could only be provided by explicit consent, since doing otherwise 
> could fall foul of our equivalent of "common carrier".

We are at a point where an ISP or rather a MSP (Mail Service Provider) 
not having AV software installed could be considered as a liability. I 
recall reading (a BNA Legal and Business newsletters) about a proposed 
Canadian bill that would make it a requirement otherwise fines and 
shutdowns can be imposed in return ISP liabilities will be reduced. I 
have not followed up to tell you if it was passed or not.

I think there is a difference in the V and S in AVS.  AV is something 
that can system wide. AS is more subjective.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 13 Aug 2010 21:17:05 +0200
From: Alessandro Vesely <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

Steve Atkins wrote:
> They're required to hold for ten minutes now. That won't often be needed,
> but they're required to do so if the SMTP server feels so inclined.
>
> The actual length of time an SMTP server will require an SMTP client to 
> wait while it delivers the message is going to be driven by the operational 
> requirements of the SMTP server (whether that be content-based analysis, 
> internal proxying or even just waiting for an IO slot to write the message to 
> disk), not by any RFC limits.

While that's correct, the annoyance occurs because it is the last step before 
accepting the message.  SMTP extensions can be used to anticipate certain 
actions; for example, draft-vesely-vhlo fantasizes about retrieving DKIM keys 
at the beginning of a transaction.  Unlike internal/local actions, external 
lookups may affect timeouts beyond an admin's control.

> But that open connection consumes a significant amount of resources at 
> both the sender and the receiver, so there's already a significant 
> pressure to reduce it at large receivers. So I wouldn't expect it to 
> increase without limit.

Introducing reputation systems is not going to reduce it.

> But if there's a choice between occasionally keeping a connection open for 
> another minute (possibly while you wait for other attempted deliveries 
> from the same sender) and accepting mail for delivery that will then need 
> to be bounced or discarded any decent engineer is going to do anything 
> they can to avoid accepting that mail until they've made a delivery 
> decision.

+1

> I don't believe that legitimate senders of email are likely to commonly 
> see really long delays at the end of data, and I don't much care if other 
> senders are inconvenienced.

Caching naturally favors /large/ senders, irrespectively of legitimacy.

Date: Fri, 13 Aug 2010 14:03:52 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

Peter J. Holzer wrote:

>> In our implementation, we use CBV (Callback Verification) and this  
>> resolves at least 50%, 70% to even has high as 90% of the "bad" MAIL  
>> FROM: problem. Currently it is among the highest filter in our suite of 
>> SMTP filters.
> 
> Apart from other objections agains CBV, this only removes those cases
> which were mostly harmless in the first place: If the forged sender
> doesn't exist, the NDN cannot be delivered and will be silently
> discarded (or sent to a local "double bounce" address where they will
> probably be ignored ;-)). If the forged sender *does* exist, CBV won't
> detect that it is forged and an NDN may be sent to the hapless victim of
> the forgery. 

But pete, we don't want to accept mail, create bounces and a create 
larger outgoing queue overhead than necessary.  The whole point is to 
reduced payload transfers, acceptance and bounce overheads, even it 
they will quickly fail.  If you don't have to send a bounce, why do it?

> SPF, DKIM, BATV, etc. do a better job guarding against
> address forgery.

Can't comment about BATV.

IMV, DKIM is pretty useless without some sort of domain protection 
technology wrapper such as policy, unless the suggestion is that a 
DKIM validation sans policy failure has a payoff?  Current DKIM mail 
capturing shows a pretty high volume of spammers.

SPF, +1, works great iff the site is using a hard failure policy. 
Relaxed policy is wasted overhead, what value does a "Maybe, I don't 
know." result have?  Why bother?  Relaxed provision in these protocols 
just add to confusion, ambiguity, derivatives, and add receiver 
overhead waste, solves nothing.

But I do see many soft/neutral SPF results rejected by a follow up 
CBV. :) Let me see if I can find one in today's log...., just within 
the hour:

20100813 13:01:07 testorder  : FLT RBL SPF CEP CBV
20100813 13:01:11 sapfilter  : pass (time:4406)
20100813 13:01:11 saprbl     : testing 70.112.63.202.sbl.spamhaus.org
20100813 13:01:12 saprbl     : testing 70.112.63.202.bl.spamcop.net
20100813 13:01:12 saprbl     : pass (time:401)
20100813 13:01:12 sapspf     : v=spf1 include:spf-a.hotmail.com
                                       include:spf-b.hotmail.com
                                       include:spf-c.hotmail.com
                                       include:spf-d.hotmail.com
 
include:_spf-ssg-a.microsoft.com ~all
20100813 13:01:12 sapspf     : softfail (time:520)
20100813 13:01:12 sapcep     : disabled
20100813 13:01:12 sapcbv     : MX lookup: msn.com
20100813 13:01:12 sapcbv     : total mx records: 5
20100813 13:01:12 sapcbv     : mx: 65.55.37.120     mx4.hotmail.com
20100813 13:01:12 sapcbv     : mx: 65.55.37.88      mx4.hotmail.com
20100813 13:01:12 sapcbv     : mx: 65.55.92.184     mx4.hotmail.com
20100813 13:01:12 sapcbv     : mx: 65.55.92.168     mx4.hotmail.com
20100813 13:01:12 sapcbv     : mx: 65.54.188.72     mx4.hotmail.com
20100813 13:01:12 try mx     : mx4.hotmail.com ip: 65.55.37.120
20100813 13:01:12 # connecting to 65.55.37.120
20100813 13:01:13 S: 220 col0-mc4-f39.Col0.hotmail.com Sending ......
20100813 13:01:13 C: NOOP WCSAP v2.11 Wildcat! Sender Authentication 
Protocol
20100813 13:01:13 S: 250 OK
20100813 13:01:13 C: HELO [208.247.131.15]
20100813 13:01:13 S: 250 col0-mc4-f39.Col0.hotmail.com (3.11.0.113) 
Hello [208.247.131.15]
20100813 13:01:13 C: MAIL FROM:<>
20100813 13:01:13 S: 250 <>....Sender OK
20100813 13:01:13 C: RCPT TO:<[email-masked]>
20100813 13:01:13 S: 550 Requested action not taken: mailbox unavailable
20100813 13:01:13 C: QUIT
20100813 13:01:13 sapcbv     : 550
20100813 13:01:13 finaltest  : CBV GlobalResult=0 CodeResponse=550
20100813 13:01:13 result     : reject (0)
20100813 13:01:13 smtp code  : 550
20100813 13:01:13 reason     : Rejected by WCSAP CBV

Relaxed SPF policy is a big waste of receiver resources.

Sure, the bounce would of catch this as a first try, but that would of 
meant payload transfer and acceptance.  Lets not forget the savings in 
AV checking  the DATA state or post smtp.  Lets not forget about value 
of not passing the email to the user which was OK for this transfer so 
a user would of saw this.

IMV, it is better to reject at SMTP as best you do so so, rather than 
passing on potentially harmful email to users.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 13 Aug 2010 18:15:09 +0200
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: NDNs considered harmful
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--SkvwRMAIpAhPCcCJ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2010-08-12 12:58:51 -0400, Hector Santos wrote:
> Paul Smith wrote:
>>  On 12/08/2010 14:28, Rosenwald, Jordan wrote:
>>> True statement, but that means the senders of the other 5% are now left
>>> in the dark as to what happened to their mail.
>>> Is there a proposed solution to that?
>>
>> Maybe we just recommend sending NDNs to people if their email is DKIMed=
=20
>> or if it came from a server matching SPF rules, or if the return path =
=20
>> uses BATV
>>
>> It's better than never sending them at all, and those provisions make=20
>> it reasonably certain that the sender's email address wasn't forged.
>>
>> Also, it might encourage people to put in place the anti-forgery methods.
>
> +1
>
> In our implementation, we use CBV (Callback Verification) and this =20
> resolves at least 50%, 70% to even has high as 90% of the "bad" MAIL =20
> FROM: problem. Currently it is among the highest filter in our suite of=
=20
> SMTP filters.

Apart from other objections agains CBV, this only removes those cases
which were mostly harmless in the first place: If the forged sender
doesn't exist, the NDN cannot be delivered and will be silently
discarded (or sent to a local "double bounce" address where they will
probably be ignored ;-)). If the forged sender *does* exist, CBV won't
detect that it is forged and an NDN may be sent to the hapless victim of
the forgery. SPF, DKIM, BATV, etc. do a better job guarding against
address forgery.

	hp

--=20
   _  | Peter J. Holzer    | Openmoko has already embedded
|_|_) | Sysadmin WSR       | voting system.
| |   | [email-masked]         | Named "If you want it -- write it"
__/   | http://www.hjp.at/ |  -- Ilja O. on [email-masked]

--SkvwRMAIpAhPCcCJ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--SkvwRMAIpAhPCcCJ--

Date: Fri, 13 Aug 2010 15:55:48 +0200
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Call Back Verification (was: NDNs considered harmful)
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--9jxsPFA5p3P2qPhR
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2010-08-13 08:18:52 -0400, Rich Kulawiec wrote:
> On Thu, Aug 12, 2010 at 08:36:08PM -0400, Hector Santos wrote:
> > I don't see how it allows spammers to bypass security measures.=20
>=20
> That's probably because you haven't read the original source material
> that I referenced. [1]

That might be because that source was a) vague ("the spam-l archives" -
I don't know the spam-l list, but on any moderately busy mailing list a
reference to "the archives" is mostly useless - you would have to
reference a specific thread or at least a (rather short) period of time)
and b) not public - I assume that
http://spam-l.com/mailman/private/spam-l/ is the archive, and I cannot
access it without a password).

> [1] For example, and this condensed outline of just one of many possible
> scenarios is NOT a substitute for reading the original source material:
> consider what happens when an abuser registers a throwaway domain and
> points the MX's for it at the victim's MX's, then uses a few million
> zombies to simultaneously send traffic putatively from that throwaway
> domain to mail servers which use callbacks.

That would be a DDoS attack, but I don't see how it "bypasses security
measures".

	hp

--=20
   _  | Peter J. Holzer    | Openmoko has already embedded
|_|_) | Sysadmin WSR       | voting system.
| |   | [email-masked]         | Named "If you want it -- write it"
__/   | http://www.hjp.at/ |  -- Ilja O. on [email-masked]

--9jxsPFA5p3P2qPhR
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--9jxsPFA5p3P2qPhR--

Date: Fri, 13 Aug 2010 09:41:44 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

Rich Kulawiec wrote:
> [ On-list or off-list replies only, please; never both. ]

[Do us a favor and present your "list etiquette" note to most of the 
posters here as well. I generally try to prune all the unnecessary 
addresses but forgive the few oops.]

> On Thu, Aug 12, 2010 at 08:36:08PM -0400, Hector Santos wrote:
>> The valid return path is a SMTP requirement and it MUST be valid at
>> the time it is issued by the sender.  Testing it is a VALID option
>> to perform for the simple reason ERROR REPORTING is a required
>> possibility.  It must not be invalid.
> 
> Welcome to 2010. 

Huh?  This isn't new stuff or issues.

> That's all very nice, but practical reality
> overrides it. 

No it doesn't. Following the SMTP recommendations generally worked the 
best.

> The RFCs allow many things, but some of those
> are no longer a very good idea, and some of them are downright
> dangerous.  

I have no problem with the SMTP specification, except when there are 
relaxed provisions (in any protocol) which allowed for abuse, 
ambiguity and most of the issues we have today.

> Or to put it another way, de facto operational practice
> supersedes de jure specifications indicating what we *could* do,
> if we were freed of all constraints imposed by reality.

I get your point, but I don't agree with you. I don't think the 
industry would of survive this long if there was not a solid set of 
inherent and expected protocol consistency followed by all automated 
software.

In todays highly abusive "bad guy" age, one fundamental idea is 
following SMTP compliancy which works extremely well to filter a good 
chuck of the bad actors. That includes return path verification.

>> I don't see how it allows spammers to bypass security measures. 
> 
> That's probably because you haven't read the original source material
> that I referenced. 

I probably didn't need too because u said nothing knew. But I am sure 
I have in the past and I just didn't agree. Usually the same 
characters with one sided opinions.

> [1]  Extensive analysis of this technique was done
> during the summer of 2004 on spam-l, when we caught the incompetent
> morons at Verizon doing it. 

Verizon had a poor implementation that I recalled allowed for 
potential loops.  It wasn't a good implementation and poor case for 
judging.

> That analysis conclusively demonstrates that
> callbacks support, enable and facilitate spam and DoS/DDoS attacks.

Verizon.NET gets attacks for all sorts of reasons simply because of 
who they are. CBV was not the culprit.

> This is so well established, by the way, that it's a BCP to blacklist
> those caught doing it, and there are multiple public and private resources
> which do exactly that.

This would suggest bounces in general are subject to blacklisting. 
However, I will note an instance where it was personal issue to block 
people they didn't like.

> More generally, any putatively defensive technique which permits unknown
> third parties to generate outbound traffic from *your* operation to
> arbitrary destinations of *their* choosing is clearly a very bad idea.

Again, that would imply that a general bounce is prone to a high 
incident of blacklisting and that is simply not the case.  We would 
have major problems is that was the case.  That is not to suggest that 
many system do not count or analysis the bounce sources.  But to black 
one is generally because of abuse and in the case of CBV - poor 
implementations.

> ---Rsk
> 
> [1] For example, and this condensed outline of just one of many possible
> scenarios is NOT a substitute for reading the original source material:
> consider what happens when an abuser registers a throwaway domain and
> points the MX's for it at the victim's MX's, then uses a few million
> zombies to simultaneously send traffic putatively from that throwaway
> domain to mail servers which use callbacks.  This is not a theoretical
> attack, by the way.

Well, first of all, I am not advocating any CBV standardization or 
across the board adoption. It is a implementation just like any other 
non-standard implementations of various anti-forgery techniques in 
play by various systems.

Second, seven years of implementation with a few thousand 
installations has shown a high popularity and very successful case of 
usage with little to no repercussions.

Third, there were early issues that were learned and required fine 
tuning. honeypots, for example.

Forth, in our implementation, CBV is the last test in a suite of tests.

     RBM (operator defined rule-based message filters)
     DNSRBL
     SPF
     CBV

Fifth, Caching is important.

Lastly, the above scenario is not specific to dealing or a root cause 
by CBV, but a general issue across the board.  Lacking any 
anti-forging techniques, you are vulnerable to a higher degree with 
your BOUNCE requirements - that was the problem created by SORBIG Dual 
Blitz DoD attacks in 2003 that finally work up the industry and begin 
the efforts to find and invent new anti-spam email techniques .  These 
attacks have to be dealt with regardless if you have CBV or not.

Look, I am sticker for optimization. I would be the first to throw 
away CBV or any other idea if it didn't have value and was a major 
overhead issue.  Taking your advice, pointing to SPAM-L is just not 
convincing enough, wasn't back in 2003, nor 2004 with the Verizon.NET 
version, although I do recall it was poor, and hasn't been since that 
point on.

Other software also support CBV out of the box, I believe Exim and I 
believe there is SendMail CBV Milter and there are Microsoft SMTP 
ActiveX tools, etc, etc.  Clearly, they are not all wrong in believe 
in an method that does work.

With that said, will I advocate a standard for this?  No. Its a KLUDGE 
and wouldn't work well in a widely adopted manner.  Furthermore, no 
extension will make it work because a key reason why CBV does works is 
because it deals with legacy SMTP client issues which in my view, is 
the #1 problem in addressing the spam dilemma.

Thanks for your comments and input.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 13 Aug 2010 14:18:19 +0100
From: Tony Finch <[email-masked]>
To: Steve Atkins <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

On Thu, 12 Aug 2010, Steve Atkins wrote:
>
> >  The growth of sending per-recipient tailored messages seemed to be
> > the cause.
>
> Only in higher-end mailing list managers.

A significant number of my users send per-recipient customized mailshots
to alumni using a Microsoft Office mailmerge facility.

The mean number of recipients we get on mail to our MXs is 1.1. The
overall number (for both internal and external mail) is higher, about 1.3,
because of multi-recipient message submission and because of our internal
mailing list server. This mean is much lower at the moment than it is
during term.

Here's a historical graph which shows the proportion has not changed very
much over the last several years. MsgIn and MsgOut count accepted and
delivered messages respectively. XRcpt counts additional deliveries, i.e.
it's the 0.3 part of the 1.3 mean above. The lump in 2007-2008 is due to
poor spam filtering.
http://canvas.csi.cam.ac.uk/stats/ppsw/img/rawtraffic.2160days.gif

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
MALIN HEBRIDES: NORTHWESTERLY, BACKING SOUTHWESTERLY FOR A TIME, 4 OR 5,
OCCASIONALLY 6 AT FIRST IN SOUTHEAST MALIN. SLIGHT OR MODERATE. RAIN LATER.
GOOD, OCCASIONALLY MODERATE.

Date: Fri, 13 Aug 2010 08:18:52 -0400
From: Rich Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

[ On-list or off-list replies only, please; never both. ]

On Thu, Aug 12, 2010 at 08:36:08PM -0400, Hector Santos wrote:
> The valid return path is a SMTP requirement and it MUST be valid at
> the time it is issued by the sender.  Testing it is a VALID option
> to perform for the simple reason ERROR REPORTING is a required
> possibility.  It must not be invalid.

Welcome to 2010.  That's all very nice, but practical reality
overrides it.  The RFCs allow many things, but some of those
are no longer a very good idea, and some of them are downright
dangerous.  Or to put it another way, de facto operational practice
supersedes de jure specifications indicating what we *could* do,
if we were freed of all constraints imposed by reality.

> I don't see how it allows spammers to bypass security measures. 

That's probably because you haven't read the original source material
that I referenced. [1]  Extensive analysis of this technique was done
during the summer of 2004 on spam-l, when we caught the incompetent
morons at Verizon doing it. That analysis conclusively demonstrates that
callbacks (that is: callbacks to external sites, not call forwards to
internal sites, which are quite useful in some cases) support, enable
and facilitate spam and DoS/DDoS attacks.

This is so well established, by the way, that it's a BCP to blacklist
those caught doing it, and there are multiple public and private resources
which do exactly that.

More generally, any putatively defensive technique which permits unknown
third parties to generate outbound traffic from *your* operation to
arbitrary destinations of *their* choosing is clearly a very bad idea.

---Rsk

[1] For example, and this condensed outline of just one of many possible
scenarios is NOT a substitute for reading the original source material:
consider what happens when an abuser registers a throwaway domain and
points the MX's for it at the victim's MX's, then uses a few million
zombies to simultaneously send traffic putatively from that throwaway
domain to mail servers which use callbacks.  This is not a theoretical
attack, by the way.

Subject: Re: per user post-data rejects, Processing after the end of DATA
From: Nathaniel Borenstein <[email-masked]>
Date: Fri, 13 Aug 2010 07:33:49 -0400
Cc: [email-masked], [email-masked]
To: "Carl S. Gutekunst" <[email-masked]>
Sender: [email-masked]

On Aug 12, 2010, at 7:21 PM, Carl S. Gutekunst wrote:

> Wow. Not in my experience, definitely not for ordinary B-to-B, where the median is probably over 1.5, and not even for legit bulk.

Unless my math is even rustier than I think, you probably mean the mean.  The only way the median could be 1.5 would be if exactly half of all the RCPT-TO's had one recipient, while the rest had more. Even then, the next message to arrive would push the median to either 1.0 or 2.0.

From: Ray Bellis <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Date: Fri, 13 Aug 2010 07:48:27 +0000
Accept-Language: en-GB, en-US
Sender: [email-masked]

> 
> That's certainly true, although uncommon in my own experience because providers make operational choices  to avoid it. As much as possible, the ability to enable or disable a service is kept at the same (or higher) granularity than the MX names.

Yes, that would have been nice.

Unfortunately, where we had this (at a previous job) we had some customers on the generic "customer of this ISP" domain that were paying for AV, and some that weren't.

Simply giving them all the service wasn't possible, as the AV engine was licensed on a per-user basis.

We also had to operate on the basis that AV and anti-spam services could only be provided by explicit consent, since doing otherwise could fall foul of our equivalent of "common carrier".

Ray

Date: Fri, 13 Aug 2010 00:11:55 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Ray Bellis <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

Ray Bellis wrote:
>> An AV that's configured with different rules for different recipients.
>>     
>
> Or one that's only configured for the customers that are paying for it.

That's certainly true, although uncommon in my own experience because 
providers make operational choices  to avoid it. As much as possible, 
the ability to enable or disable a service is kept at the same (or 
higher) granularity than the MX names. For example, all Google Apps 
domains refer to a common set of MX names, but Gmail also enforces a 
uniform anti-virus policy across the service. Highly configurable 
services like Postini and Message Labs use different MX names for each 
administrative domain, ensuring sending MTAs send a separate copy to each.

Where I ran into problems (a "disposition conflict") was when different 
recipients in the same administrative domain (with the same MX name) 
were able to set different policies. That might be because of 
administrative delegations within a single customer, or because we gave 
individual users some knobs they could tweak. Many folk's intuition 
would be that anti-virus would not be a tweakable service; but you would 
be wrong. Today's enterprise class anti-malware filters are almost as 
hairy and indeterminate as spam filters, and that's because malware 
writers have gotten so darn clever.

<csg>

From: Ray Bellis <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Date: Fri, 13 Aug 2010 06:41:42 +0000
Accept-Language: en-GB, en-US
Sender: [email-masked]

> An AV that's configured with different rules for different recipients.

Or one that's only configured for the customers that are paying for it.

Ray

Date: Fri, 13 Aug 2010 01:08:38 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: [email-masked]
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

John Levine wrote:
>> "Spam" filters might include virus scanning, too, and the false
>> positive rates on virus scanning have gotten totally out of hand in
>> the past year or so.
> 
> I believe you, but what kind of AV is going to decide it's a virus for
> one recipient but not for the other?

Right.  I think this is different - virus vs decisions about 
subjective spam or attachment holds, etc.  An AVS detected virus, 
which implies the virus "DNA" was recorded in some AVS database, 
should make this a 'system-wide' rejection policy.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 12 Aug 2010 21:36:24 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: John Levine <[email-masked]>
CC: [email-masked]
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

John Levine wrote:
>> "Spam" filters might include virus scanning, too, and the false
>> positive rates on virus scanning have gotten totally out of hand in
>> the past year or so.
>>     
>
> I believe you, but what kind of AV is going to decide it's a virus for
> one recipient but not for the other?

An AV that's configured with different rules for different recipients. 
Unlikely in firewall/appliance or in a departmental MTA; painfully 
common in hosted / cloud computing.

For example, a hosted security suite might use multiple AV engines, some 
of which work in real time and others that require offline processing. 
The real time filter is more prone to false positives, but can reject at 
the SMTP layer. The offline one is more accurate, but  is noticeably 
slower, has lower privacy guaranties, and must deliver to quarantine 
(cannot reject, must not NDN).

So: the customer decides that for those users that require strict 
privacy or that are stupid enough to hurt themselves by delivering a 
live virus from quarantine, get the real-time filter. The more 
sophisticated users get the better filter, particularly if they've been 
incentivized to use it because it generates signatures that can improve 
the accuracy of the real-time filter.

If I could have coded the above, I would have. But I couldn't figure out 
a way to make it work. Compromise: put everything in quarantine, but 
only allow users with administrative privileges the ability to deliver 
from quarantine.

Actually, cloud computing is going to be weighing more and more on these 
issues.

IMHO, users stupid enough to download and run a self-extracting zip that 
was flagged in Large, Unfriendly Letters as malware deserve what they 
get. But I get paid to not think that way. ;-)

<csg>

Date: 13 Aug 2010 04:00:36 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: per user post-data rejects, Processing after the end of DATA
Organization: 
Cc: [email-masked]
Sender: [email-masked]

>"Spam" filters might include virus scanning, too, and the false
>positive rates on virus scanning have gotten totally out of hand in
>the past year or so.

I believe you, but what kind of AV is going to decide it's a virus for
one recipient but not for the other?

R's,
John

Date: Thu, 12 Aug 2010 20:36:08 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Rich Kulawiec <[email-masked]>
CC: [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

Rich Kulawiec wrote:
> On Thu, Aug 12, 2010 at 12:58:51PM -0400, Hector Santos wrote:
>> In our implementation, we use CBV (Callback Verification) and this
> 
> This should never be used; it enables spammers to bypass security
> measures, it facilitates DoS/DDoS attacks, and it's easily gamed.
> We've known this for most of a decade; figured it out when we watched
> Verizon deploy it and promptly get used to target third parties.
> See the archives of spam-l for copious discussion and analysis.

-1.

The valid return path is a SMTP requirement and it MUST be valid at 
the time it is issued by the sender.  Testing it is a VALID option to 
perform for the simple reason ERROR REPORTING is a required 
possibility.  It must not be invalid.

Implemented a CBV correctly, its works extremely well.  Ours has a 
unique OPEN RELAY test which tries a 2nd bad RCPT TO with a bad remote 
domain. If the host accepts it, its a CBV failure and the original 
transaction is rejected.

Just today our small support site had over 500 CBV rejections of bad 
return paths - that is 500 possible errant Bounces that did not have 
to be perform.

None of our customer installations have seen DoS/DDoS due to this 
which is an separate issue dealt with connection and session balancing 
and limits.  I don't see how it allows spammers to bypass security 
measures. It can however, make them more compliant - a valid return 
path is a SMTP requirement, not a MAYBE, not a COULD be valid later or 
tomorrow - but at the time it is provided.  Not doing so puts pressure 
on the system, which is something we try to alleviate by filtering 
SMTP non-compliant clients.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 12 Aug 2010 17:13:48 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: [email-masked]
CC: John R Levine <[email-masked]>, Ned Freed <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

[email-masked] wrote:
>> So if they've decided it's spam, what's the problem with throwing it 
>> away?
>
> The same problem as always: False positives. While I think the various
> admonitions we have in various documents that try and discourage silently
> dropping messages are unrealistic, the fact remains that silent draps 
> create
> reliability issues. Rejects address this issue and are preferable, as 
> long
> as they don't create additional issues.

"Spam" filters might include virus scanning, too, and the false positive 
rates on virus scanning have gotten totally out of hand in the past year 
or so. This is not a trivial matter; the kind of content that triggers 
false positives in the AV scanners tend to be the kind that users get 
upset over when it goes missing, e.g., compound documents, spreadsheets, 
and PDFs.

<csg>

From: [email-masked]
Cc: Ned Freed <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Thu, 12 Aug 2010 16:51:53 -0700 (PDT)
Subject: Re: per user post-data rejects, Processing after the end of DATA
To: John R Levine <[email-masked]>
Sender: [email-masked]

> > One of the typical ways this plays out is the system spam filter, for whatever
> > reason, misses some spam. (Hardly an uncommon occurance.) One user then takes
> > it upon themselves to add a rule to reject the spam they received. But other
> > users don't do this.

> So if they've decided it's spam, what's the problem with throwing it away?

The same problem as always: False positives. While I think the various
admonitions we have in various documents that try and discourage silently
dropping messages are unrealistic, the fact remains that silent draps create
reliability issues. Rejects address this issue and are preferable, as long
as they don't create additional issues.

				Ned

From: [email-masked]
Cc: [email-masked], [email-masked]
Date: Thu, 12 Aug 2010 16:43:53 -0700 (PDT)
Subject: Re: per user post-data rejects, Processing after the end of DATA
To: "Carl S. Gutekunst" <[email-masked]>
Sender: [email-masked]

> Dave CROCKER wrote:
> > Might as well ask an even more basic question:
> >
> > I had the impression that multi-recipient RCPT-TO usage had
> > essentially disappeared.  The growth of sending per-recipient tailored
> > messages seemed to be the cause.

> Wow. Not in my experience, definitely not for ordinary B-to-B, where the
> median is probably over 1.5, and not even for legit bulk.

This matches my experience as well.

> I had a particularly memorable sender a year ago that sent newsletters.
> They highly optimized their RCPT TO's, so not only would they send to
> multiple recipients in the same message, they'd combine recipients in
> different domains if the MX's allowed it. It was a headache for me
> because some of my customers were careless with their MX records, and
> mail would be combined for recipients that were on different mailhosts.

Indeed. A not uncommon support request we get is how to raise the recipient
limit from its default of 128. It's much too low for a nonnegligable number of
applications - in some cases by as much as two orders of magnitude. (Yes, the
settings are all documented, but you know how that goes...)

				Ned

Date: Thu, 12 Aug 2010 16:31:07 -0700
From: Douglas Otis <[email-masked]>
To: [email-masked]
CC: Dave CROCKER <[email-masked]>, [email-masked]
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

  On 8/12/10 3:55 PM, Dave CROCKER wrote:
>
>
> Might as well ask an even more basic question:
>
> I had the impression that multi-recipient RCPT-TO usage had 
> essentially disappeared.  The growth of sending per-recipient tailored 
> messages seemed to be the cause.
>
> If my impression is wrong, is there any useful (and accurate) 
> characterization we can make of different types of SMTP (relaying) 
> usage patterns and their popularity?
Dave,

The average number of recipients within a spam message has dropped.  
Perhaps they can afford to remain inconspicuous, since they are using 
plenty of compromised accounts.

-Doug

Subject: Re: per user post-data rejects, Processing after the end of DATA
From: Steve Atkins <[email-masked]>
Date: Thu, 12 Aug 2010 16:25:23 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 12, 2010, at 3:55 PM, Dave CROCKER wrote:

> 
> 
> Might as well ask an even more basic question:
> 
> I had the impression that multi-recipient RCPT-TO usage had essentially disappeared.

Decreased, probably, but nowhere near disappeared.

>  The growth of sending per-recipient tailored messages seemed to be the cause.

Only in higher-end mailing list managers. The main advantages of avoiding multiple recipients is the ability to use VERP for automated bounce handling and, to a slightly lesser degree, customized URLs for list management.

If I send mail from my MUA to two recipients at the same domain, or use any of a number of less powerful MLMs, they'll probably use multiple recipients for a single message.

> 
> If my impression is wrong, is there any useful (and accurate) characterization we can make of different types of SMTP (relaying) usage patterns and their popularity?
> 

Cheers,
  Steve

Date: Thu, 12 Aug 2010 16:21:15 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: [email-masked]
CC: [email-masked]
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

Dave CROCKER wrote:
> Might as well ask an even more basic question:
>
> I had the impression that multi-recipient RCPT-TO usage had 
> essentially disappeared.  The growth of sending per-recipient tailored 
> messages seemed to be the cause.

Wow. Not in my experience, definitely not for ordinary B-to-B, where the 
median is probably over 1.5, and not even for legit bulk.

I had a particularly memorable sender a year ago that sent newsletters. 
They highly optimized their RCPT TO's, so not only would they send to 
multiple recipients in the same message, they'd combine recipients in 
different domains if the MX's allowed it. It was a headache for me 
because some of my customers were careless with their MX records, and 
mail would be combined for recipients that were on different mailhosts.

<csg>

Date: 12 Aug 2010 19:10:16 -0400
From: "John R Levine" <[email-masked]>
To: "Ned Freed" <[email-masked]>
Cc: "IETF SMTP list" <[email-masked]>
Subject: Re: per user post-data rejects, Processing after the end of DATA
Cleverness: None detected
Sender: [email-masked]

> No, that's not even close to the typical use-case. The typical use-case is much
> simpler: Per-user Sieves or similar user-specified filters that support
> reject/ereject or its equivalent.

Oh, OK.

> One of the typical ways this plays out is the system spam filter, for whatever
> reason, misses some spam. (Hardly an uncommon occurance.) One user then takes
> it upon themselves to add a rule to reject the spam they received. But other
> users don't do this.

So if they've decided it's spam, what's the problem with throwing it away?

>> or if they're reasonably confident in their filters, just not deliver 
>> it to the second recipient.

> Whould would mean, in some cases, ignoring specific whiteisting directives that
> you're contracurally obliged to support. THat's a good way to get yourself
> sued. (I Happen to know this last has been a real, not theoretical, issue.)

I think you have my example backwards.  The second recipient is the one 
whose filter decided he didn't want the mail.  You deliver the mail to the 
person who said he wanted it.

I don't deny that there are some scenarios where per recipient rejections 
could be useful.  But I'm still having trouble coming up with scenarios 
where after looking at the body of the message, it's significantly wrong 
just to drop it for the receipients who don't want it.  How realistic is 
it to have a per-user content based rejection that's based on something 
other than concluding that a message is spam?

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

Date: Thu, 12 Aug 2010 15:55:33 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: Re: per user post-data rejects, Processing after the end of DATA
Sender: [email-masked]

Might as well ask an even more basic question:

I had the impression that multi-recipient RCPT-TO usage had essentially 
disappeared.  The growth of sending per-recipient tailored messages seemed to be 
the cause.

If my impression is wrong, is there any useful (and accurate) characterization 
we can make of different types of SMTP (relaying) usage patterns and their 
popularity?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: [email-masked]
Cc: [email-masked], "Murray S. Kucherawy" <[email-masked]>, SMTP Interest Group <[email-masked]>
Date: Thu, 12 Aug 2010 15:38:28 -0700 (PDT)
Subject: RE: Changing RFC 5322 guidance about crlf.crlf response delay
To: Tony Finch <[email-masked]>
Sender: [email-masked]

> On Wed, 11 Aug 2010, [email-masked] wrote:
> >
> > > > 	"Long delays after the <CRLF>.<CRLF> is received can
> > > > 	result in timeouts and duplicate messages.  Deferring
> > > > 	detailed message analysis until after the SMTP
> > > > 	connection has closed can result in non-delivery
> > > > 	notifications, possibly sent to incorrect addresses.  A
> > > > 	receiver-SMTP MUST carefully balance these two
> > > > 	considerations, i.e., the time required to respond to
> > > > 	the final <CRLF>.<CRLF> end of data indicator and the
> > > > 	desirable goal of rejecting undeliverable or
> > > > 	unacceptable messages at SMTP time."
> >
> > > I like this text.  I think it reflects current operational realities quite
> > > nicely.
> >
> > I agree. The only suggestion I have is that an informational referenece to
> > Craig Partridge's oringal document on the timeout issue might be helpful to
> > include after the first sentence.

> Yes. Maybe also add to the second sentence a cross-reference to section
> 7.1 on spoofing?

+1

				Ned

From: [email-masked]
Cc: [email-masked], [email-masked]
Date: Thu, 12 Aug 2010 15:22:52 -0700 (PDT)
Subject: Re: per user post-data rejects, Processing after the end of DATA
To: John Levine <[email-masked]>
Sender: [email-masked]

> In article <[email-masked]> you write:
> >
> >
> >> Hello,
> >
> >> if rejecting after CRLF.CRLF is better than bouncing, what about
> >> discussing draft-hall-prdr-00 and doing it more normative? ...

> >I never understood why the interest level in this was so low.

> Probably because the problem it solves is not one that many people
> have.  The scenario appears to be one where you have per-user content
> filters AND you filter during the SMTP session AND you reject mail
> that fails filtering AND the message is marginal enough that it passes
> one user's filter and fails another's.

No, that's not even close to the typical use-case. The typical use-case is much
simpler: Per-user Sieves or similar user-specified filters that support
reject/ereject or its equivalent. The problem occurs when one recipient rejects
and the another does not, or when both reject but with different reasons.
(Ereject deals with the latter by allowing a generic "everybody hates you,
kindly go eat some worms" response, but it would be nice to retain the original
reason strings.)

One of the typical ways this plays out is the system spam filter, for whatever
reason, misses some spam. (Hardly an uncommon occurance.) One user then takes
it upon themselves to add a rule to reject the spam they received. But other
users don't do this.

I can also assure you that the number of Sieve users out there is NOT small, so
this is a real issue. The usual way it's dealt with, unfortunately, is to
disallow use of reject/ereject, which solves the problem, at the cost of
further compromising email handling integrity since now users will just use
discard.

I suspect the real reason these proposals haven't gotten sufficient traction is
because people don't understand the problem they actually solve.

> It seems much more likely in that scenario that rather than rejecting,
> a system would drop it in the second recipient's spam folder,

What spam folder? You're making a lot of assumptions about what features
are available in many setups.

> or if
> they're reasonably confident in their filters, just not deliver it to
> the second recipient.

Whould would mean, in some cases, ignoring specific whiteisting directives that
you're contracurally obliged to support. THat's a good way to get yourself
sued. (I Happen to know this last has been a real, not theoretical, issue.)

				Ned

Date: Thu, 12 Aug 2010 17:20:45 -0400
From: Rich Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

On Thu, Aug 12, 2010 at 12:58:51PM -0400, Hector Santos wrote:
> In our implementation, we use CBV (Callback Verification) and this

This should never be used; it enables spammers to bypass security
measures, it facilitates DoS/DDoS attacks, and it's easily gamed.
We've known this for most of a decade; figured it out when we watched
Verizon deploy it and promptly get used to target third parties.
See the archives of spam-l for copious discussion and analysis.

Best practice is to make an accept/reject/defer decision during the
SMTP connection, thus (almost) entirely avoiding the need to send NDNs.
(The "almost" allows for the edge cases, which can be limited in number
and duration by engineering mail system architecture to ensure that
outward-facing systems are always in possession of the information
necessary to make decisions on behalf of internal systems.)

---Rsk

Date: Thu, 12 Aug 2010 14:38:32 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Paul Smith <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: NDNs considered harmful
Sender: [email-masked]

Paul Smith wrote:

> Well, if we were the ISP we could - but we're not... The ISP could be 
> anyone - we do the end-user mail server software.
> 
> (I'm not asking for a solution really - just saying that the 'NDN' issue 
> isn't that easy - if an upstream ISP blocks outgoing port 25 (blocking 
> sending using MX routing), and then blocks all outgoing mail because of 
> 'spam' (NDNs) then you have no choice but to not send NDNs).

I'm not sure I follow. ISP have more "power" when it comes to "Users", 
and if is blocking port 25, then this generally implies a home, 
non-business tier account. If its was a business tier, an ISP blocking 
your mail service is open to Tortious Interference (business 
interruption) claims.

> (NB - We try to encourage users to switch to incoming SMTP if possible 
> because then an SMTP reject happens, but if they have dynamic IP 
> addresses it won't work, or if they are are technically incompetent 
> (many are) then even just opening up a port in a firewall to allow SMTP 
> in is not possible.)

We still have many of the low-end, ex-BBSers sysops who have home 
setups or home business setups with home ISP accounts.

Many use the DynaDNS or whatever you call it stuff and I generally try 
today to avoid their support issues because its nothing but headaches 
trying to get them to be satisfied with trying to setup a real mail 
server operation.  Delays, throttling, tarpitting, proxies, etc, is 
always an issue with them.  Seems to work fine one moment and puff, 
its back to problems.  Not very reliable.

What we are doing now is basically giving them PORT 587 and ESMTP AUTH 
outbound MTA sender features and I am doing so reluctantly. Before we 
relied on prearranged allow IP relay, smart hosting PORT 25 outbound 
only.  The growth of these SOHO is high though. Many are 
moving/starting businesses are home and ISP business fees are higher.

I suspect this is going to help this smaller market, but highly loyal 
customer base, however, its going to be interesting to see how the 
their ISPs are going to react to this or if its going to improve the 
networking.  I suspect it will - probably as a short term.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 12 Aug 2010 14:08:17 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: IETF SMTP list <[email-masked]>
Subject: Re: NDNs considered harmful
Sender: [email-masked]

[email-masked] wrote:
>>   On 12/08/2010 15:32, John R Levine wrote:
>> > On my tiny mail system, most but not quite all of the bounces can be
>> > handled as rejections at SMTP time.  The ones I can't are generally
>> > deliveries to scripts where the script decides whether it can accept
>> > the mail.  When those say nope, can't deliver that, when is it worth
>> > generating a bounce?
> 
>> We have had several cases where our customers have been blacklisted by
>> their ISP for sending 'spam', when, in fact, they have just been sending
>> NDNs to all the spam they receive.
> 
> Operationally speaking, once you've accepted responsibility for a 
> message, you lose a lot of handling options, one of them being to 
> blindly send NDNs for stuff like invalid recipients without first 
> checking to see if the message is spam.
> 
> This is why it's important to validate recipient addresses at the
> administrative boundary, and if you cannot do that, you need to perform a
> spam/virus check before generating any sort of response.

+1.

We observed ~60% in overhead reduction simply by following the SMTP 
Section 3.3 suggestion for delay MAIL FROM verification:

    3.3 Mail Transactions

    .....

    Despite the apparent scope of this requirement, there are
    circumstances in which the acceptability of the reverse-path may
    not be determined until one or more forward-paths (in RCPT
    commands) can be examined.  In those cases, the server MAY
    reasonably accept the reverse-path (with a 250 reply) and then
    report problems after the forward-paths are received and
    examined.  Normally, failures produce 550 or 553 replies.

This became important when adding SPF or any other MAIL FROM: 
verification. The 60% rejection at RCPT TO corresponded to a 60% 
savings in MAIL FROM SPF lookup reduction.

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: [email-masked]
Cc: John R Levine <[email-masked]>, "Rosenwald, Jordan" <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Thu, 12 Aug 2010 09:54:14 -0700 (PDT)
Subject: Re: NDNs considered harmful
To: Paul Smith <[email-masked]>
Sender: [email-masked]

>   On 12/08/2010 15:32, John R Levine wrote:
> >
> >> True statement, but that means the senders of the other 5% are now left
> >> in the dark as to what happened to their mail.
> >> Is there a proposed solution to that?
> >
> > Talking out of my hat here, it is my impression that spam is much more
> > likely to be sent to bogus addresses than legit mail is, so however
> > the spam fraction of your mailstream, the spam fraction of your
> > bounces is likely to be even higher.
> >
> > On my tiny mail system, most but not quite all of the bounces can be
> > handled as rejections at SMTP time.  The ones I can't are generally
> > deliveries to scripts where the script decides whether it can accept
> > the mail.  When those say nope, can't deliver that, when is it worth
> > generating a bounce?

> We have had several cases where our customers have been blacklisted by
> their ISP for sending 'spam', when, in fact, they have just been sending
> NDNs to all the spam they receive.

Operationally speaking, once you've accepted responsibility for a message, you
lose a lot of handling options, one of them being to blindly send NDNs for
stuff like invalid recipients without first checking to see if the message is
spam.

This is why it's important to validate recipient addresses at the
administrative boundary, and if you cannot do that, you need to perform a
spam/virus check before generating any sort of response.

> This is really quite tricky to handle in these cases - usually we just
> turn off the NDN sending.

While that will solve the problem it is, IMO, a poor operational choice.

> (It's tricky in these cases because it's a 'non-standard' setup -

You know, there might actually be a reason why the standards fail to describe
certain setups...

> with a
> catch-all POP3 account at an ISP, then a mail collector collecting from
> there, and sorting messages out and rejecting messages to unrecognised
> users - what should happen there?

What should happen is that you shouldn't be doing things this way. I'm sorry,
but POP3 wasn't designed to be used for this. (And neither is IMAP.) The
semantics are all wrong - final delivery has occurred and success DSNs may have
been sent, the message envelope is gone and is very difficult to reconstruct in
a reliable fashion, etc. etc.

If there's really a need for a "pull" rather than a "push" delivery solution,
you should be using ODMR. ETRN is also an option in some cases, although it's
more of a "triggered push" than a "pull".

Now, unfortunately, while ODMR and ETRN at least get the delivery semantics
right, they both fail to meet the needs of the present-day Internet in an
important way: They do not provide a means of performing address validation at
the point of ingress. (This is also a problem for secondary MXes.) Various
prooposals have been made to address this problem (e.g., use DNS protocols to
serve up validation rules and lists of valid recipients), but AFAIK none have
gotten much traction. There are also some nonstandard tricks you can use (e.g.,
forward-validating milters) but these tend not to work well in "pull"
scenarios.

> You can't do an SMTP reject, because
> the message has already been 'accepted for final delivery' by the ISP,
> so you have to send an NDN or not - the user isn't going to read the
> messages, so an NDN is appropriate, but causes problems.)

Your basic problem can be addressed by being more selective in the NDNs you
send. There's no need to eliminate NDNs altogether. But some of your issues are
due to the protocol misuse you're engaged in, so some reconsideration of that
is also in order.

				Ned

Date: Thu, 12 Aug 2010 18:14:10 +0100
From: Paul Smith <[email-masked]>
To: "Derek J. Balling" <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: NDNs considered harmful
Sender: [email-masked]

  On 12/08/2010 17:54, Derek J. Balling wrote:
> On Aug 12, 2010, at 12:26 PM, Paul Smith wrote:
>> (I'm not asking for a solution really - just saying that the 'NDN' issue isn't that easy - if an upstream ISP blocks outgoing port 25 (blocking sending using MX routing), and then blocks all outgoing mail because of 'spam' (NDNs) then you have no choice but to not send NDNs).
> ... or change upstream ISPs to one with a clue. :-)
Well, yes - but if that was an acceptable answer, then we wouldn't 
really be having this discussion.... :)

<2 page diatribe against big, useless, ISPs (is that an tautology?) 
deleted because no one cares... >

Date: Thu, 12 Aug 2010 12:58:51 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Paul Smith <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

Paul Smith wrote:
> 
>  On 12/08/2010 14:28, Rosenwald, Jordan wrote:
>> True statement, but that means the senders of the other 5% are now left
>> in the dark as to what happened to their mail.
>> Is there a proposed solution to that?
> 
> Maybe we just recommend sending NDNs to people if their email is DKIMed 
> or if it came from a server matching SPF rules, or if the return path 
> uses BATV
>
> It's better than never sending them at all, and those provisions make it 
> reasonably certain that the sender's email address wasn't forged.
> 
> Also, it might encourage people to put in place the anti-forgery methods.

+1

In our implementation, we use CBV (Callback Verification) and this 
resolves at least 50%, 70% to even has high as 90% of the "bad" MAIL 
FROM: problem. Currently it is among the highest filter in our suite 
of SMTP filters.

Seven years of daily stats can be viewed at:

      http://www.winserver.com/public/spamstats.wct

showing filtering stats at each SMTP state.

Note: The %accept column went to zero since 2006 because "strings" 
were changed and the stats log scanner was never updated to detect 
these.  Adding greylisting also skewed the DATA counts.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Subject: Re: NDNs considered harmful
From: "Derek J. Balling" <[email-masked]>
Date: Thu, 12 Aug 2010 12:54:37 -0400
Cc: John R Levine <[email-masked]>, IETF SMTP list <[email-masked]>
To: Paul Smith <[email-masked]>
Sender: [email-masked]

On Aug 12, 2010, at 12:26 PM, Paul Smith wrote:
> (I'm not asking for a solution really - just saying that the 'NDN' issue isn't that easy - if an upstream ISP blocks outgoing port 25 (blocking sending using MX routing), and then blocks all outgoing mail because of 'spam' (NDNs) then you have no choice but to not send NDNs).

... or change upstream ISPs to one with a clue. :-)

From: [email-masked]
Cc: [email-masked], "Rosenwald, Jordan" <[email-masked]>, SMTP Interest Group <[email-masked]>
Date: Wed, 11 Aug 2010 14:51:14 -0700 (PDT)
Subject: Re: Processing after the end of DATA
To: Dave CROCKER <[email-masked]>
Sender: [email-masked]

> On 8/11/2010 8:30 AM, [email-masked] wrote:
> >> Completely agree for several reasons, including that the only senders
> >> that take advantage of "as fast as you can process" scenarios are
> >> spammers.  If I take a second or two to accept a message from a legit
> >> sender they neither know nor care.  But to a spammer, the faster we
> >> receive and delivery the more mail gets through before a content filter
> >> or blocklist can update.  We saw this happen when we migrated to our
> >> current system 3 years ago.  Spam was delivered at many hundreds of
> >> times faster than mail from AOL, Hotmail and Gmail combined. Responding
> >> as quickly as my system will allow with<CRLF>.<CRLF>  helps no one I
> >> want to help.
> >
> > You know, that's very nicely put. I'm going to use that last line the next time
> > this comes up.

> Isn't this effectively a form of greylisting?

No. It would only be greylisting if the intent was to cause a temporary failure
and a later retry. That's not the case. Rather, the intent is to avoid
accepting more mail than you can deliver by slowing down the acceptance process
just a little, but staying well within the vast majority of timeouts
in actual operational use.

> I've seen mixed results with greylisting, given that some legit senders have
> SMTP clients that are, well, problematic.  (Nearly lost a consulting gig because
> the initial query note got mishandled from greylisting.)

Since we're not talking about greylisting this isn't really relevant, but FWIW
one of the effects of widespread use of greylisting has been to force certain
sorts of senders to clean up their acts. This may be one of the most
significant virtues of greylisting, actually.

> In any event, to the extent it's good to add some delay, the question is how
> much.  The other question is why this is good for long-term -- since
> specifications last for a decade or two -- and that bad actors won't simply
> adapt as the behavior becomes institutionalized.

I see why you want to put a specific value in the specification, but this is
actually the reason it's a bad idea: These are the sorts of things that change.
Right now latencies are generally low and relay senders seem willing to wait
for a fair amount of time while submission senders aren't. But any or all of
this could shift. And a small shift in any direction could easily result in any
set of specific values becoming suboptimal.

What doesn't change - and cannot change without funamental changes to SMTP
itself - are the tradeoffs involved. That's why describing them is a good idea.

I'll also point out that this is just one example of a more general issue: This
specification and quite a few others have been written with the view that the
most, and in comes cases the only, important thing is for everyone to be as
good a network citixen as possible.

This may have been fine advice thirty years ago, but it isn't now. The Internet
can be a hostile place and protecting yourself is important. Nobody wins when
your mail server is so overrun with garbage to to point that legitimate email
is blocked.

Please note that I am not advocating the removal of the "good network citizen"
stuff. It's important to have that too, but for mostly historical reasons there
are too many places like this one where the advice is much too one sided.

				Ned

Date: Thu, 12 Aug 2010 17:26:58 +0100
From: Paul Smith <[email-masked]>
To: John R Levine <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: Re: NDNs considered harmful
Sender: [email-masked]

  On 12/08/2010 17:16, John R Levine wrote:
>> (It's tricky in these cases because it's a 'non-standard' setup - 
>> with a catch-all POP3 account at an ISP, then a mail collector 
>> collecting from there, and sorting messages out and rejecting 
>> messages to unrecognised users - what should happen there? You can't 
>> do an SMTP reject, because the message has already been 'accepted for 
>> final delivery' by the ISP, so you have to send an NDN or not - the 
>> user isn't going to read the messages, so an NDN is appropriate, but 
>> causes problems.)
>
> Could you do some sort of technical hack, a call-forward to whatever 
> is doing the sorting to find out what addresses are valid?

Well, if we were the ISP we could - but we're not... The ISP could be 
anyone - we do the end-user mail server software.

(I'm not asking for a solution really - just saying that the 'NDN' issue 
isn't that easy - if an upstream ISP blocks outgoing port 25 (blocking 
sending using MX routing), and then blocks all outgoing mail because of 
'spam' (NDNs) then you have no choice but to not send NDNs).

(NB - We try to encourage users to switch to incoming SMTP if possible 
because then an SMTP reject happens, but if they have dynamic IP 
addresses it won't work, or if they are are technically incompetent 
(many are) then even just opening up a port in a firewall to allow SMTP 
in is not possible.)

Date: 12 Aug 2010 12:16:40 -0400
From: "John R Levine" <[email-masked]>
To: "Paul Smith" <[email-masked]>
Cc: "IETF SMTP list" <[email-masked]>
Subject: Re: NDNs considered harmful
Cleverness: None detected
Sender: [email-masked]

> (It's tricky in these cases because it's a 'non-standard' setup - with a 
> catch-all POP3 account at an ISP, then a mail collector collecting from 
> there, and sorting messages out and rejecting messages to unrecognised users 
> - what should happen there? You can't do an SMTP reject, because the message 
> has already been 'accepted for final delivery' by the ISP, so you have to 
> send an NDN or not - the user isn't going to read the messages, so an NDN is 
> appropriate, but causes problems.)

Could you do some sort of technical hack, a call-forward to whatever is 
doing the sorting to find out what addresses are valid?

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

Date: Thu, 12 Aug 2010 17:07:43 +0100
From: Paul Smith <[email-masked]>
To: John R Levine <[email-masked]>
CC: "Rosenwald, Jordan" <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: NDNs considered harmful
Sender: [email-masked]

  On 12/08/2010 15:32, John R Levine wrote:
>
>> True statement, but that means the senders of the other 5% are now left
>> in the dark as to what happened to their mail.
>> Is there a proposed solution to that?
>
> Talking out of my hat here, it is my impression that spam is much more 
> likely to be sent to bogus addresses than legit mail is, so however 
> the spam fraction of your mailstream, the spam fraction of your 
> bounces is likely to be even higher.
>
> On my tiny mail system, most but not quite all of the bounces can be 
> handled as rejections at SMTP time.  The ones I can't are generally 
> deliveries to scripts where the script decides whether it can accept 
> the mail.  When those say nope, can't deliver that, when is it worth 
> generating a bounce?

We have had several cases where our customers have been blacklisted by 
their ISP for sending 'spam', when, in fact, they have just been sending 
NDNs to all the spam they receive.

This is really quite tricky to handle in these cases - usually we just 
turn off the NDN sending.

(It's tricky in these cases because it's a 'non-standard' setup - with a 
catch-all POP3 account at an ISP, then a mail collector collecting from 
there, and sorting messages out and rejecting messages to unrecognised 
users - what should happen there? You can't do an SMTP reject, because 
the message has already been 'accepted for final delivery' by the ISP, 
so you have to send an NDN or not - the user isn't going to read the 
messages, so an NDN is appropriate, but causes problems.)

From: [email-masked]
Cc: John R Levine <[email-masked]>, "Rosenwald, Jordan" <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Thu, 12 Aug 2010 08:46:21 -0700 (PDT)
Subject: Re: NDNs considered harmful
To: "Derek J. Balling" <[email-masked]>
Sender: [email-masked]

> On Aug 12, 2010, at 10:32 AM, John R Levine wrote:
> > Talking out of my hat here, it is my impression that spam is much more likely to be sent to bogus addresses than legit mail is, so however the spam fraction of your mailstream, the spam fraction of your bounces is likely to be even higher.

> Maybe. But I've seen some *really* effective anti-spam/anti-blowback
> mechanisms catch that junk and throw it on the floor at the "original sender's"
> side, while still allowing "legit" bounces to go through just fine (the
> Ironport A/S stuff works really, really, well, the last time I used it).

That's been my experience as well.

> At the end of the day, though, there's an obligation to tell the sender "this
> message with your address on it was undeliverable". It's up to them to filter
> as they see fit. If it's truly as you say, overwhelming, they can just as
> easily drop everything with FROM:<> into the bit-bucket themselves, and lose
> the "5% that matter baby" along with the bathwater.

+1

> But that's THEIR decision to make, and no one else's.

Yep. I'll also point out that since senders can track the messages they send,
they also have the option of correlating NDNs to those messages and throwing
out the ones that don't line up with anything.

				Ned

Date: Thu, 12 Aug 2010 11:37:15 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: IETF SMTP list <[email-masked]>
Subject: Re: NDNs considered harmful
Sender: [email-masked]

Derek J. Balling wrote:

> At the end of the day, though, there's an obligation to tell the 
> sender "this message with your address on it was undeliverable". 

+1

> It's up to them to filter as they see fit. If it's truly as you 
> say, overwhelming, they can just as easily drop everything with FROM:<> 
> into the bit-bucket themselves, and lose the "5% that matter baby" 
> along with the bathwater.

> But that's THEIR decision to make, and no one else's.

Strictly local policy, not to be used as a basis for protocol design 
or breaking protocol.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 12 Aug 2010 08:27:16 -0700
To: John C Klensin <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Draft to Full, versus cycling at Draft
Cc: SMTP Interest Group <[email-masked]>, [email-masked]
Sender: [email-masked]

Hi John,
At 00:35 12-08-10, John C Klensin wrote:
>Interesting.  I completely disagree.  Suggesting more attention
>to  a tradeoff and making that tradeoff more clear is a change
>in general guidance, not a change in conformance rules or other
>normative requirements (much less a "new requirement").  If
>changes in general guidance are not acceptable, then  the only
>changes that can be made between Draft and Full are editorial.

In the IETF it's also a matter of how you say it. :-)

I agree that suggesting more attention to a tradeoff and making a 
tradeoff more clear is not a change in conformance rules.

The academic exercise was about whether a change which is too 
technical or substantive means having to cycle to Draft.  One of the 
reasons for the levels is "to verify
that the text of the specification is adequately clear and 
accurate".  "A Draft Standard must be well-understood and known to be 
quite stable, both in its semantics and as a basis for developing an 
implementation".  The specification can be advanced to Full Standard 
once significant implementation and successful operational experience 
has been obtained.  The "standards of evidence are materially beyond 
what can be reasonably accomplished".  A pragmatic approach is to 
leave it to the advocates to make a convincing argument that the 
specification fulfills the requirements.  The "specific way to make 
the argument is left to the advocate".

Put simply, changes in general guidance are acceptable.

Regards,
-sm  

Date: Thu, 12 Aug 2010 11:14:05 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: IETF SMTP list <[email-masked]>
Subject: Re: NDNs considered harmful
Sender: [email-masked]

John R Levine wrote:
> 
>> True statement, but that means the senders of the other 5% are now left
>> in the dark as to what happened to their mail.
>> Is there a proposed solution to that?
> 
> Talking out of my hat here, it is my impression that spam is much more 
> likely to be sent to bogus addresses than legit mail is, so however the 
> spam fraction of your mailstream, the spam fraction of your bounces is 
> likely to be even higher.
> 
> On my tiny mail system, most but not quite all of the bounces can be 
> handled as rejections at SMTP time.  The ones I can't are generally 
> deliveries to scripts where the script decides whether it can accept the 
> mail.  When those say nope, can't deliver that, when is it worth 
> generating a bounce?

Yes - in the larger scheme of things - absolutely yes.

Much of that concern are removed when you move those Scripts to DATA 
level processing.  Tiny systems can more than well afford to do this 
type of DATA filtering. There is no difference between DATA processing 
or POST SMTP processing  - the results are both theoretically and 
empirically the same.  So what we are really talking about here is 
using software or techniques that provides that capability.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 12 Aug 2010 10:59:53 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "Rosenwald, Jordan" <[email-masked]>
CC: [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

Not doing what is not expected works. IMO, most of the problem can 
been observed with setups that accept mail first or are less 
aggressive at SMTP filtering - the Dynamic vs POST SMTP setup.  They 
are normally put into the position to thinking of and/or implementing 
discarding of mail as a general solution.  Even if 95% of the 
connections are bad people, rejecting 80%, 90% of that at SMTP is a 
major spam reduction.

The behavior is akin to blindly allowing thousands of people into your 
Disco, then going around the bar and disco floor throwing out the 
people you don't want.  Filtering them at the door seems to work for 
the Disco industry.

In my opinion, in the past, larger volume systems had to accept mail 
first for post smtp mail processing because it was the only way they 
could scale the volume.

Today, that has changed.  Even the AOL, YAHOO, and many of the older 
large email systems have moved more to SMTP level filtering.  Smaller 
volume systems long enjoyed SMTP level filtering because it was 
feasible for them to do.

Rosenwald, Jordan wrote:
> True statement, but that means the senders of the other 5% are now left
> in the dark as to what happened to their mail.
> Is there a proposed solution to that?
> 
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of John Levine
> Sent: Wednesday, August 11, 2010 5:51 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: NDNs considered harmful
> 
> 
>>> I note, again fwiw, that I've been trying to get various advocates
>>> for a ban (or near-ban) on NDNs to write that separate document and
>>> propose a specific model at regular intervals since well before
>>> 2821 was completed.
> 
>> I'm new to that particular topic.  Can you explain its motivation or
>> point me to a discussion thread that lays it out so I can get some
>> context?
> 
> Nothing surprising -- on today's Internet where 95% of mail is spam
> and essentially all the spam has forged return addresses, no matter
> how careful you are, most of your NDNs will be blowback to people who
> didn't send the mail.
> 
> R's,
> John
> 
> 
> 
> 
> 

-- 
Sincerely

Hector Santos
http://www.santronics.com

Subject: Re: NDNs considered harmful
From: "Derek J. Balling" <[email-masked]>
Date: Thu, 12 Aug 2010 10:58:11 -0400
Cc: "Rosenwald, Jordan" <[email-masked]>, "IETF SMTP list" <[email-masked]>
To: John R Levine <[email-masked]>
Sender: [email-masked]

On Aug 12, 2010, at 10:32 AM, John R Levine wrote:
> Talking out of my hat here, it is my impression that spam is much more likely to be sent to bogus addresses than legit mail is, so however the spam fraction of your mailstream, the spam fraction of your bounces is likely to be even higher.

Maybe. But I've seen some *really* effective anti-spam/anti-blowback mechanisms catch that junk and throw it on the floor at the "original sender's" side, while still allowing "legit" bounces to go through just fine (the Ironport A/S stuff works really, really, well, the last time I used it).

At the end of the day, though, there's an obligation to tell the sender "this message with your address on it was undeliverable". It's up to them to filter as they see fit. If it's truly as you say, overwhelming, they can just as easily drop everything with FROM:<> into the bit-bucket themselves, and lose the "5% that matter baby" along with the bathwater.

But that's THEIR decision to make, and no one else's.

Date: 12 Aug 2010 10:32:39 -0400
From: "John R Levine" <[email-masked]>
To: "Rosenwald, Jordan" <[email-masked]>
Cc: "IETF SMTP list" <[email-masked]>
Subject: RE: NDNs considered harmful
Cleverness: None detected
Sender: [email-masked]

> True statement, but that means the senders of the other 5% are now left
> in the dark as to what happened to their mail.
> Is there a proposed solution to that?

Talking out of my hat here, it is my impression that spam is much more 
likely to be sent to bogus addresses than legit mail is, so however the 
spam fraction of your mailstream, the spam fraction of your bounces is 
likely to be even higher.

On my tiny mail system, most but not quite all of the bounces can be 
handled as rejections at SMTP time.  The ones I can't are generally 
deliveries to scripts where the script decides whether it can accept the 
mail.  When those say nope, can't deliver that, when is it worth 
generating a bounce?

R's,
John

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of John Levine
> Sent: Wednesday, August 11, 2010 5:51 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: NDNs considered harmful
>
>
>>> I note, again fwiw, that I've been trying to get various advocates
>>> for a ban (or near-ban) on NDNs to write that separate document and
>>> propose a specific model at regular intervals since well before
>>> 2821 was completed.
>
>> I'm new to that particular topic.  Can you explain its motivation or
>> point me to a discussion thread that lays it out so I can get some
>> context?
>
> Nothing surprising -- on today's Internet where 95% of mail is spam
> and essentially all the spam has forged return addresses, no matter
> how careful you are, most of your NDNs will be blowback to people who
> didn't send the mail.

Date: Thu, 12 Aug 2010 15:03:57 +0100
From: Paul Smith <[email-masked]>
To: "Rosenwald, Jordan" <[email-masked]>
CC: John Levine <[email-masked]>, [email-masked], [email-masked]
Subject: Re: NDNs considered harmful
Sender: [email-masked]

  On 12/08/2010 14:28, Rosenwald, Jordan wrote:
> True statement, but that means the senders of the other 5% are now left
> in the dark as to what happened to their mail.
> Is there a proposed solution to that?

Maybe we just recommend sending NDNs to people if their email is DKIMed 
or if it came from a server matching SPF rules, or if the return path 
uses BATV

It's better than never sending them at all, and those provisions make it 
reasonably certain that the sender's email address wasn't forged.

Also, it might encourage people to put in place the anti-forgery methods.

>
>
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of John Levine
> Sent: Wednesday, August 11, 2010 5:51 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: NDNs considered harmful
>
>
>>> I note, again fwiw, that I've been trying to get various advocates
>>> for a ban (or near-ban) on NDNs to write that separate document and
>>> propose a specific model at regular intervals since well before
>>> 2821 was completed.
>> I'm new to that particular topic.  Can you explain its motivation or
>> point me to a discussion thread that lays it out so I can get some
>> context?
> Nothing surprising -- on today's Internet where 95% of mail is spam
> and essentially all the spam has forged return addresses, no matter
> how careful you are, most of your NDNs will be blowback to people who
> didn't send the mail.
>
> R's,
> John
>
>
>

Content-Class: urn:content-classes:message
Subject: RE: NDNs considered harmful
Date: Thu, 12 Aug 2010 09:28:29 -0400
From: "Rosenwald, Jordan" <[email-masked]>
To: John Levine <[email-masked]>, <[email-masked]>
CC: <[email-masked]>
Sender: [email-masked]

True statement, but that means the senders of the other 5% are now left
in the dark as to what happened to their mail.
Is there a proposed solution to that?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of John Levine
Sent: Wednesday, August 11, 2010 5:51 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: NDNs considered harmful

>> I note, again fwiw, that I've been trying to get various advocates
>> for a ban (or near-ban) on NDNs to write that separate document and
>> propose a specific model at regular intervals since well before
>> 2821 was completed.

>I'm new to that particular topic.  Can you explain its motivation or
>point me to a discussion thread that lays it out so I can get some
>context?

Nothing surprising -- on today's Internet where 95% of mail is spam
and essentially all the spam has forged return addresses, no matter
how careful you are, most of your NDNs will be blowback to people who
didn't send the mail.

R's,
John

Date: Thu, 12 Aug 2010 03:35:52 -0400
From: John C Klensin <[email-masked]>
To: SM <[email-masked]>, [email-masked]
cc: SMTP Interest Group <[email-masked]>
Subject: Re: Draft to Full, versus cycling at Draft
Sender: [email-masked]

--On Wednesday, August 11, 2010 12:58 -0700 SM <[email-masked]>
wrote:

> 
> Hi Dave,
> At 09:41 11-08-10, Dave CROCKER wrote:
>> My note, here, isn't about that.  It's a 'process' question,
>> meant  mostly for academic consideration:
>> 
>>    Is this the sort of change that is appropriate for going
>>    from  Draft to Full?
> 
> The short answer is no.
> 
>> I would have thought that it was too technical and
>> substantive and  that, at the least, the doc would have to
>> cycle at, perhaps, Draft.
> 
> It would have to be recycled to Proposed.  If there isn't any
> new requirements, one could argue that this is a narrow change
> to address a specific problem identified during deployment.

Interesting.  I completely disagree.  Suggesting more attention
to  a tradeoff and making that tradeoff more clear is a change
in general guidance, not a change in conformance rules or other
normative requirements (much less a "new requirement").  If
changes in general guidance are not acceptable, then  the only
changes that can be made between Draft and Full are editorial.

That is, of course, independent of whether consensus could be
reached about the change.

best,
   john

Date: 11 Aug 2010 23:18:59 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: mail proxies, Processing after the end of DATA
Organization: 
Cc: [email-masked]
Sender: [email-masked]

>Spam filtering appliances and services sometimes run as, effectively,
>proxies rather than store-and-forward MTAs. Postini works in this
>way, I think, as do some of the enterprise filtering appliances that
>people protect their Exchange servers with.

Yes.  Postini even has a patent on the technique of using callforwards
to provide realtime responses in a spam filtering system, although I
suspect that if they ever litigated it, they'd find it had been
invented multiple times long before they did.

Don't Barracudas also work that way?

R's,
John

Date: 11 Aug 2010 23:16:03 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Organization: 
Cc: [email-masked]
Sender: [email-masked]

>SMTP session.  I'm hearing a reasonably strong consensus that it is now 
>acceptable -- and possibly essential -- to do some serious processing before 
>responding.

Yes, definitely.  One of the reasons people hate my favorite MTA,
qmail, is that it implemented the advice to respond as fast as
possible, queued incoming mail immediately, and deferred all
rejections until later, even invalid mailboxes.  I, and most other
qmail admins, have replaced the standard SMTP daemon with one that
does a lot more, in my case including DCC lookups and spamassassin, as
well as most invalid user rejections, before responding to the client.

R's,
John

Date: Wed, 11 Aug 2010 19:08:32 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Hansen <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

Tony Hansen wrote:

> On 8/11/2010 11:14 AM, John C Klensin wrote:
>>
>> FWIW,
>>
>> (1) My recommendation would be to retain the timeout, but note
>> that server implementations should be aware that some clients
>> will ignore the spec for operational reasons and apply a much
>> smaller number.  My guess is that a discussion of the
>> appropriateness of doing that doesn't belong in an erratum/
>> corrigendum to 5321 but in a separate document (if at all).

> Most of the notes about clients severely ignoring the spec appear to be 
> from MUAs that should really be using Submit instead of SMTP.

hmmmm, MUAs are generally active updated software today, so I find it 
odd they would among the severely ignorant abusers

How would SUBMIT (port 587) be different than PUBLIC SMTP (port 25) in 
regard to timeouts?

I can only see where SUBMIT requiring AUTH (not an option) 
short-circuits some|all  filtering processing at the transport level.

This is important for SUBMIT at the EHLO level in regards the increase 
use of SOHO NATS.  SUBMIT requires EHLO to be domain or IP-literal 
correct. It also requires AUTH.   Due to increase use of NATS with LAN 
user machines, the SMTP client could potentially use the wrong EHLO 
domain or [IP-Literal].  Thunderbird had this problem.

The solution:

   1) Help the TBIRD People to add a option to set the EHLO domain. In the
      versions from 2.0+, you can do this in the EDITOR CONFIG 
settings. See

      mail.smtpserver.smtp#.hello_argument

      You would hard code the [IP] or public domain of the NAT server.

   2) Add logic to the MSA server to delay EHLO checking until the
      AUTH is determined for PORT 587 connects.

      Since AUTH is a SUBMIT requirement, I believe the EHLO correctness
      requirement can be relaxed.

I wrote to John and Pete about this when the issue was seen (~2+ years 
ago). I don't know if any text was added to mention the issue in 
4909bis (I have not checked).

This might explain why some USERS are incapable of using SUBMIT even 
if they wanted to over LAN and NAT and a MUA that is using the User's 
PC IP does not match the IP of the SMTP nat machine connection.   If 
the SUBMIT server has a EHLO IP-literal vs Connection IP check, it 
will fail.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 11 Aug 2010 23:44:47 +0100
From: Tony Finch <[email-masked]>
To: Steve Atkins <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On Wed, 11 Aug 2010, Steve Atkins wrote:
>
> Spam filtering appliances and services sometimes run as, effectively,
> proxies rather than store-and-forward MTAs. Postini works in this
> way, I think, as do some of the enterprise filtering appliances that
> people protect their Exchange servers with.

GNU SAUCE works this way and dates from 1997 so the approach isn't that
new :-)

A key difference is that they never have to fan-out deliveries and they
have operational control over the destination, so deployments should be
able to keep a fair grip on delays.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
FISHER: SOUTHWEST 3 OR 4, BACKING EAST 4 OR 5. SLIGHT. SHOWERS. GOOD.

Date: Wed, 11 Aug 2010 18:32:17 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Finch <[email-masked]>
CC: [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Tony Finch wrote:
> On Wed, 11 Aug 2010, Hector Santos wrote:
>> Tony Finch wrote:
>>> A "smart host" is a general purpose outgoing SMTP relay for use by stub or
>>> queueless SMTP clients.
>> +1, i.e, like a list server.
> 
> No, a list server is something else.

Sorry I cut out your text. I was referring to what clients can connect 
to, including a List Server SMTP client, if any.

Our list server transport method is designed so it can either be set 
to create a mail spool of emails at the location eyeballed by our SMTP 
router or it can be set to a direct Mail Host domain/ip, "Smart Host" 
or otherwise - a direct connection to an SMTP receiver.

One point that hasn't been mentioned, which may be important here, in 
our implementation, all authenticated or authorized clients MAY get a 
complete pass any SMTP level filtering/processing.

So the DATA Processing delays do not play a major way in Port 587 
submissions or POST 25  AUTH, IP, POPB4SMTP authentication transactions.

Going to a SMART HOST generally implies some sort of authentication or 
pre-arranged authorization - otherwise it just an open relay.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 11 Aug 2010 23:29:09 +0100
From: Tony Finch <[email-masked]>
To: [email-masked]
cc: "Murray S. Kucherawy" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: RE: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

On Wed, 11 Aug 2010, [email-masked] wrote:
>
> > > 	"Long delays after the <CRLF>.<CRLF> is received can
> > > 	result in timeouts and duplicate messages.  Deferring
> > > 	detailed message analysis until after the SMTP
> > > 	connection has closed can result in non-delivery
> > > 	notifications, possibly sent to incorrect addresses.  A
> > > 	receiver-SMTP MUST carefully balance these two
> > > 	considerations, i.e., the time required to respond to
> > > 	the final <CRLF>.<CRLF> end of data indicator and the
> > > 	desirable goal of rejecting undeliverable or
> > > 	unacceptable messages at SMTP time."
>
> > I like this text.  I think it reflects current operational realities quite
> > nicely.
>
> I agree. The only suggestion I have is that an informational referenece to
> Craig Partridge's oringal document on the timeout issue might be helpful to
> include after the first sentence.

Yes. Maybe also add to the second sentence a cross-reference to section
7.1 on spoofing?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: WESTERLY OR SOUTHWESTERLY 3 OR 4, INCREASING 5 IN WEST.
SLIGHT, OCCASIONALLY MODERATE LATER. SHOWERS. GOOD.

Subject: Re: Processing after the end of DATA
From: Steve Atkins <[email-masked]>
Date: Wed, 11 Aug 2010 15:25:26 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 11, 2010, at 3:01 PM, Tony Finch wrote:

> 
> On Wed, 11 Aug 2010, Arnt Gulbrandsen wrote:
>> 
>> Some SMTP servers (even some smarthosts) will act as an SMTP client before
>> replying to the dot. That was true twenty years ago (when random unix boxes
>> acted as a general spool and MUAs could run e.g. sendmail -bs or smail
>> -iforget to talk to an SMTP server), and it's true now.
> 
> I think that's only true if you are running sendmail -odi (deliver
> interactively option). Smail does local delivery in the foreground and
> remote delivery in the background (dunno if that includes DNS lookups).
> Exim can't be told to do foreground deliveries while talking to an SMTP
> client, and qmail and postfix don't have the right architecture to allow
> it.
> 
> This sort of supports my earlier comment about the design of mailers
> changing in this respect about 15 years ago.

Spam filtering appliances and services sometimes run as, effectively,
proxies rather than store-and-forward MTAs. Postini works in this
way, I think, as do some of the enterprise filtering appliances that
people protect their Exchange servers with.

If anything, they're a more recent approach to mail filtering than
their competitors that are cobbled together using the store-and-forward
MTAs you mention. They have significant deployment advantages
and some disadvantages.

Cheers,
  Steve

Date: Wed, 11 Aug 2010 23:17:53 +0100
From: Tony Finch <[email-masked]>
To: Tony Hansen <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

On Wed, 11 Aug 2010, Tony Hansen wrote:
>
> Most of the notes about clients severely ignoring the spec appear to be from
> MUAs that should really be using Submit instead of SMTP.

We once had problems with a bank that had adjusted their sender-SMTP
timeouts to something like 10 or 30 seconds. (They were running Postfix
but Postfix was not to blame.)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
THAMES DOVER WIGHT PORTLAND PLYMOUTH: WEST OR SOUTHWEST, VEERING NORTHWEST, 3
OR 4, INCREASING 5 AT TIMES. SLIGHT, OCCASIONALLY MODERATE IN PORTLAND AND
PLYMOUTH. SHOWERS. GOOD.

Date: Wed, 11 Aug 2010 23:01:51 +0100
From: Tony Finch <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: John C Klensin <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On Wed, 11 Aug 2010, Arnt Gulbrandsen wrote:
>
> Some SMTP servers (even some smarthosts) will act as an SMTP client before
> replying to the dot. That was true twenty years ago (when random unix boxes
> acted as a general spool and MUAs could run e.g. sendmail -bs or smail
> -iforget to talk to an SMTP server), and it's true now.

I think that's only true if you are running sendmail -odi (deliver
interactively option). Smail does local delivery in the foreground and
remote delivery in the background (dunno if that includes DNS lookups).
Exim can't be told to do foreground deliveries while talking to an SMTP
client, and qmail and postfix don't have the right architecture to allow
it.

This sort of supports my earlier comment about the design of mailers
changing in this respect about 15 years ago.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST FAIR ISLE: VARIABLE, BECOMING NORTHEASTERLY, 3 OR 4. SLIGHT OR
MODERATE. RAIN OR SHOWERS. MODERATE OR GOOD.

Date: 11 Aug 2010 21:50:38 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: NDNs considered harmful
Organization: 
Cc: [email-masked]
Sender: [email-masked]

>> I note, again fwiw, that I've been trying to get various advocates
>> for a ban (or near-ban) on NDNs to write that separate document and
>> propose a specific model at regular intervals since well before
>> 2821 was completed.

>I'm new to that particular topic.  Can you explain its motivation or
>point me to a discussion thread that lays it out so I can get some
>context?

Nothing surprising -- on today's Internet where 95% of mail is spam
and essentially all the spam has forged return addresses, no matter
how careful you are, most of your NDNs will be blowback to people who
didn't send the mail.

R's,
John

Date: 11 Aug 2010 21:45:40 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: per user post-data rejects, Processing after the end of DATA
Organization: 
Cc: [email-masked]
Sender: [email-masked]

In article <[email-masked]> you write:
>
>
>> Hello,
>
>> if rejecting after CRLF.CRLF is better than bouncing, what about
>> discussing draft-hall-prdr-00 and doing it more normative? ...

>I never understood why the interest level in this was so low.

Probably because the problem it solves is not one that many people
have.  The scenario appears to be one where you have per-user content
filters AND you filter during the SMTP session AND you reject mail
that fails filtering AND the message is marginal enough that it passes
one user's filter and fails another's.

It seems much more likely in that scenario that rather than rejecting,
a system would drop it in the second recipient's spam folder, or if
they're reasonably confident in their filters, just not deliver it to
the second recipient.

R's,
John

Date: Wed, 11 Aug 2010 17:43:12 -0400
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
From: Jeff Macdonald <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Wed, Aug 11, 2010 at 5:24 PM, Steve Atkins <[email-masked]> wrote:
>
>
> On Aug 11, 2010, at 2:06 PM, Jeff Macdonald wrote:
>
>> On Wed, Aug 11, 2010 at 2:40 PM, Steve Atkins <[email-masked]> wrote:
>>> I don't believe that legitimate senders of email are likely to commonly see really long delays at the end of data, and I don't much care if other senders are inconvenienced.
>>
>> As a sender, over the years, I've seen duplicate messages to Y!
>> increase. The MTA we use has RFC suggested limits. We increased those
>> limits to reduce duplicates. We didn't yell at Y! that they should
>> follow the RFC better. It was easy to visualize that with the massive
>> amount of mailboxes they have that things would take longer than they
>> should from time to time.
>>
>> So I find it strange that some folks are seeking to suggest lower
>> values instead of realizing that those MTA clients are making a
>> conscience decision to not follow RFC guidelines at the risk of
>> increased duplicates. I'd say reducing those numbers would be an
>> endorsement of their bad behaviour. I don't think that is something we
>> should be doing.
>>
>> If clarification is sought, I like John's language. I'd also say keep
>> the 10 minute time-out. Otherwise I think the existing language is
>> fine.
>
> I'm not sure what you're responding to here. It doesn't appear to
> be anything implied in the mail of mine you're responding to -
> neither the quoted sentence nor the rest of it.
>
> I'm certainly not suggesting lower timeout values.

Oh, sorry about that. I wasn't implying you suggested that. I'm
basically in agreement with what you said, except that I have seen
longer delays. However I don't view those delays as an inconvenience,
just a fact of life in the fight against spam.

-- 
Jeff Macdonald
Ayer, MA

Date: Wed, 11 Aug 2010 22:40:08 +0100
From: Tony Finch <[email-masked]>
To: Hector Santos <[email-masked]>
cc: [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On Wed, 11 Aug 2010, Hector Santos wrote:
> Tony Finch wrote:
> >
> > A "smart host" is a general purpose outgoing SMTP relay for use by stub or
> > queueless SMTP clients.
>
> +1, i.e, like a list server.

No, a list server is something else.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
FITZROY: NORTHERLY 4 OR 5, INCREASING 6 OR 7 LATER IN SOUTH. SLIGHT OR
MODERATE. RAIN OR SHOWERS. MODERATE OR GOOD, OCCASIONALLY POOR IN SOUTH.

Date: Wed, 11 Aug 2010 17:24:55 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Paul Smith <[email-masked]>
CC: Dave CROCKER <[email-masked]>, Tony Finch <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Hi Paul,

Good idea, but I am not sure a 2nd SMTP "status" call is really 
necessary here.

Overall, Extensions means Change (client and server), and Change means 
software can be adapted in any nice way for a better fit.  Of course, 
it excludes supporting legacy software.

Probably the most backward compatible way is to use a temporary 45x 
rejection if the server sees that a lengthy delay is possible, 
complete the SMTP session (get the QUIT etc), but continue the 
processing in the background and record the results (250, 55x).  If 
and when the retransmit takes place, the data processing is 
preempted/skipped and the real final reply code can be immediately 
provided.

Very similar to a Grey List concept.

Maybe using a 1 minute data processing limit before a deferred status 
is triggered  with a 451 response.

Your idea is good in that it eliminates the retransmission, but the 
software coding is more complex and its not backward compatible with 
legacy clients.

Maybe a combination of the your DEFERREDSTATUS idea with recommended 
processing suggestion for legacy clients would work nicely. :)  Modern 
clients don't have to do a retransmission, but legacy clients will 
work the same with retransmissions.

Interested in working together on an SMTP extension I-D for this?

-- 
Sincerely

Hector Santos
http://www.santronics.com

Paul Smith wrote:
> 
>  On 11/08/2010 15:30, Hector Santos wrote:
>> +1. If we are looking for insights for "adjusted" recommendations, 
>> this would be a good time as there is no doubt the direction and 
>> growth of DATA level processing is permanent now.  The hardware allows 
>> for these new framework feasible in a scaled up manner and there is no 
>> way we can tell people "don't do that" and go back to old school 
>> methods of having post-SMTP (post acceptance) only data processing.
>>
>> Probably a BCP or a new I-D for SMTP DATA Processing Filtering 
>> Guidelines would be very welcome and useful.
>>
>> I would not mind drafting it if people think it is a good idea.
>>
> Daft question, but is there room for an SMTP extension here, given that 
> some senders really don't like waiting 10 minutes (or even 1 minute in 
> some cases) for a response for some reason:
> 
> - EHLO keyword 'DEFERREDSTATUS <x seconds>
> 
> - SENDER commands 'DEFERSTATUS' and 'GETDEFERRED'
> 
> Transaction
> 
> S: 220 ESMTP server ready
> C: EHLO mydomain.com
> S: 220-DEFERREDSTATUS 600
> S: 220 HELP
> C: DEFERSTATUS
> S: 250 OK
> C: MAIL FROM, RCPT TO, DATA etc
> C: .
> S: 250 2.5.6 Message accepted for now - id: bibblebobble237
> (or could give other 2/4/5xx response immediately)
> C: QUIT
> 
> <at least 10 minutes (600 seconds) later>
> 
> S: 220 ESMTP server ready
> C: EHLO mydomain.com
> S: 220-DEFERREDSTATUS 600
> S: 220 HELP
> C: GETDEFERRED bibblebobble237
> S: 550 Don't like that message
> C: QUIT
> 
> So, the sender could choose to have a quick turnaround and get responses 
> later, or wait and get them now. When the client choose deferred status 
> notification the server can process later, but has no responsibility to 
> generate a DSN.
> 
> Servers could still give immediate failures if they want, but if they 
> give a "deferred" code (I chose 2.5.6 out of thin air), then the sender 
> comes back later to find out what happened to the message.
> 
> Obviously the server could still lie about the response (eg give a 2xx 
> response when it really threw away spam) or take a few seconds before 
> deciding whether to give an immediate or deferred response, as they do 
> now, it could just cut down on duplicate processing.
> 
> 
> 
> 
> 

Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
From: Steve Atkins <[email-masked]>
Date: Wed, 11 Aug 2010 14:24:12 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 11, 2010, at 2:06 PM, Jeff Macdonald wrote:

> On Wed, Aug 11, 2010 at 2:40 PM, Steve Atkins <[email-masked]> wrote:
>> I don't believe that legitimate senders of email are likely to commonly see really long delays at the end of data, and I don't much care if other senders are inconvenienced.
> 
> As a sender, over the years, I've seen duplicate messages to Y!
> increase. The MTA we use has RFC suggested limits. We increased those
> limits to reduce duplicates. We didn't yell at Y! that they should
> follow the RFC better. It was easy to visualize that with the massive
> amount of mailboxes they have that things would take longer than they
> should from time to time.
> 
> So I find it strange that some folks are seeking to suggest lower
> values instead of realizing that those MTA clients are making a
> conscience decision to not follow RFC guidelines at the risk of
> increased duplicates. I'd say reducing those numbers would be an
> endorsement of their bad behaviour. I don't think that is something we
> should be doing.
> 
> If clarification is sought, I like John's language. I'd also say keep
> the 10 minute time-out. Otherwise I think the existing language is
> fine.

I'm not sure what you're responding to here. It doesn't appear to
be anything implied in the mail of mine you're responding to -
neither the quoted sentence nor the rest of it.

I'm certainly not suggesting lower timeout values.

Cheers,
  Steve

Date: Wed, 11 Aug 2010 17:11:23 -0400
Subject: Re: Draft to Full, versus cycling at Draft
From: Jeff Macdonald <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Wed, Aug 11, 2010 at 3:35 PM, Hector Santos <[email-masked]> wrote:
>
> Tony,
>
> The only thing I see is that it could|should have a CLIENT perspective as
> well:
>
> � �Client writers MUST pay close attention to the timeout specified in
> � �section 6.1 of RFC 5321.
>
> Because the problem is mostly with the clients not following the 10 minute
> recommendation. �I don't know if it should be mentioned but...
>
> � �.... Clients using anything less than the recommended 10 minutes
> � �could promote retransmission issues.
>
> That is not in any way promoting that servers can take its sweet time with
> 10 minute delays.
>

+1

-- 
Jeff Macdonald
Ayer, MA

Date: Wed, 11 Aug 2010 17:06:54 -0400
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
From: Jeff Macdonald <[email-masked]>
To: Steve Atkins <[email-masked]>
Cc: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Wed, Aug 11, 2010 at 2:40 PM, Steve Atkins <[email-masked]> wrote:
> I don't believe that legitimate senders of email are likely to commonly see really long delays at the end of data, and I don't much care if other senders are inconvenienced.

As a sender, over the years, I've seen duplicate messages to Y!
increase. The MTA we use has RFC suggested limits. We increased those
limits to reduce duplicates. We didn't yell at Y! that they should
follow the RFC better. It was easy to visualize that with the massive
amount of mailboxes they have that things would take longer than they
should from time to time.

So I find it strange that some folks are seeking to suggest lower
values instead of realizing that those MTA clients are making a
conscience decision to not follow RFC guidelines at the risk of
increased duplicates. I'd say reducing those numbers would be an
endorsement of their bad behaviour. I don't think that is something we
should be doing.

If clarification is sought, I like John's language. I'd also say keep
the 10 minute time-out. Otherwise I think the existing language is
fine.

-- 
Jeff Macdonald
Ayer, MA

From: [email-masked]
Cc: SMTP Interest Group <[email-masked]>
Date: Wed, 11 Aug 2010 10:53:38 -0700 (PDT)
Subject: Re: Processing after the end of DATA
To: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
Sender: [email-masked]

> Hello,

> if rejecting after CRLF.CRLF is better than bouncing, what about
> discussing draft-hall-prdr-00 and doing it more normative?

There have been several drafts over the years describing similar proposals. I
think this is the best of the bunch, mostly because it goes just far enough and
doesn't get into other things like trying to return entire DSNs in the SMTP
response stream.

I never understood why the interest level in this was so low. Anyone who has an
LMTP client can implement this pretty easily - the semantics are very similar.
THe server can be more difficult, but even so it's not that bad.

That said, this is unlikely to reduce blowback spam unless you required that
everyone implement it in order to send email, which of course we cannot do.
Consider: If the sender is legitimate, then a partial-success/partial-failure
scenario is a nonissue because the DSN is also legitimate.

If, OTOH, the sender isn't legitimate, then there's no incentive for them to
implement this - spammers can only benefit from blowback, and therefore have
no incentive to go out of their way to stop it.

> For an email with two recipients, the first one of who wants to receive
> the mail and the second -- not, there is currently no option but to say
> 250 OK after CRLF.CRLF and send later DSN.  With the idea described in
> draft-hall-prdr-00, it is possible to avoid the currently unavoidable
> DSN.  Any suggestion, how this draft can be revived?

Simple: Post an update and see if there's any interest in moving it to
proposed. I would cetainly support it.

> On 10.08.2010 20:18, David MacQuigg wrote:
> > My system is too small to draw any conclusions, but my impression is
> > that we are overly-concerned about a very small fraction of the mailflow
> > - those few messages that are legit, but get a false reject from a
> > statistical filter. We really don't care about the proper SMTP response
> > if it is spam.

But we do care in the case of false positive.

> > The only advantage I see in running the spam filter
> > before ACCEPT is a few less messages at the bottom of my spam bucket. I
> > wish I had some numbers from a larger system.

> A user might have installed redirect for her incoming mails to another
> mail provider.  The other provider might SMTP reject incoming spam.  If
> there is spam for your user, your server does accept it, then sends it
> to the other mail provider, and that second server SMTP rejects it, then
> your server will have to send DSN for that SPAM (provided that you do
> not want to discard emails).  With rejection-preferences similar on both
> servers, the amount of such DSNs will decrease, when both servers reject
> after DATA.  You have no influence on what other servers do with the
> spam (smtp rejecting or inserting in spam folders).

> > Would be nice if SMTP had a 'conditional ACCEPT': "Sorry, your
> > transmitter ID 'yahoo.com' cannot be verified, Your message has been
> > routed to the recipient's quarantine." Imagine the effect that would
> > have on all the lame excuses for an invalid or hard-to-verify HELO name.

> What do you want to communicate to the sender with this conditional
> ACCEPT: "Dear Sender, your suspicious email has arrived.  It might be
> read soon, it might be read late, or it might not be read at all."?

> The problem with the spam folders are the false positives.  Many users
> do check the spam folders/quarantines not that often as their Inbox and
> with less attention.  This might lead to reading a message too late, or
> even overseeing it.  I prefer to immediately notify the sender that her
> message was not delivered, including in the SMTP response alternative
> ways to contact the recipient (and spamassassin-evaluation of the mail),
> rather than let the sender hope that her email was properly delivered,
> while there is a risk that the recipient oversees that message.

Exactly - while PRDR will have little if any effect on existing blowback
issues, it adds to our handling options in the false positive case.

				Ned

Date: Wed, 11 Aug 2010 12:58:07 -0700
To: [email-masked]
From: SM <[email-masked]>
Subject: Re: Draft to Full, versus cycling at Draft
Cc: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

Hi Dave,
At 09:41 11-08-10, Dave CROCKER wrote:
>My note, here, isn't about that.  It's a 'process' question, meant 
>mostly for academic consideration:
>
>    Is this the sort of change that is appropriate for going from 
> Draft to Full?

The short answer is no.

>I would have thought that it was too technical and substantive and 
>that, at the least, the doc would have to cycle at, perhaps, Draft.

It would have to be recycled to Proposed.  If there isn't any new 
requirements, one could argue that this is a narrow change to address 
a specific problem identified during deployment.

To keep it simple, let's say that you could get away with it if there 
isn't any substantive objection.  That's a bar higher than consensus.

Regards,
-sm 

Date: Wed, 11 Aug 2010 15:35:47 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Hansen <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Draft to Full, versus cycling at Draft
Sender: [email-masked]

Tony,

The only thing I see is that it could|should have a CLIENT perspective 
as well:

     Client writers MUST pay close attention to the timeout specified in
     section 6.1 of RFC 5321.

Because the problem is mostly with the clients not following the 10 
minute recommendation.  I don't know if it should be mentioned but...

     .... Clients using anything less than the recommended 10 minutes
     could promote retransmission issues.

That is not in any way promoting that servers can take its sweet time 
with 10 minute delays.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Tony Hansen wrote:
> 
> Hmmm, if the following were added to 4409bis somewhere, I don't think it 
> should affect going from Draft => Full:
> 
>     X.Y Timeouts
> 
>     It has been observed that MUAs have in existing practice not been
>     allowing a long time for the Submit Server to respond to
>     the final <CRLF>.<CRLF>.
> 
>     Submit Server writers MUST pay close attention to the Server
>     timeout specified in section 6.1 of RFC 5321, where it says:
> 
>         ... a receiver-SMTP MUST seek to minimize the time
>         required to respond to the final <CRLF>.<CRLF> end of
>         data indicator.
> 
>     MUAs and other Submit clients SHOULD wait at least the normal TCP/IP
>     timeout lengths of time after sending the final <CRLF>.<CRLF>, if not
>     the full timeout specified in section 4.5.3.2.6 of RFC 5321.
> 
> or something along those lines.
> 
>     Tony Hansen
> 
> On 8/11/2010 12:41 PM, Dave CROCKER wrote:
>>
>>
>> On 8/11/2010 9:19 AM, Tony Hansen wrote:
>>> I'm wondering if, instead of changing the 5321 guidance, there needs 
>>> to be text
>>> added to 4409bis about timeouts being shorter than those normally 
>>> used by SMTP.
>>
>> Any useful guidance should be made wherever it is helpful.  This issue 
>> is certainly in the realm of 'useful'.
>>
>> My note, here, isn't about that.  It's a 'process' question, meant 
>> mostly for academic consideration:
>>
>>    Is this the sort of change that is appropriate for going from Draft 
>> to Full?
>>
>> I would have thought that it was too technical and substantive and 
>> that, at the least, the doc would have to cycle at, perhaps, Draft.
>>
>> To repeat:  this is meant purely as an academic exercise.  I'm curious.
>>
>> d/
> 
> 
> 

Date: Wed, 11 Aug 2010 12:36:20 -0700
From: David MacQuigg <[email-masked]>
To: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Dilyan,  We are getting way off topic for this list, so I invite you and 
anyone else who is interested, to a continuation of this discussion at 
http://groups.google.com/group/registry_internet.

Дилян Палаузов wrote:
 >
 > On 10.08.2010 20:18, David MacQuigg wrote:
 >> My system is too small to draw any conclusions, but my impression is
 >> that we are overly-concerned about a very small fraction of the mailflow
 >> - those few messages that are legit, but get a false reject from a
 >> statistical filter. We really don't care about the proper SMTP response
 >> if it is spam. The only advantage I see in running the spam filter
 >> before ACCEPT is a few less messages at the bottom of my spam bucket. I
 >> wish I had some numbers from a larger system.
 >
 > A user might have installed redirect for her incoming mails to 
another mail provider.  The other provider might SMTP reject incoming 
spam.  If there is spam for your user, your server does accept it, then 
sends it to the other mail provider, and that second server SMTP rejects 
it, then your server will have to send DSN for that SPAM (provided that 
you do not want to discard emails).  With rejection-preferences similar 
on both servers, the amount of such DSNs will decrease, when both 
servers reject after DATA.  You have no influence on what other servers 
do with the spam (smtp rejecting or inserting in spam folders).

We have quite a bit of influence.  We insist that our recipients turn 
off all spam filtering, chain-forwarding, SPF checks, whatever might 
conflict with our acting as Receiver for their box67.com email address.  
Typically, they will have a private mailbox known only to our 
Receiver/Forwarder.  Failures resulting from SMTP rejects, DSNs, or any 
other problem at the recipient's mailbox are treated the same as a 
mailbox being suddenly offline.  Incoming mail to that address is 
rejected until we can resolve the problem with the recipient.

 >> Would be nice if SMTP had a 'conditional ACCEPT': "Sorry, your
 >> transmitter ID 'yahoo.com' cannot be verified, Your message has been
 >> routed to the recipient's quarantine." Imagine the effect that would
 >> have on all the lame excuses for an invalid or hard-to-verify HELO name.
 >
 > What do you want to communicate to the sender with this conditional 
ACCEPT: "Dear Sender, your suspicious email has arrived.  It might be 
read soon, it might be read late, or it might not be read at all."?

No, the message I suggested had a specific reason for the failure.  It 
should also have a link to a webpage with instructions on how to fix the 
failure, like what we have now on REJECT messages:  
http://open-mail.org/3strikes/REJECT.html

 > The problem with the spam folders are the false positives.  Many 
users do check the spam folders/quarantines not that often as their 
Inbox and with less attention.  This might lead to reading a message too 
late, or even overseeing it.  I prefer to immediately notify the sender 
that her message was not delivered, including in the SMTP response 
alternative ways to contact the recipient (and spamassassin-evaluation 
of the mail), rather than let the sender hope that her email was 
properly delivered, while there is a risk that the recipient oversees 
that message.

False rejects are indeed the main problem with modern email systems, but 
this problem isn't solved by rejecting a small fraction of messages with 
the highest spam scores.  There are still plenty of messages, almost all 
spam, that get a score low enough to be kept in quarantine, and there is 
always a risk that the recipient will overlook a legitimate message.  
The only good solution to the false reject problem is to offer senders a 
way to avoid the statistical filters entirely.  That is what a 
reputation system does.  When a message comes from yahoo.com (or any 
other A-rated domain), it goes straight to our recipient's inbox.

************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

Date: Wed, 11 Aug 2010 15:28:26 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

Steve Atkins wrote:
> 
>> For example, a delay time of up to 2 seconds is probably pretty safe.
> 
> Many of the things you're going to want to do in that time will take
> longer than that. There's a long tail, but I'd expect to see some
> processing take a minute or more in rare cases.

There is definitely a long tail here. The cases will depend on how 
these scripts are designed.  If a script developer understands it 
could be applied at DATA, he/she will improve it for speed.   That is 
what happen to us. An example script provided that illustrated word 
list scanner reading off a text file, grew with popularity and their 
word list file grew over time increasingly causing delays into the 
minutes when large payloads were transferred.  Greater client timeouts 
drops and retransmits issues was a result.  Improved updates of the 
scanner moved the processing time into seconds by using caching, 
memory map files, pre-compiling of words, etc.

No one wants long delays to causes problems.

> What's the failure mode at that point in the transaction - are we
> defending against a transport failure, or the client violating spec
> with timeouts or...?

Two things IMV

   - most clients do not wait 10 mins. Its much shorter. We see around
     3-5 minutes.

   - The realization DATA processing is the new and more desirable 
"thing"
     and we need standards "adjustments" in order to provide continued
     consideration for the "old things" (legacy software).

Implementators simply need to be reminded that data processing time 
should be as minimal as possible, but also not trust a SMTP 
recommendation of 10 minutes to be universally followed when they 
begin to implement more DATA processing work.

For us, to help enforce/control the issue, we added a sysop-definable 
GlobalTimeOut with a default of 5 minutes for processing all 
call-outs, hooks, shims, what have you.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 11 Aug 2010 14:49:00 -0400
From: Tony Hansen <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Draft to Full, versus cycling at Draft
Sender: [email-masked]

Hmmm, if the following were added to 4409bis somewhere, I don't think it 
should affect going from Draft => Full:

     X.Y Timeouts

     It has been observed that MUAs have in existing practice not been
     allowing a long time for the Submit Server to respond to
     the final <CRLF>.<CRLF>.

     Submit Server writers MUST pay close attention to the Server
     timeout specified in section 6.1 of RFC 5321, where it says:

         ... a receiver-SMTP MUST seek to minimize the time
         required to respond to the final <CRLF>.<CRLF> end of
         data indicator.

     MUAs and other Submit clients SHOULD wait at least the normal TCP/IP
     timeout lengths of time after sending the final <CRLF>.<CRLF>, if not
     the full timeout specified in section 4.5.3.2.6 of RFC 5321.

or something along those lines.

     Tony Hansen

On 8/11/2010 12:41 PM, Dave CROCKER wrote:
>
>
> On 8/11/2010 9:19 AM, Tony Hansen wrote:
>> I'm wondering if, instead of changing the 5321 guidance, there needs 
>> to be text
>> added to 4409bis about timeouts being shorter than those normally 
>> used by SMTP.
>
> Any useful guidance should be made wherever it is helpful.  This issue 
> is certainly in the realm of 'useful'.
>
> My note, here, isn't about that.  It's a 'process' question, meant 
> mostly for academic consideration:
>
>    Is this the sort of change that is appropriate for going from Draft 
> to Full?
>
> I would have thought that it was too technical and substantive and 
> that, at the least, the doc would have to cycle at, perhaps, Draft.
>
> To repeat:  this is meant purely as an academic exercise.  I'm curious.
>
> d/

Date: Wed, 11 Aug 2010 14:41:18 -0400
From: John C Klensin <[email-masked]>
To: [email-masked], "Murray S. Kucherawy" <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

--On Wednesday, August 11, 2010 10:19 -0700 Dave CROCKER
<[email-masked]> wrote:

>>> 	"Long delays after the<CRLF>.<CRLF>  is received can
>>> 	result in timeouts and duplicate messages.  Deferring
>>> 	detailed message analysis until after the SMTP
>>> 	connection has closed can result in non-delivery
>>> 	notifications, possibly sent to incorrect addresses.  A
>>> 	receiver-SMTP MUST carefully balance these two
>>> 	considerations, i.e., the time required to respond to
>>> 	the final<CRLF>.<CRLF>  end of data indicator and the
>>> 	desirable goal of rejecting undeliverable or
>>> 	unacceptable messages at SMTP time."
>> 
>> I like this text.  I think it reflects current operational
>> realities quite nicely.
> 
> Although we can't offer a precise algorithm, because we don't
> know enough and because network variances make this
> problematic to do at all, the draft text doesn't provide any
> assistance beyond "thar be dragons".  At the least, we should
> try to offer tradeoffs, factors, and may even (sudder) a
> heuristic.
> 
> For example, a delay time of up to 2 seconds is probably
> pretty safe.

Dave,

In the spirit of my recent response to Murray, let me suggest
that, if one wants in-depth guidance here, it would be
appropriate to construct a separate document that contains a
real analysis of the situation, the tradeoffs involved, some
numbers and the basis for them, etc.  Squeezing that into 5321
would almost certainly be too complex and might have to wait,
while a separate document could presumably be moved through the
system on its own merits.  If one could get consensus for the
recommendations that would be great.  Even if it could not,
publishing it as some insights into the subject would be useful.
The second option would not exist if one wanted to fold the
whole picture into 5321bis.  If such a document existed, Ned's
suggestion would apply symmetrically: a reference to Craig's
analysis for the "duplicates" part and a reference to the new
document for the "blowback, SMTP-time analysis, and tradeoffs"
one.

    john

Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
From: Steve Atkins <[email-masked]>
Date: Wed, 11 Aug 2010 11:40:04 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 11, 2010, at 11:16 AM, Dave CROCKER wrote:

> 
> 
> On 8/11/2010 10:28 AM, Steve Atkins wrote:
>> Many of the things you're going to want to do in that time will take
>> longer than that. There's a long tail, but I'd expect to see some
>> processing take a minute or more in rare cases.
> 
> 
> I believe that, to be meaningful, the guidance is going to have to specify a number that balances between leaving enough time for useful analysis versus avoiding the vast majority of timeouts.

I suggested two minutes in my wording, though I don't expect it to commonly exceed fifteen seconds.

> 
> A specification that encourages retransmission is a Bad Thing.  A specification that does not leave enough time for 'most' useful analysis is also not helpful.  Note that 'most' is a statistical assessment and that ultimately this topic is an efficiency hack.  Having some analysis be required to be deferred isn't wonderful, but it's better than having to defer all of it.
> 
> Taking your comment, I'll ask about a revision up to one minute for server analysis.
> 
> However the idea that every message submission across the net will require the sending side to hold for one minute, every time, bothers the heck out of me.

They're required to hold for ten minutes now. That won't often be needed, but they're required to do so if the SMTP server feels so inclined. 

The actual length of time an SMTP server will require an SMTP client to wait while it delivers the message is going to be driven by the operational requirements of the SMTP server (whether that be content-based analysis, internal proxying or even just waiting for an IO slot to write the message to disk), not by any RFC limits.

But that open connection consumes a significant amount of resources at both the sender and the receiver, so there's already a significant pressure to reduce it at large receivers. So I wouldn't expect it to increase without limit. But if there's a choice between occasionally keeping a connection open for another minute (possibly while you wait for other attempted deliveries from the same sender) and accepting mail for delivery that will then need to be bounced or discarded any decent engineer is going to do anything they can to avoid accepting that mail until they've made a delivery decision.

(The other alternative to holding the connection open at that point is simply to return a 450 response to <crlf>.<crlf>, then make delivery decisions at your leisure and cache those for when the SMTP client attempts redelivery. This causes howls of outrage from the SMTP client end of the relationship - and also causes much the same breakage as greylisting does).

I don't believe that legitimate senders of email are likely to commonly see really long delays at the end of data, and I don't much care if other senders are inconvenienced.

But none of that is going to change based on any changes to 5321#6.1 - all we're able to do there is modify the wording to more accurately reflect existing practice. Providing some background might convince implementors to make better decisions, but I suspect that's going to affect primarily the SMTP client developer, if anyone.

Cheers,
  Steve

Date: Wed, 11 Aug 2010 14:30:24 -0400
From: John C Klensin <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: RE: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

--On Wednesday, August 11, 2010 09:53 -0700 "Murray S.
Kucherawy" <[email-masked]> wrote:

> 
>> -----Original Message-----
>> From: [email-masked] [mailto:owner-ietf-
>> [email-masked]] On Behalf Of John C Klensin
>> Sent: Wednesday, August 11, 2010 8:15 AM
>> To: [email-masked]; SMTP Interest Group
>> Subject: Re: Changing RFC 5322 guidance about crlf.crlf
>> response delay
>> 
>> I note, again fwiw, that I've been trying to get various
>> advocates for a ban (or near-ban) on NDNs to write that
>> separate document and propose a specific model at regular
>> intervals since well before 2821 was completed.
> 
> I'm new to that particular topic.  Can you explain its
> motivation or point me to a discussion thread that lays it out
> so I can get some context?

Briefly, there was an argument in DRUMS when 2821 was being
developed, and a much longer and more intense argument when what
became 5321 was under discussion, that the nature of the
blowback problem was such that non-delivery notification
messages should be entirely banned unless the relevant SMTP
server was in a position to strongly authenticate the reverse
path.  

The people who felt most strongly about that believed that, if
the only alternative to generating an NDN was to simply drop the
message, the latter was preferable.  Of course, that would
change the Internet email model from one in which a sender could
assume delivery if there was no indication of a problem to one
in which delivery was uncertain and best-efforts, implying a
possible requirement for more delivery notifications.

DRUMS felt constrained to stay within existing practice and 5321
was constrained by both existing practice and the Proposed ->
Draft rules.  Whatever the merits of the various flavors of
"reject only, never bounce" argument, they represented a change.
There was also some question as to whether some variations were
realistic in a relay environment (a rejection to a relay could
force that relay to generate an NDN unless one just dropped the
message).  

So the people taking "reject only" positions were strongly
encouraged to generate I-Ds describing their models, see if they
could get consensus for them, and then see if they could get
sufficient implementations and interoperability testing to
"catch up" with 2821bis or 5321bis and be incorporated there.
For whatever reason, the drafts never appeared so we don't know
what might have happened if they had.

>> 	"Long delays after the <CRLF>.<CRLF> is received can
>> 	result in timeouts and duplicate messages.  Deferring
>> 	detailed message analysis until after the SMTP
>> 	connection has closed can result in non-delivery
>> 	notifications, possibly sent to incorrect addresses.  A
>> 	receiver-SMTP MUST carefully balance these two
>> 	considerations, i.e., the time required to respond to
>> 	the final <CRLF>.<CRLF> end of data indicator and the
>> 	desirable goal of rejecting undeliverable or
>> 	unacceptable messages at SMTP time."
> 
> I like this text.  I think it reflects current operational
> realities quite nicely.

Thanks.  Certainly we have known since 1123 that there was an
operational design tradeoff in this area.  The original 1123/
2821/ 5321 text just about said that.  If we can explain the
tradeoff a little better, even while avoiding making a specific
recommendation about the decision (or going all the way to
"never bounce"), that probably helps.

    john

Date: Wed, 11 Aug 2010 11:16:41 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

On 8/11/2010 10:28 AM, Steve Atkins wrote:
> Many of the things you're going to want to do in that time will take
> longer than that. There's a long tail, but I'd expect to see some
> processing take a minute or more in rare cases.

I believe that, to be meaningful, the guidance is going to have to specify a 
number that balances between leaving enough time for useful analysis versus 
avoiding the vast majority of timeouts.

A specification that encourages retransmission is a Bad Thing.  A specification 
that does not leave enough time for 'most' useful analysis is also not helpful. 
  Note that 'most' is a statistical assessment and that ultimately this topic is 
an efficiency hack.  Having some analysis be required to be deferred isn't 
wonderful, but it's better than having to defer all of it.

Taking your comment, I'll ask about a revision up to one minute for server 
analysis.

However the idea that every message submission across the net will require the 
sending side to hold for one minute, every time, bothers the heck out of me.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 11 Aug 2010 14:10:57 -0400
From: John C Klensin <[email-masked]>
To: [email-masked], Tony Hansen <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: Draft to Full, versus cycling at Draft
Sender: [email-masked]

--On Wednesday, August 11, 2010 09:41 -0700 Dave CROCKER
<[email-masked]> wrote:

>...
> My note, here, isn't about that.  It's a 'process' question,
> meant mostly for academic consideration:
> 
>     Is this the sort of change that is appropriate for going
> from Draft to Full?
> 
> I would have thought that it was too technical and substantive
> and that, at the least, the doc would have to cycle at,
> perhaps, Draft.
> 
> To repeat:  this is meant purely as an academic exercise.  I'm
> curious.

Dave, purely personal opinion and in the context of your
proposed academic exercise...

My understanding of the "what can be changed during advancement
in grade" question has always been "no new requirements that
would cause a conforming implementation to become
non-conforming".   That actually raises a separate, even more
academic, question, which is whether violating that rule forces
recycling at Draft or sends one back to Proposed.  If there were
really a new requirement, one might reasonably be required to
publish and then demonstrate that the condition was implemented
and described in an interoperable way.  That would require going
through Proposed under the existing rules.

For the particular 5381 case, my suggested text was quite
consciously intended to avoid running afoul of that question.
The existing text is guidance, my proposed new text is guidance,
and all it really says is that SMTP server implementations are
required to consider the tradeoffs.  That isn't a testable
conformance requirement so the issue of change of grade should
be irrelevant.

Actually changing the timeout requirement would be a
conformance-affecting change but, given that it is stated as a
minimum time to wait, I think only if we increased it.

academically and hair-splittingly yours,
   john

Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
From: Steve Atkins <[email-masked]>
Date: Wed, 11 Aug 2010 10:28:53 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 11, 2010, at 10:19 AM, Dave CROCKER wrote:

> 
> 
> 
> On 8/11/2010 9:53 AM, Murray S. Kucherawy wrote:
>> 
>> 
>>> 	"Long delays after the<CRLF>.<CRLF>  is received can
>>> 	result in timeouts and duplicate messages.  Deferring
>>> 	detailed message analysis until after the SMTP
>>> 	connection has closed can result in non-delivery
>>> 	notifications, possibly sent to incorrect addresses.  A
>>> 	receiver-SMTP MUST carefully balance these two
>>> 	considerations, i.e., the time required to respond to
>>> 	the final<CRLF>.<CRLF>  end of data indicator and the
>>> 	desirable goal of rejecting undeliverable or
>>> 	unacceptable messages at SMTP time."
>> 
>> I like this text.  I think it reflects current operational realities quite nicely.
> 
> Although we can't offer a precise algorithm, because we don't know enough and because network variances make this problematic to do at all, the draft text doesn't provide any assistance beyond "thar be dragons".  At the least, we should try to offer tradeoffs, factors, and may even (sudder) a heuristic.
> 
> For example, a delay time of up to 2 seconds is probably pretty safe.

Many of the things you're going to want to do in that time will take
longer than that. There's a long tail, but I'd expect to see some
processing take a minute or more in rare cases.

What's the failure mode at that point in the transaction - are we
defending against a transport failure, or the client violating spec
with timeouts or...?

Cheers,
  Steve

From: [email-masked]
Cc: SMTP Interest Group <[email-masked]>
Date: Wed, 11 Aug 2010 10:22:05 -0700 (PDT)
Subject: RE: Changing RFC 5322 guidance about crlf.crlf response delay
To: "Murray S. Kucherawy" <[email-masked]>
Sender: [email-masked]

> > 	"Long delays after the <CRLF>.<CRLF> is received can
> > 	result in timeouts and duplicate messages.  Deferring
> > 	detailed message analysis until after the SMTP
> > 	connection has closed can result in non-delivery
> > 	notifications, possibly sent to incorrect addresses.  A
> > 	receiver-SMTP MUST carefully balance these two
> > 	considerations, i.e., the time required to respond to
> > 	the final <CRLF>.<CRLF> end of data indicator and the
> > 	desirable goal of rejecting undeliverable or
> > 	unacceptable messages at SMTP time."

> I like this text.  I think it reflects current operational realities quite
> nicely.

I agree. The only suggestion I have is that an informational referenece to
Craig Partridge's oringal document on the timeout issue might be helpful to
include after the first sentence. There's no reason to have a lengthy
discussion here of why timeouts can lead to duplicates, but a reference to such
a discussion makes sense IMO.

				Ned

Date: Wed, 11 Aug 2010 10:19:02 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Murray S. Kucherawy" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

On 8/11/2010 9:53 AM, Murray S. Kucherawy wrote:
>
>> -----Original Message-----
>> From: [email-masked] [mailto:owner-ietf-
>> [email-masked]] On Behalf Of John C Klensin
>> Sent: Wednesday, August 11, 2010 8:15 AM
>> To: [email-masked]; SMTP Interest Group
>> Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
>>
>> I note, again fwiw, that I've been trying to get various
>> advocates for a ban (or near-ban) on NDNs to write that separate
>> document and propose a specific model at regular intervals since
>> well before 2821 was completed.
>
> I'm new to that particular topic.  Can you explain its motivation or point me to a discussion thread that lays it out so I can get some context?
>
>> 	"Long delays after the<CRLF>.<CRLF>  is received can
>> 	result in timeouts and duplicate messages.  Deferring
>> 	detailed message analysis until after the SMTP
>> 	connection has closed can result in non-delivery
>> 	notifications, possibly sent to incorrect addresses.  A
>> 	receiver-SMTP MUST carefully balance these two
>> 	considerations, i.e., the time required to respond to
>> 	the final<CRLF>.<CRLF>  end of data indicator and the
>> 	desirable goal of rejecting undeliverable or
>> 	unacceptable messages at SMTP time."
>
> I like this text.  I think it reflects current operational realities quite nicely.

Although we can't offer a precise algorithm, because we don't know enough and 
because network variances make this problematic to do at all, the draft text 
doesn't provide any assistance beyond "thar be dragons".  At the least, we 
should try to offer tradeoffs, factors, and may even (sudder) a heuristic.

For example, a delay time of up to 2 seconds is probably pretty safe.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

X-Mail-Sent-By-AEGEE.org-Account: didopalauzov
Date: Wed, 11 Aug 2010 19:11:49 +0200
From: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Hello,

if rejecting after CRLF.CRLF is better than bouncing, what about 
discussing draft-hall-prdr-00 and doing it more normative?

For an email with two recipients, the first one of who wants to receive 
the mail and the second -- not, there is currently no option but to say 
250 OK after CRLF.CRLF and send later DSN.  With the idea described in 
draft-hall-prdr-00, it is possible to avoid the currently unavoidable 
DSN.  Any suggestion, how this draft can be revived?

On 10.08.2010 20:18, David MacQuigg wrote:
> My system is too small to draw any conclusions, but my impression is
> that we are overly-concerned about a very small fraction of the mailflow
> - those few messages that are legit, but get a false reject from a
> statistical filter. We really don't care about the proper SMTP response
> if it is spam. The only advantage I see in running the spam filter
> before ACCEPT is a few less messages at the bottom of my spam bucket. I
> wish I had some numbers from a larger system.

A user might have installed redirect for her incoming mails to another 
mail provider.  The other provider might SMTP reject incoming spam.  If 
there is spam for your user, your server does accept it, then sends it 
to the other mail provider, and that second server SMTP rejects it, then 
your server will have to send DSN for that SPAM (provided that you do 
not want to discard emails).  With rejection-preferences similar on both 
servers, the amount of such DSNs will decrease, when both servers reject 
after DATA.  You have no influence on what other servers do with the 
spam (smtp rejecting or inserting in spam folders).

> Would be nice if SMTP had a 'conditional ACCEPT': "Sorry, your
> transmitter ID 'yahoo.com' cannot be verified, Your message has been
> routed to the recipient's quarantine." Imagine the effect that would
> have on all the lame excuses for an invalid or hard-to-verify HELO name.

What do you want to communicate to the sender with this conditional 
ACCEPT: "Dear Sender, your suspicious email has arrived.  It might be 
read soon, it might be read late, or it might not be read at all."?

The problem with the spam folders are the false positives.  Many users 
do check the spam folders/quarantines not that often as their Inbox and 
with less attention.  This might lead to reading a message too late, or 
even overseeing it.  I prefer to immediately notify the sender that her 
message was not delivered, including in the SMTP response alternative 
ways to contact the recipient (and spamassassin-evaluation of the mail), 
rather than let the sender hope that her email was properly delivered, 
while there is a risk that the recipient oversees that message.

Със здраве
   Дилян

Date: Wed, 11 Aug 2010 10:05:54 -0700
To: Tony Hansen <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Cc: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

Hi Tony,
At 09:19 11-08-10, Tony Hansen wrote:
>I'm wondering if, instead of changing the 5321 guidance, there needs 
>to be text added to 4409bis about timeouts being shorter than those 
>normally used by SMTP.

I vague recall that the question of timeouts came up during a 
discussion about 4409bis.

Regards,
-sm 

Date: Wed, 11 Aug 2010 10:01:57 -0700
To: [email-masked]
From: SM <[email-masked]>
Subject: Re: Processing after the end of DATA
Cc: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

Hi Dave,
At 08:44 11-08-10, Dave CROCKER wrote:
>Isn't this effectively a form of greylisting?

Yes, and it is quite effective for greylisting.

>I've seen mixed results with greylisting, given that some legit 
>senders have SMTP clients that are, well, problematic.  (Nearly lost 
>a consulting gig because the initial query note got mishandled from 
>greylisting.)

It also requires some whitelisting to deal with problematic hosts.

Regards,
-sm 

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Wed, 11 Aug 2010 09:53:44 -0700
Subject: RE: Changing RFC 5322 guidance about crlf.crlf response delay
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:owner-ietf-
> [email-masked]] On Behalf Of John C Klensin
> Sent: Wednesday, August 11, 2010 8:15 AM
> To: [email-masked]; SMTP Interest Group
> Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
> 
> I note, again fwiw, that I've been trying to get various
> advocates for a ban (or near-ban) on NDNs to write that separate
> document and propose a specific model at regular intervals since
> well before 2821 was completed.

I'm new to that particular topic.  Can you explain its motivation or point me to a discussion thread that lays it out so I can get some context?

> 	"Long delays after the <CRLF>.<CRLF> is received can
> 	result in timeouts and duplicate messages.  Deferring
> 	detailed message analysis until after the SMTP
> 	connection has closed can result in non-delivery
> 	notifications, possibly sent to incorrect addresses.  A
> 	receiver-SMTP MUST carefully balance these two
> 	considerations, i.e., the time required to respond to
> 	the final <CRLF>.<CRLF> end of data indicator and the
> 	desirable goal of rejecting undeliverable or
> 	unacceptable messages at SMTP time."

I like this text.  I think it reflects current operational realities quite nicely.

Date: Wed, 11 Aug 2010 09:41:58 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Tony Hansen <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Draft to Full, versus cycling at Draft
Sender: [email-masked]

On 8/11/2010 9:19 AM, Tony Hansen wrote:
> I'm wondering if, instead of changing the 5321 guidance, there needs to be text
> added to 4409bis about timeouts being shorter than those normally used by SMTP.

Any useful guidance should be made wherever it is helpful.  This issue is 
certainly in the realm of 'useful'.

My note, here, isn't about that.  It's a 'process' question, meant mostly for 
academic consideration:

    Is this the sort of change that is appropriate for going from Draft to Full?

I would have thought that it was too technical and substantive and that, at the 
least, the doc would have to cycle at, perhaps, Draft.

To repeat:  this is meant purely as an academic exercise.  I'm curious.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 11 Aug 2010 12:19:19 -0400
From: Tony Hansen <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

One comment below

On 8/11/2010 11:14 AM, John C Klensin wrote:
> Dave,
>
> You do mean 5321, don't you?
>
> Of course, if the IESG hadn't put YAM into constipation, we
> could modify the pre-eval document and not have to have a
> discussion about errata.  :-(
>
> FWIW,
>
> (1) My recommendation would be to retain the timeout, but note
> that server implementations should be aware that some clients
> will ignore the spec for operational reasons and apply a much
> smaller number.  My guess is that a discussion of the
> appropriateness of doing that doesn't belong in an erratum/
> corrigendum to 5321 but in a separate document (if at all).
>    
Most of the notes about clients severely ignoring the spec appear to be 
from MUAs that should really be using Submit instead of SMTP.

I'm wondering if, instead of changing the 5321 guidance, there needs to 
be text added to 4409bis about timeouts being shorter than those 
normally used by SMTP.

     Tony Hansen

> I note, again fwiw, that I've been trying to get various
> advocates for a ban (or near-ban) on NDNs to write that separate
> document and propose a specific model at regular intervals since
> well before 2821 was completed.
>
>
> (2) How would people feel about moving toward changing the
> "substantive text" to read something like:
>
> 	"To avoid receiving duplicate messages as the result of
> 	timeouts, a receiver-SMTP MUST carefully balance the
> 	time required to respond to the final<CRLF>.<CRLF>  end
> 	of data indicator against the desirable goal of
> 	rejecting undeliverable or unacceptable messages at SMTP
> 	time rather than generating NDN messages later. "
>
> I'll have to check whether "NDN messages" is the right reference
> at that point, but it seems to me from the discussion that the
> above should be about the right intent.   We could go even a
> little further and recast the paragraph as something like:
>
> 	"Long delays after the<CRLF>.<CRLF>  is received can
> 	result in timeouts and duplicate messages.  Deferring
> 	detailed message analysis until after the SMTP
> 	connection has closed can result in non-delivery
> 	notifications, possibly sent to incorrect addresses.  A
> 	receiver-SMTP MUST carefully balance these two
> 	considerations, i.e., the time required to respond to
> 	the final<CRLF>.<CRLF>  end of data indicator and the
> 	desirable goal of rejecting undeliverable or
> 	unacceptable messages at SMTP time."
>
> best,
>     john (as your friendly consensus-driven editor)
>
>
> --On Wednesday, August 11, 2010 06:52 -0700 Dave CROCKER
> <[email-masked]>  wrote:
>
>    
>> Folks,
>>
>> Responses to my question make it pretty clear (to me, at
>> least) that the current text in 5322 has become off the mark,
>> regarding best current practices for crlf.crlf response delay.
>>
>> At the time the guidance was developed, the philosophy was to
>> reduce delay to the minimum and, therefore, defer any post
>> crlf.crlf processing until after the SMTP session.  I'm
>> hearing a reasonably strong consensus that it is now
>> acceptable -- and possibly essential -- to do some serious
>> processing before responding.
>>
>> We should consider creating a Errata to cover this, but it is
>> not yet obvious to me exactly what the text should be.
>>
>> I assume that no one is suggesting changing the 10-minute
>> timeout.  However, we've heard that some Senders don't honor
>> it.  We should consider whether to ignore these
>> non-conformists versus change the time out.  If the latter,
>> what is a more appropriate timeout?
>>
>> Whatever the timeout, the consensus appears to be that it is
>> OK to have "some" delay in responding.
>>
>> Since there will still be a timeout defining an upper limit
>> and there will still be highly variable network delays meaning
>> that the processing can't be allowed to come close to the
>> timeout, it cannot be acceptable to give guidance that
>> essentially says "take as long as you like before responding
>> to the crlf.crlf."   So, what guidance should be given?
>>
>> The substantive issue is the text:
>>
>>      
>>> "To avoid receiving duplicate messages as the result of
>>> timeouts, a receiver-SMTP MUST seek to minimize the time
>>>     required to respond to the final<CRLF>.<CRLF>  end of data
>>>     indicator. "
>>>        
>> Does anyone suggest considering modification to any other text
>> in the document?
>>
>> Does anyone have useful text to suggest for adding or changing?
>>
>> d/
>>      
>
>
>
>    

Date: Wed, 11 Aug 2010 08:44:54 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
CC: "Rosenwald, Jordan" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/11/2010 8:30 AM, [email-masked] wrote:
>> Completely agree for several reasons, including that the only senders
>> that take advantage of "as fast as you can process" scenarios are
>> spammers.  If I take a second or two to accept a message from a legit
>> sender they neither know nor care.  But to a spammer, the faster we
>> receive and delivery the more mail gets through before a content filter
>> or blocklist can update.  We saw this happen when we migrated to our
>> current system 3 years ago.  Spam was delivered at many hundreds of
>> times faster than mail from AOL, Hotmail and Gmail combined. Responding
>> as quickly as my system will allow with<CRLF>.<CRLF>  helps no one I
>> want to help.
>
> You know, that's very nicely put. I'm going to use that last line the next time
> this comes up.

Isn't this effectively a form of greylisting?

I've seen mixed results with greylisting, given that some legit senders have 
SMTP clients that are, well, problematic.  (Nearly lost a consulting gig because 
the initial query note got mishandled from greylisting.)

In any event, to the extent it's good to add some delay, the question is how 
much.  The other question is why this is good for long-term -- since 
specifications last for a decade or two -- and that bad actors won't simply 
adapt as the behavior becomes institutionalized.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 11 Aug 2010 08:39:16 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Arnt Gulbrandsen <[email-masked]>
CC: Dave CROCKER <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/11/2010 8:32 AM, Arnt Gulbrandsen wrote:
>> I am actually hoping for normative language.
>
> I know, but I'm not up to writing politely today. Brokenware has been riling me
> all week and my brain's affected. Sorry (about both parts).

Hmnmm...This sub-thread suggests a potentially useful April 1 draft, discussing 
various linguistic templates for normative and non-normative language that 
varies in its style for encoding affective and judgmental sub-text...

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: [email-masked]
Cc: SMTP Interest Group <[email-masked]>
Date: Wed, 11 Aug 2010 08:30:29 -0700 (PDT)
Subject: RE: Processing after the end of DATA
To: "Rosenwald, Jordan" <[email-masked]>
Sender: [email-masked]

> On Aug 9, 2010, at 5:10 PM, Dave CROCKER wrote:
> > I did a 10-second scan of 5321 and didn't find the relevant text, but
> I'd like
> > to resolve -- and preferably squash -- this issue quickly.

> Just came into the office and saw nearly 20 emails on this.  Great
> conversation, but not "quickly" :)

> At 19:27 09-08-10, [email-masked] wrote:
> > I have a somewhat different take on this. First of all, I have always
> > thought the admonition that you MUST minimize the amount of time spent

> > before responding to the trailing dot to the greatest extent possible
> > was, well, bunk.
> > (It's also an effectively unenforceable MUST - who can say you've done

> > all you can or not? - which is bad in its own right.) While it is
> > important not to spend too much time, the difference between a
> > millisecond delay and a 2 second delay is, in this situation, not
> worth worrying about.

> Completely agree for several reasons, including that the only senders
> that take advantage of "as fast as you can process" scenarios are
> spammers.  If I take a second or two to accept a message from a legit
> sender they neither know nor care.  But to a spammer, the faster we
> receive and delivery the more mail gets through before a content filter
> or blocklist can update.  We saw this happen when we migrated to our
> current system 3 years ago.  Spam was delivered at many hundreds of
> times faster than mail from AOL, Hotmail and Gmail combined. Responding
> as quickly as my system will allow with <CRLF>.<CRLF> helps no one I
> want to help.

You know, that's very nicely put. I'm going to use that last line the next time
this comes up.

				Ned

Date: Wed, 11 Aug 2010 17:32:27 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: [email-masked]
Cc: Dave CROCKER <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 08/11/2010 05:22 PM, Dave CROCKER wrote:
> As specification language goes, I don't know what directives like
> "should keep in mind" means in practical terms.

The technical term for that is "handwaving". I got a fine specimen into 
a finished RFC. A particular kind of probably inadvisable behaviour 
"causes bad karma".

> I am actually hoping for normative language.

I know, but I'm not up to writing politely today. Brokenware has been 
riling me all week and my brain's affected. Sorry (about both parts).

Arnt

Date: Wed, 11 Aug 2010 08:22:18 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Arnt Gulbrandsen <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/11/2010 8:10 AM, Arnt Gulbrandsen wrote:
> True. Sorry about contributing to topic drift.

oh, the excursion was fairly brief and well-indulged, IMO.  Small rounds of this 
sort of review are probably good for the community collective...

> I think my answer is that the receiver's permissible delay is governed by three
> concerns:
>
> 1. Doing as much as possible so as not to provoke blowback. Wasting an SMTP
> client's time is not so bad, wasting a human reader's time is very bad.

I certainly took a community consensus that SMTP-time rejection is FAR better 
than a later DSN.  And it might make sense to have some text that highlights 
this preference.

> 2. Staying within the client's specified time limit, ie. n minutes minus RTT,
> and noting that RTT can still be high: "The client is not required to wait
> longer than n minutes, and while the network is less temperamental now than it
> the past, RTTs of 1-2 minutes may still be observed". (In the nineties an Indian
> friend of mine suffered RTTs up to half an hour. I'm just guessing that two
> minutes may be possible now, since people do use VPN over TCP and such horrors.)

There's a famous story of Bob Braden, from ISI, having a sabbatical in England 
and having his connection back to Los Angeles hang.  He went for dinner and came 
back some hours later.  The connection resumed...

My guess is that the RTT factor needs a multiple, to account for spikes in 
delay.  Besides deciding on what the multiple should be, I'm a bit skittish 
about our playing with that number at all, rather than something more 
simplistic.  This just doesn't seem like an activity that should entail a 
sophisticated calculation and RTT counts as sophisticated, IMO.

> 3. Accommodating standards-violating clients and NATboxes that drop connections
> much too quickly. "Implementers should keep in mind that in the past, some NAT
> implementations and clients have been observed to drop connections after less
> than a minute" or whatever.

As specification language goes, I don't know what directives like "should keep 
in mind" means in practical terms.  (This is the second time this week I've seen 
language like this creep into specification discussions; clearly it's a common 
issue.)  I think we the purpose of a specification is to specify and that the 
question, in this case, is whether to recommend a different timeout and/or to 
recommend how the other side should handle a shorter timeout.  (I think the 
latter has no solution.)

> I think that 4.5.3.2 could/should grow a paragraph mentioning these factors,
> without upper case, and that they apply to all the client timeouts in principle,
> but particularly 4.5.3.2.6 in practice.

I am actually hoping for normative language.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 11 Aug 2010 16:17:01 +0100
From: Paul Smith <[email-masked]>
To: Hector Santos <[email-masked]>
CC: Dave CROCKER <[email-masked]>, Tony Finch <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

  On 11/08/2010 15:30, Hector Santos wrote:
> +1. If we are looking for insights for "adjusted" recommendations, 
> this would be a good time as there is no doubt the direction and 
> growth of DATA level processing is permanent now.  The hardware allows 
> for these new framework feasible in a scaled up manner and there is no 
> way we can tell people "don't do that" and go back to old school 
> methods of having post-SMTP (post acceptance) only data processing.
>
> Probably a BCP or a new I-D for SMTP DATA Processing Filtering 
> Guidelines would be very welcome and useful.
>
> I would not mind drafting it if people think it is a good idea.
>
Daft question, but is there room for an SMTP extension here, given that 
some senders really don't like waiting 10 minutes (or even 1 minute in 
some cases) for a response for some reason:

- EHLO keyword 'DEFERREDSTATUS <x seconds>

- SENDER commands 'DEFERSTATUS' and 'GETDEFERRED'

Transaction

S: 220 ESMTP server ready
C: EHLO mydomain.com
S: 220-DEFERREDSTATUS 600
S: 220 HELP
C: DEFERSTATUS
S: 250 OK
C: MAIL FROM, RCPT TO, DATA etc
C: .
S: 250 2.5.6 Message accepted for now - id: bibblebobble237
(or could give other 2/4/5xx response immediately)
C: QUIT

<at least 10 minutes (600 seconds) later>

S: 220 ESMTP server ready
C: EHLO mydomain.com
S: 220-DEFERREDSTATUS 600
S: 220 HELP
C: GETDEFERRED bibblebobble237
S: 550 Don't like that message
C: QUIT

So, the sender could choose to have a quick turnaround and get responses 
later, or wait and get them now. When the client choose deferred status 
notification the server can process later, but has no responsibility to 
generate a DSN.

Servers could still give immediate failures if they want, but if they 
give a "deferred" code (I chose 2.5.6 out of thin air), then the sender 
comes back later to find out what happened to the message.

Obviously the server could still lie about the response (eg give a 2xx 
response when it really threw away spam) or take a few seconds before 
deciding whether to give an immediate or deferred response, as they do 
now, it could just cut down on duplicate processing.

Date: Wed, 11 Aug 2010 11:15:48 -0400
From: John C Klensin <[email-masked]>
To: [email-masked], SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

(sorry -- wrong sender address used again)

Dave,

You do mean 5321, don't you?

Of course, if the IESG hadn't put YAM into constipation, we
could modify the pre-eval document and not have to have a
discussion about errata.  :-(

FWIW, 

(1) My recommendation would be to retain the timeout, but note
that server implementations should be aware that some clients
will ignore the spec for operational reasons and apply a much
smaller number.  My guess is that a discussion of the
appropriateness of doing that doesn't belong in an erratum/
corrigendum to 5321 but in a separate document (if at all).

I note, again fwiw, that I've been trying to get various
advocates for a ban (or near-ban) on NDNs to write that separate
document and propose a specific model at regular intervals since
well before 2821 was completed.

(2) How would people feel about moving toward changing the
"substantive text" to read something like:

	"To avoid receiving duplicate messages as the result of
	timeouts, a receiver-SMTP MUST carefully balance the
	time required to respond to the final <CRLF>.<CRLF> end
	of data indicator against the desirable goal of
	rejecting undeliverable or unacceptable messages at SMTP
	time rather than generating NDN messages later. "

I'll have to check whether "NDN messages" is the right reference
at that point, but it seems to me from the discussion that the
above should be about the right intent.   We could go even a
little further and recast the paragraph as something like:

	"Long delays after the <CRLF>.<CRLF> is received can
	result in timeouts and duplicate messages.  Deferring
	detailed message analysis until after the SMTP
	connection has closed can result in non-delivery
	notifications, possibly sent to incorrect addresses.  A
	receiver-SMTP MUST carefully balance these two
	considerations, i.e., the time required to respond to
	the final <CRLF>.<CRLF> end of data indicator and the
	desirable goal of rejecting undeliverable or
	unacceptable messages at SMTP time."

best,
   john (as your friendly consensus-driven editor)

--On Wednesday, August 11, 2010 06:52 -0700 Dave CROCKER
<[email-masked]> wrote:

> 
> Folks,
> 
> Responses to my question make it pretty clear (to me, at
> least) that the current text in 5322 has become off the mark,
> regarding best current practices for crlf.crlf response delay.
> 
> At the time the guidance was developed, the philosophy was to
> reduce delay to the minimum and, therefore, defer any post
> crlf.crlf processing until after the SMTP session.  I'm
> hearing a reasonably strong consensus that it is now
> acceptable -- and possibly essential -- to do some serious
> processing before responding.
> 
> We should consider creating a Errata to cover this, but it is
> not yet obvious to me exactly what the text should be.
> 
> I assume that no one is suggesting changing the 10-minute
> timeout.  However, we've heard that some Senders don't honor
> it.  We should consider whether to ignore these
> non-conformists versus change the time out.  If the latter,
> what is a more appropriate timeout?
> 
> Whatever the timeout, the consensus appears to be that it is
> OK to have "some" delay in responding.
> 
> Since there will still be a timeout defining an upper limit
> and there will still be highly variable network delays meaning
> that the processing can't be allowed to come close to the
> timeout, it cannot be acceptable to give guidance that
> essentially says "take as long as you like before responding
> to the crlf.crlf."   So, what guidance should be given?
> 
> The substantive issue is the text:
> 
>> "To avoid receiving duplicate messages as the result of
>> timeouts, a receiver-SMTP MUST seek to minimize the time
>>    required to respond to the final <CRLF>.<CRLF> end of data
>>    indicator. "
> 
> Does anyone suggest considering modification to any other text
> in the document?
> 
> Does anyone have useful text to suggest for adding or changing?
> 
> d/

Date: Wed, 11 Aug 2010 11:14:49 -0400
From: John C Klensin <[email-masked]>
To: [email-masked], SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

Dave,

You do mean 5321, don't you?

Of course, if the IESG hadn't put YAM into constipation, we
could modify the pre-eval document and not have to have a
discussion about errata.  :-(

FWIW, 

(1) My recommendation would be to retain the timeout, but note
that server implementations should be aware that some clients
will ignore the spec for operational reasons and apply a much
smaller number.  My guess is that a discussion of the
appropriateness of doing that doesn't belong in an erratum/
corrigendum to 5321 but in a separate document (if at all).

I note, again fwiw, that I've been trying to get various
advocates for a ban (or near-ban) on NDNs to write that separate
document and propose a specific model at regular intervals since
well before 2821 was completed.

(2) How would people feel about moving toward changing the
"substantive text" to read something like:

	"To avoid receiving duplicate messages as the result of
	timeouts, a receiver-SMTP MUST carefully balance the
	time required to respond to the final <CRLF>.<CRLF> end
	of data indicator against the desirable goal of
	rejecting undeliverable or unacceptable messages at SMTP
	time rather than generating NDN messages later. "

I'll have to check whether "NDN messages" is the right reference
at that point, but it seems to me from the discussion that the
above should be about the right intent.   We could go even a
little further and recast the paragraph as something like:

	"Long delays after the <CRLF>.<CRLF> is received can
	result in timeouts and duplicate messages.  Deferring
	detailed message analysis until after the SMTP
	connection has closed can result in non-delivery
	notifications, possibly sent to incorrect addresses.  A
	receiver-SMTP MUST carefully balance these two
	considerations, i.e., the time required to respond to
	the final <CRLF>.<CRLF> end of data indicator and the
	desirable goal of rejecting undeliverable or
	unacceptable messages at SMTP time."

best,
   john (as your friendly consensus-driven editor)

--On Wednesday, August 11, 2010 06:52 -0700 Dave CROCKER
<[email-masked]> wrote:

> 
> Folks,
> 
> Responses to my question make it pretty clear (to me, at
> least) that the current text in 5322 has become off the mark,
> regarding best current practices for crlf.crlf response delay.
> 
> At the time the guidance was developed, the philosophy was to
> reduce delay to the minimum and, therefore, defer any post
> crlf.crlf processing until after the SMTP session.  I'm
> hearing a reasonably strong consensus that it is now
> acceptable -- and possibly essential -- to do some serious
> processing before responding.
> 
> We should consider creating a Errata to cover this, but it is
> not yet obvious to me exactly what the text should be.
> 
> I assume that no one is suggesting changing the 10-minute
> timeout.  However, we've heard that some Senders don't honor
> it.  We should consider whether to ignore these
> non-conformists versus change the time out.  If the latter,
> what is a more appropriate timeout?
> 
> Whatever the timeout, the consensus appears to be that it is
> OK to have "some" delay in responding.
> 
> Since there will still be a timeout defining an upper limit
> and there will still be highly variable network delays meaning
> that the processing can't be allowed to come close to the
> timeout, it cannot be acceptable to give guidance that
> essentially says "take as long as you like before responding
> to the crlf.crlf."   So, what guidance should be given?
> 
> The substantive issue is the text:
> 
>> "To avoid receiving duplicate messages as the result of
>> timeouts, a receiver-SMTP MUST seek to minimize the time
>>    required to respond to the final <CRLF>.<CRLF> end of data
>>    indicator. "
> 
> Does anyone suggest considering modification to any other text
> in the document?
> 
> Does anyone have useful text to suggest for adding or changing?
> 
> d/

Date: Wed, 11 Aug 2010 17:10:43 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: Dave CROCKER <[email-masked]>
Cc: Tony Finch <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 08/11/2010 03:42 PM, Dave CROCKER wrote:
> But as Tony notes, the specific nature of the action after <CRLF>.<CRLF>
> wasn't my question. My question was solely about SMTP Receiver delay in
> making the response... for whatever reason -- and note that spam
> filtering was actually the trigger condition for the question, not
> message relaying. But again, the details weren't part of the question.

True. Sorry about contributing to topic drift.

I think my answer is that the receiver's permissible delay is governed 
by three concerns:

1. Doing as much as possible so as not to provoke blowback. Wasting an 
SMTP client's time is not so bad, wasting a human reader's time is very bad.

2. Staying within the client's specified time limit, ie. n minutes minus 
RTT, and noting that RTT can still be high: "The client is not required 
to wait longer than n minutes, and while the network is less 
temperamental now than it the past, RTTs of 1-2 minutes may still be 
observed". (In the nineties an Indian friend of mine suffered RTTs up to 
half an hour. I'm just guessing that two minutes may be possible now, 
since people do use VPN over TCP and such horrors.)

3. Accommodating standards-violating clients and NATboxes that drop 
connections much too quickly. "Implementers should keep in mind that in 
the past, some NAT implementations and clients have been observed to 
drop connections after less than a minute" or whatever.

I think that 4.5.3.2 could/should grow a paragraph mentioning these 
factors, without upper case, and that they apply to all the client 
timeouts in principle, but particularly 4.5.3.2.6 in practice.

Arnt

Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
From: Steve Atkins <[email-masked]>
Date: Wed, 11 Aug 2010 08:01:21 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 11, 2010, at 6:52 AM, Dave CROCKER wrote:

> 
> Folks,
> 
> Responses to my question make it pretty clear (to me, at least) that the current text in 5322 has become off the mark, regarding best current practices for crlf.crlf response delay.
> 
> At the time the guidance was developed, the philosophy was to reduce delay to the minimum and, therefore, defer any post crlf.crlf processing until after the SMTP session.  I'm hearing a reasonably strong consensus that it is now acceptable -- and possibly essential -- to do some serious processing before responding.
> 
> We should consider creating a Errata to cover this, but it is not yet obvious to me exactly what the text should be.
> 
> I assume that no one is suggesting changing the 10-minute timeout.  However, we've heard that some Senders don't honor it.  We should consider whether to ignore these non-conformists versus change the time out.  If the latter, what is a more appropriate timeout?

Ignore the non-conformists. As with many other bits of 5321, ignoring the timeouts can make delivery of mail more economical in resources at the expense of reliability. That's a reasonable operational decision to make, in some cases, but not something to encourage. Adding an explanatory note as to why that timeout is so long might help implementors make better decisions, though.

> 
> Whatever the timeout, the consensus appears to be that it is OK to have "some" delay in responding.
> 
> Since there will still be a timeout defining an upper limit and there will still be highly variable network delays meaning that the processing can't be allowed to come close to the timeout, it cannot be acceptable to give guidance that essentially says "take as long as you like before responding to the crlf.crlf."  So, what guidance should be given?
> 
> The substantive issue is the text:
> 
>> "To avoid receiving duplicate messages as the result of timeouts, a
>>   receiver-SMTP MUST seek to minimize the time required to respond to
>>   the final <CRLF>.<CRLF> end of data indicator. "

"A receiver-SMTP SHOULD reject a message it does not intend to deliver
during the SMTP transaction rather than sending an error message to
the reverse-path after accepting the message if it can determine that
before responding to the <CRLF>.<CRLF> end of data indicator.

However, to avoid receiving duplicate messages a receiver-SMTP
SHOULD minimize the time required to respond the <CRLF>.<CRLF>
end of data indicator and MUST respond within X minutes."

(Where X of about 2 seems reasonable).

The first paragraph has broader implications, but I think they're
a good thing to consider.

Cheers,
  Steve

> 
> Does anyone suggest considering modification to any other text in the document?
> 
> Does anyone have useful text to suggest for adding or changing?
> 
> d/
> -- 
> 
>  Dave Crocker
>  Brandenburg InternetWorking
>  bbiw.net
> 

Date: Wed, 11 Aug 2010 10:48:49 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

Dave CROCKER wrote:
> 
> The substantive issue is the text:
> 
>> "To avoid receiving duplicate messages as the result of timeouts, a
>>    receiver-SMTP MUST seek to minimize the time required to respond to
>>    the final <CRLF>.<CRLF> end of data indicator. "
> 
> Does anyone suggest considering modification to any other text in the 
> document?
> 
> Does anyone have useful text to suggest for adding or changing?

I think its fine, but the only thing I can see is adding insights 
requiring real timeout values seen in practice.  Its not 10 mins.

    "Practice has shown the recommended 10 minute timeout is not
     common place and closer to 3-5 minutes.  While receivers should
     DATA processing needs make this process as fast as possible, it
     SHOULD limit accumulated DATA processing time to less than
     5 minutes."

My concern is that there would be codification to DISCOURAGE data 
level processing and that quite frankly is unrealistic.  So we embrace 
it and provide insight for implementator.

Of course, a constant concern would be backward compatibility to 
address legacy Operations.  We are all for that.

However, any change proposal only applies to active systems that are 
in position to adapt.  So any new adjusted "guidelines" should be 
proactive with new ideas to better address both new and old operations.

I can only see adding notes regarding how unrealistic a 10 minute 
timeout is in practice for LEGACY systems.

New systems will mostly like adapt and adjust and add a 10 min timeout 
as currently recommended by 5321.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 11 Aug 2010 10:30:17 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Dave CROCKER <[email-masked]>
CC: Tony Finch <[email-masked]>, Arnt Gulbrandsen <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Dave CROCKER wrote:
> 
> On 8/11/2010 5:41 AM, Tony Finch wrote:
>> Again you are talking about outgoing email. Dave's question is about 
>> what you
>> do with incoming email: How much do you do before replying to 
>> <CRLF>.<CRLF>?
> 
> 
> Arnt is correct that one of the actions that might be taken before 
> responding to
> the end of DATA is a delivery, possibly thereby involving an SMTP Sender
> activity.
> 
> But as Tony notes, the specific nature of the action after <CRLF>.<CRLF>
> wasn't my question.  My question was solely about SMTP Receiver delay in 
> making the response... for whatever reason -- and note that spam 
> filtering was actually the trigger condition for the question, not 
> message relaying.  But again, the details weren't part of the question.

+1. If we are looking for insights for "adjusted" recommendations, 
this would be a good time as there is no doubt the direction and 
growth of DATA level processing is permanent now.  The hardware allows 
for these new framework feasible in a scaled up manner and there is no 
way we can tell people "don't do that" and go back to old school 
methods of having post-SMTP (post acceptance) only data processing.

Probably a BCP or a new I-D for SMTP DATA Processing Filtering 
Guidelines would be very welcome and useful.

I would not mind drafting it if people think it is a good idea.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 11 Aug 2010 10:06:42 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Tony Finch wrote:
> On Wed, 11 Aug 2010, Arnt Gulbrandsen wrote:

>> My rendering of the same: Back then, sensible admins set up 
>> outbound queues everywhere, but had most hosts punt to a 
>> special relay, so that long-queued mail is kept in one place. 
>> Good practice now too. I wouldn't call that a  submission server, 
>> but what is the proper name for it?
> 
> Again you are talking about outgoing email. Dave's question is about 
> what you do with incoming email: How much do you do before replying 
> to <CRLF>.<CRLF>?
> 
> A "smart host" is a general purpose outgoing SMTP relay for use 
> by stub or queueless SMTP clients.

+1, i.e, like a list server.

Right, if we want to gain some level of insight for some "adjusted" 
guidelines with these increasing DATA processing SMTP frameworks, 
product field operations experiences has shown the SMTP recommended 10 
minutes is not the common timeout - its more like 3-5 minutes as a 
pretty safe value for worst case DATA delays.

A Global Timeout is a suggested design requirement to make sure all 
DATA scripts finished within the expected SMTP guideline. For us, the 
default value was set around 5 mins.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 11 Aug 2010 06:52:56 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: SMTP Interest Group <[email-masked]>
Subject: Changing RFC 5322 guidance about crlf.crlf response delay
Sender: [email-masked]

Folks,

Responses to my question make it pretty clear (to me, at least) that the current 
text in 5322 has become off the mark, regarding best current practices for 
crlf.crlf response delay.

At the time the guidance was developed, the philosophy was to reduce delay to 
the minimum and, therefore, defer any post crlf.crlf processing until after the 
SMTP session.  I'm hearing a reasonably strong consensus that it is now 
acceptable -- and possibly essential -- to do some serious processing before 
responding.

We should consider creating a Errata to cover this, but it is not yet obvious to 
me exactly what the text should be.

I assume that no one is suggesting changing the 10-minute timeout.  However, 
we've heard that some Senders don't honor it.  We should consider whether to 
ignore these non-conformists versus change the time out.  If the latter, what is 
a more appropriate timeout?

Whatever the timeout, the consensus appears to be that it is OK to have "some" 
delay in responding.

Since there will still be a timeout defining an upper limit and there will still 
be highly variable network delays meaning that the processing can't be allowed 
to come close to the timeout, it cannot be acceptable to give guidance that 
essentially says "take as long as you like before responding to the crlf.crlf." 
  So, what guidance should be given?

The substantive issue is the text:

> "To avoid receiving duplicate messages as the result of timeouts, a
>    receiver-SMTP MUST seek to minimize the time required to respond to
>    the final <CRLF>.<CRLF> end of data indicator. "

Does anyone suggest considering modification to any other text in the document?

Does anyone have useful text to suggest for adding or changing?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 11 Aug 2010 06:42:14 -0700
From: Dave CROCKER <[email-masked]>
Organization: Brandenburg InternetWorking
To: Tony Finch <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/11/2010 5:41 AM, Tony Finch wrote:
> Again you are talking about outgoing email. Dave's question is about what you
> do with incoming email: How much do you do before replying to <CRLF>.<CRLF>?

Arnt is correct that one of the actions that might be taken before responding to
the end of DATA is a delivery, possibly thereby involving an SMTP Sender
activity.

But as Tony notes, the specific nature of the action after <CRLF>.<CRLF>
wasn't my question.  My question was solely about SMTP Receiver delay in making 
the response... for whatever reason -- and note that spam filtering was actually 
the trigger condition for the question, not message relaying.  But again, the 
details weren't part of the question.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 11 Aug 2010 14:47:58 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: Tony Finch <[email-masked]>
Cc: John C Klensin <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 08/11/2010 02:41 PM, Tony Finch wrote:
> Again you are talking about outgoing email. Dave's question is about what
> you do with incoming email: How much do you do before replying to
> <CRLF>.<CRLF>?
>
> A "smart host" is a general purpose outgoing SMTP relay for use by stub or
> queueless SMTP clients.

Some SMTP servers (even some smarthosts) will act as an SMTP client 
before replying to the dot. That was true twenty years ago (when random 
unix boxes acted as a general spool and MUAs could run e.g. sendmail -bs 
or smail -iforget to talk to an SMTP server), and it's true now. IMO it 
makes sense too, although not always. It has problems, but so do the 
alternatives.

Arnt

Date: Wed, 11 Aug 2010 13:41:50 +0100
From: Tony Finch <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: John C Klensin <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On Wed, 11 Aug 2010, Arnt Gulbrandsen wrote:
> On 08/11/2010 12:48 PM, Tony Finch wrote:
> > Dave and I are talking about server side processing. You seem to be
> > talking about client retry logic.
>
> The problems never change, only the terminology ;)
>
> My rendering of the same: Back then, sensible admins set up outbound queues
> everywhere, but had most hosts punt to a special relay, so that long-queued
> mail is kept in one place. Good practice now too. I wouldn't call that a
> submission server, but what is the proper name for it?

Again you are talking about outgoing email. Dave's question is about what
you do with incoming email: How much do you do before replying to
<CRLF>.<CRLF>?

A "smart host" is a general purpose outgoing SMTP relay for use by stub or
queueless SMTP clients.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
DOGGER: VARIABLE 3 OR 4, BECOMING CYCLONIC 5 OR 6. SLIGHT OR MODERATE. RAIN OR
THUNDERY SHOWERS. MODERATE OR GOOD.

Date: Wed, 11 Aug 2010 07:45:12 -0400
From: John C Klensin <[email-masked]>
To: Tony Finch <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

--On Wednesday, August 11, 2010 11:48 +0100 Tony Finch
<[email-masked]> wrote:

> 
> On Tue, 10 Aug 2010, John C Klensin wrote:
>> --On Tuesday, August 10, 2010 09:48 -0700 Dave CROCKER
>> <[email-masked]> wrote:
>> 
>> > As I recall, delivermail didn't do queuing.  So, yes, the
>> > delivery attempt was immediate.
>> > 
>> > Sendmail behaved the same way, for a first attempt, and did
>> > queuing only if that   attempt failed.
>> > 
>> > MMDF always did queuing first, spawning a delivery attempt
>> > afterward. (Submission and delivery were independent Unix
>> > processes.)
>> 
>> And IIR, several of the small computer-based things, including
>> PC/TCP, would make a direct attempt to deliver and send mail
>> to what we would now call a submission server only if that
>> single direct delivery attempt failed.
> 
> Dave and I are talking about server side processing. You seem
> to be talking about client retry logic.

Well...  The reason why 821 talked about SMTP-Sender and
SMTP-Receiver (and variations) rather than "client" and "server"
was precisely tied up with that bit of confusion.  I was a
little reluctant to change it with 2821, but that seemed to be
the Right Thing to do (as well as WG consensus).  If one is
going to do queuing in the ways that Dave discusses, then one is
acting as an SMTP-Sender.  Before we tried to more clearly
delineate the notion of "submission server", a client (even what
we traditionally think of as an originating MUA) that used SMTP
to get to whatever was next was acting as an SMTP-Sender.  And a
server that was queuing mail for a potential next hop was an
SMTP-Sender too, regardless of whatever else it might be doing.

Things would obviously be different if a system or process had
already accepted mail for final delivery and was then
considering queuing it for delivery to a mail store.  But,
afaik, none of the systems Dave discussed did that, at least
until much later.

Arnt's comment that only the terminology changes is probably
relevant here.

      john

Date: Wed, 11 Aug 2010 13:28:29 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: Tony Finch <[email-masked]>
Cc: John C Klensin <[email-masked]>, [email-masked], [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 08/11/2010 12:48 PM, Tony Finch wrote:
> Dave and I are talking about server side processing. You seem to be
> talking about client retry logic.

The problems never change, only the terminology ;)

My rendering of the same: Back then, sensible admins set up outbound 
queues everywhere, but had most hosts punt to a special relay, so that 
long-queued mail is kept in one place. Good practice now too. I wouldn't 
call that a submission server, but what is the proper name for it?

Arnt

Date: Wed, 11 Aug 2010 13:19:35 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: David MacQuigg <[email-masked]>
Cc: Nathaniel Borenstein <[email-masked]>, [email-masked], SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 08/10/2010 06:20 PM, David MacQuigg wrote:
> Agree with this, but I still don't run my spam filters before the final
> ACCEPT. As long as there is a possibility that the filter will take too
> long, it isn't worth the confusion resulting from a timeout.

Someone I know had occasional problems of this sort, and instroduced a 
ten-second limit, which solved the problems. If a message still seems 
good after ten seconds of inspection, it's good enough to pass along.

Arnt

Date: Wed, 11 Aug 2010 11:48:24 +0100
From: Tony Finch <[email-masked]>
To: John C Klensin <[email-masked]>
cc: [email-masked], [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On Tue, 10 Aug 2010, John C Klensin wrote:
> --On Tuesday, August 10, 2010 09:48 -0700 Dave CROCKER
> <[email-masked]> wrote:
>
> > As I recall, delivermail didn't do queuing.  So, yes, the
> > delivery attempt was immediate.
> >
> > Sendmail behaved the same way, for a first attempt, and did
> > queuing only if that   attempt failed.
> >
> > MMDF always did queuing first, spawning a delivery attempt
> > afterward. (Submission and delivery were independent Unix
> > processes.)
>
> And IIR, several of the small computer-based things, including
> PC/TCP, would make a direct attempt to deliver and send mail to
> what we would now call a submission server only if that single
> direct delivery attempt failed.

Dave and I are talking about server side processing. You seem to be
talking about client retry logic.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
IRISH SEA: WESTERLY OR NORTHWESTERLY, 4 OR 5 INCREASING 5 OR 6, OCCASIONALLY 7
LATER IN NORTH. SLIGHT OR MODERATE. SHOWERS. GOOD.

Date: Tue, 10 Aug 2010 19:30:41 -0700
From: David MacQuigg <[email-masked]>
To: John Levine <[email-masked]>
CC: [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

John Levine wrote:
>> We really don't care about the proper SMTP response if it is spam.
>>     
>
> I suppose that's the case if you're sure that deferring the processing
> will never produce a bounce message.
>   

The deferred processing is only the statistical spam filter.  IP 
blacklist and HELO whitelist are very fast, and don't need deferral.  
Only the small fraction that passes the blacklist and fails the 
whitelist gets deferred processing, and the result of that processing is 
a spam score, never a REJECT.  The score affects only the rank of the 
message in the recipient's quarantine.  Quarantine counts as delivery, 
even if the recipient never looks at it.

The only time I ever send a bounce to a questionable return address is 
if there is something wrong on our end, and we can't deliver a message 
that has already been accepted (user mailbox suddenly off line, etc.).

The need for a bounce is rare enough that an admin can look at it and 
decide if it should be sent.

-- 
************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

Date: Tue, 10 Aug 2010 20:41:12 -0400
From: John C Klensin <[email-masked]>
To: [email-masked], Tony Finch <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

--On Tuesday, August 10, 2010 09:48 -0700 Dave CROCKER
<[email-masked]> wrote:

> As I recall, delivermail didn't do queuing.  So, yes, the
> delivery attempt was immediate.
> 
> Sendmail behaved the same way, for a first attempt, and did
> queuing only if that   attempt failed.
> 
> MMDF always did queuing first, spawning a delivery attempt
> afterward. (Submission and delivery were independent Unix
> processes.)

And IIR, several of the small computer-based things, including
PC/TCP, would make a direct attempt to deliver and send mail to
what we would now call a submission server only if that single
direct delivery attempt failed.

   john

Date: 10 Aug 2010 22:50:33 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: Processing after the end of DATA
Organization: 
Cc: [email-masked]
Sender: [email-masked]

> We really don't care about the proper SMTP response if it is spam.

I suppose that's the case if you're sure that deferring the processing
will never produce a bounce message.

R's,
John

Date: Tue, 10 Aug 2010 15:59:41 -0400
To: SMTP Interest Group <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

At 07:27 PM -0700 on 08/09/2010, [email-masked] wrote 
about Re: Processing after the end of DATA:

>One final point. This is a situation where it's useful to distinguish between
>submission and relay. In the case of submission, there's usually a person
>sitting there waiting for the message to be sent, so a shorter 
>timeout not only
>makes sense, you're going to have very little luck convincing client 
>authors to
>use a longer one. Relays, OTOH, tend to be much more tolerant of delayed
>responses. There are of course all sorts of ways to make submit operations
>responsive than relay operations.

There is a simple way of distinguishing relay from submit. Run your 
relay on Port 25 using the name from your MX record. Run submit using 
a DIFFERENT FQDN (and IPN) on Port 587 (and if you want to allow 
Luddite MUAs that will not allow the user to request Port 587, allow 
Port 25 connections). So long as it comes in on Port587 it is a 
submission. NO Relay will come into the MSA via Port25 since relay 
will get the correct server via the MX record given the different 
host IPN. The only false positive for relay would be if the MUA is 
misconfigured to connect to the MTA as opposed to MSA server via 
Port25.

From: "J.D. Falk" <[email-masked]>
Subject: Re: Processing after the end of DATA
Date: Tue, 10 Aug 2010 12:50:01 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 9, 2010, at 5:30 PM, Dave CROCKER wrote:

> On 8/9/2010 5:25 PM, Steve Atkins wrote:
>>> Can folks offer pointers to normative language?
>> 
>> Tail end of section 6.1 of 5321:
>> 
>> "To avoid receiving duplicate messages as the result of timeouts, a
>>    receiver-SMTP MUST seek to minimize the time required to respond to
>>    the final<CRLF>.<CRLF>  end of data indicator. "
>> 
>> However the same section also says:
> 
> Thanks!
> 
> And, yeah, it might be worth reviewing both normative directives.

+1

Date: Tue, 10 Aug 2010 11:18:10 -0700
From: David MacQuigg <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Carl S. Gutekunst wrote:
>
>>> Rejecting anything that you can at the SMTP level, rather than 
>>> generating a bounce, strikes me as a sufficiently Good Thing that it 
>>> outweighs any remaining benefits from the "MUST minimize" rule.  
>>> Minimizing bounces seems more valuable than minimizing SMTP 
>>> connection time, primarily because the bounces are more likely to 
>>> waste an actual human being's time at some later point.  -- Nathaniel
>>>   
>>
>> Agree with this, but I still don't run my spam filters before the 
>> final ACCEPT.  As long as there is a possibility that the filter will 
>> take too long, it isn't worth the confusion resulting from a timeout.
>
> Almost all of the commercial enterprise-grade filters I know do *all* 
> of their filtering before accept: spam, virus, content policies, etc. 
> That's because of the desire to emit a clear rejection, and 
> backscatter is a far more serious problem than duplicates. The 
> exceptions (those that do filtering after the 250 OK) don't generate 
> DSNs; they deliver to a spam folder or other quarantine.
>
> <csg>

My system is too small to draw any conclusions, but my impression is 
that we are overly-concerned about a very small fraction of the mailflow 
- those few messages that are legit, but get a false reject from a 
statistical filter.  We really don't care about the proper SMTP response 
if it is spam.  The only advantage I see in running the spam filter 
before ACCEPT is a few less messages at the bottom of my spam bucket.  I 
wish I had some numbers from a larger system.

Would be nice if SMTP had a 'conditional ACCEPT':  "Sorry, your 
transmitter ID 'yahoo.com' cannot be verified,   Your message has been 
routed to the recipient's quarantine."  Imagine the effect that would 
have on all the lame excuses for an invalid or hard-to-verify HELO name.

-- 
************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

Date: Tue, 10 Aug 2010 09:48:47 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Tony Finch <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/10/2010 9:31 AM, Tony Finch wrote:
> In my experience mailers don't do delivery during the incoming SMTP
> conversation, which sounds like a significant change relative to the
> problematic mailers of the late 1980s.

As I recall, delivermail didn't do queuing.  So, yes, the delivery attempt was 
immediate.

Sendmail behaved the same way, for a first attempt, and did queuing only if that 
  attempt failed.

MMDF always did queuing first, spawning a delivery attempt afterward. 
(Submission and delivery were independent Unix processes.)

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Tue, 10 Aug 2010 09:48:30 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

>> Rejecting anything that you can at the SMTP level, rather than 
>> generating a bounce, strikes me as a sufficiently Good Thing that it 
>> outweighs any remaining benefits from the "MUST minimize" rule.  
>> Minimizing bounces seems more valuable than minimizing SMTP 
>> connection time, primarily because the bounces are more likely to 
>> waste an actual human being's time at some later point.  -- Nathaniel
>>   
>
> Agree with this, but I still don't run my spam filters before the 
> final ACCEPT.  As long as there is a possibility that the filter will 
> take too long, it isn't worth the confusion resulting from a timeout.

Almost all of the commercial enterprise-grade filters I know do *all* of 
their filtering before accept: spam, virus, content policies, etc. 
That's because of the desire to emit a clear rejection, and backscatter 
is a far more serious problem than duplicates. The exceptions (those 
that do filtering after the 250 OK) don't generate DSNs; they deliver to 
a spam folder or other quarantine.

<csg>

Date: Tue, 10 Aug 2010 17:31:58 +0100
From: Tony Finch <[email-masked]>
To: John Levine <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On Tue, 10 Aug 2010, John Levine wrote:
>
> >Tail end of section 6.1 of 5321:
> >
> >"To avoid receiving duplicate messages as the result of timeouts, a
> >   receiver-SMTP MUST seek to minimize the time required to respond to
> >   the final <CRLF>.<CRLF> end of data indicator. "
>
> That language is taken verbatim from 2821.  In both RFCs the following
> sentence refers to RFC 1047.

And RFC 2821 took it from 1123 :-)

RFCs 1123 et seq. also say "When the receiver gets the final period
terminating the message data, it typically performs processing to deliver
the message to a user mailbox" and RFC 1047 mentions specific problems
related to mailing list expansion.

In my experience mailers don't do delivery during the incoming SMTP
conversation, which sounds like a significant change relative to the
problematic mailers of the late 1980s.

(It would be OK to do delivery during the incoming conversation if it
can be sufficiently decoupled that a slow delivery doesn't delay the
server's <CRLF>.<CRLF> reply longer than some upper bound and the
delivery continues while the client moves on.)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
BAILEY: NORTHWEST 4 OR 5. SLIGHT OR MODERATE. SHOWERS. GOOD.

Date: Tue, 10 Aug 2010 09:20:21 -0700
From: David MacQuigg <[email-masked]>
To: Nathaniel Borenstein <[email-masked]>
CC: [email-masked], SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Nathaniel Borenstein wrote:
> As usual I agree with Mr. EEF:
>
> On Aug 9, 2010, at 10:27 PM, [email-masked] wrote:
>
>   
>> I have a somewhat different take on this. First of all, I have always thought
>> the admonition that you MUST minimize the amount of time spent before
>> responding to the trailing dot to the greatest extent possible was, well, bunk.
>> (It's also an effectively unenforceable MUST - who can say you've done all you
>> can or not? - which is bad in its own right.) While it is important not to
>> spend too much time, the difference between a millisecond delay and a 2 second
>> delay is, in this situation, not worth worrying about.
>>     
>
> Rejecting anything that you can at the SMTP level, rather than generating a bounce, strikes me as a sufficiently Good Thing that it outweighs any remaining benefits from the "MUST minimize" rule.  Minimizing bounces seems more valuable than minimizing SMTP connection time, primarily because the bounces are more likely to waste an actual human being's time at some later point.  -- Nathaniel
>   

Agree with this, but I still don't run my spam filters before the final 
ACCEPT.  As long as there is a possibility that the filter will take too 
long, it isn't worth the confusion resulting from a timeout.

I have a small system, and my setup may be different for that reason, 
but I'm just not seeing the need for anything as laborious as 
statistical filtering during the SMTP session.   Heavily abused IPs get 
an immediate REJECT.  Of the remainder, most legitimate messages get 
whitelisted, based on the reputation of the domain name extracted from 
their HELO name.  That leaves a small slice in the middle that gets an 
ACCEPT, but then goes through the spam filter and to quarantine.  Less 
than 1 in 1000 spams in my inbox, and no false rejects.

Am I the only one using the HELO name for whitelisting?

-- 
************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

Content-Class: urn:content-classes:message
Subject: RE: Processing after the end of DATA
Date: Tue, 10 Aug 2010 11:05:21 -0400
From: "Rosenwald, Jordan" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 9, 2010, at 5:10 PM, Dave CROCKER wrote:
> I did a 10-second scan of 5321 and didn't find the relevant text, but
I'd like 
> to resolve -- and preferably squash -- this issue quickly.

Just came into the office and saw nearly 20 emails on this.  Great
conversation, but not "quickly" :)

At 19:27 09-08-10, [email-masked] wrote:
> I have a somewhat different take on this. First of all, I have always 
> thought the admonition that you MUST minimize the amount of time spent

> before responding to the trailing dot to the greatest extent possible 
> was, well, bunk.
> (It's also an effectively unenforceable MUST - who can say you've done

> all you can or not? - which is bad in its own right.) While it is 
> important not to spend too much time, the difference between a 
> millisecond delay and a 2 second delay is, in this situation, not
worth worrying about.

Completely agree for several reasons, including that the only senders
that take advantage of "as fast as you can process" scenarios are
spammers.  If I take a second or two to accept a message from a legit
sender they neither know nor care.  But to a spammer, the faster we
receive and delivery the more mail gets through before a content filter
or blocklist can update.  We saw this happen when we migrated to our
current system 3 years ago.  Spam was delivered at many hundreds of
times faster than mail from AOL, Hotmail and Gmail combined. Responding
as quickly as my system will allow with <CRLF>.<CRLF> helps no one I
want to help.

Subject: Re: Processing after the end of DATA
From: Nathaniel Borenstein <[email-masked]>
Date: Tue, 10 Aug 2010 08:04:05 -0400
Cc: "Carl S. Gutekunst" <[email-masked]>, Tony Hansen <[email-masked]>, Dave CROCKER <[email-masked]>, SMTP Interest Group <[email-masked]>
To: [email-masked]
Sender: [email-masked]

As usual I agree with Mr. EEF:

On Aug 9, 2010, at 10:27 PM, [email-masked] wrote:

> I have a somewhat different take on this. First of all, I have always thought
> the admonition that you MUST minimize the amount of time spent before
> responding to the trailing dot to the greatest extent possible was, well, bunk.
> (It's also an effectively unenforceable MUST - who can say you've done all you
> can or not? - which is bad in its own right.) While it is important not to
> spend too much time, the difference between a millisecond delay and a 2 second
> delay is, in this situation, not worth worrying about.

Rejecting anything that you can at the SMTP level, rather than generating a bounce, strikes me as a sufficiently Good Thing that it outweighs any remaining benefits from the "MUST minimize" rule.  Minimizing bounces seems more valuable than minimizing SMTP connection time, primarily because the bounces are more likely to waste an actual human being's time at some later point.  -- Nathaniel

Date: Tue, 10 Aug 2010 00:24:47 -0700
To: SMTP Interest Group <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Processing after the end of DATA
Cc: [email-masked], Tony Hansen <[email-masked]>
Sender: [email-masked]

At 19:27 09-08-10, [email-masked] wrote:
>I have a somewhat different take on this. First of all, I have always thought
>the admonition that you MUST minimize the amount of time spent before
>responding to the trailing dot to the greatest extent possible was, 
>well, bunk.
>(It's also an effectively unenforceable MUST - who can say you've done all you
>can or not? - which is bad in its own right.) While it is important not to
>spend too much time, the difference between a millisecond delay and a 2 second
>delay is, in this situation, not worth worrying about.

Filter processing can sometimes take much more that two seconds.  As 
others mentioned, it is better not to rely on the 10 minute DATA 
termination timeout.  Some receivers set a much shorter 
timeout.  Some senders may timeout before that 10 minute period.

>What *is* worth worrying about is deferring too much processing 
>until later. Do

 From an operational perspective, I agree.

>It can also lead, in a variety of ways, to generating excessive amounts of
>blowback.

Yes.

>One final point. This is a situation where it's useful to distinguish between
>submission and relay. In the case of submission, there's usually a person
>sitting there waiting for the message to be sent, so a shorter 
>timeout not only
>makes sense, you're going to have very little luck convincing client 
>authors to
>use a longer one. Relays, OTOH, tend to be much more tolerant of delayed
>responses. There are of course all sorts of ways to make submit operations
>responsive than relay operations.

The difference between submission and relay is that the user calls 
you when they cannot send the message.  Some MUAs may timeout well 
before the 10 minute period.

At 19:52 09-08-10, Tony Hansen wrote:
>A big timeout doesn't mean that one can takes lots of time, but it 
>might explain where someone got the idea that they *could* take lots of time.

Yes.

Regards,
-sm 

Date: Mon, 09 Aug 2010 22:40:27 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Actually, reading the patent revealed something I'd never known before: 
Ned's real name.

:-)

<csg>

Date: Mon, 09 Aug 2010 22:39:15 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: John Levine <[email-masked]>
CC: [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

John Levine wrote:
> For a while I used a very aggressive greylister which soft failed
> after the dot, kept a hash of the message, and wanted to see an
> exact copy with the same hash to let the message through.  I stopped
> using it, but not because of SMTP problems.  (A surprising number of
> ESPs reconstruct the message rather than retransmitting it, so the
> hash didn't match.)

Yes. And unlike bots, ESPs will do retries. (Lord knows they do retries, 
even on 550 errors.) At Habeas I had a customer that set the Date field 
at connect time and generated no Message-ID field at all; and they 
complained to /us/ when we logged duplicates!

I haven't seen that problem as much as I used to, though. As the volume 
demands on the have grown, the software developers seem more inclined to 
do more computation in advance and less at delivery time.

<csg>

Date: Mon, 09 Aug 2010 22:30:53 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Hector Santos wrote:
>> First of all, in the interests of full and proper disclosure, I 
>> should mention that this particular trick is patented: US Patent 
>> #7080123. (I'm the author, but Oracle owns the patent.)
>
> Ridiculous!
>
> Dupe checking method was an essential design requirement since the 80s.

Um, I kinda suspect Ned's well aware of the prior art. And you should 
actually read the claims in the patent.

<csg>

Date: Tue, 10 Aug 2010 01:13:13 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Hansen <[email-masked]>
CC: Dave CROCKER <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Tony Hansen wrote:
> 
> A big timeout doesn't mean that one can takes lots of time, but 
> it might explain where someone got the idea that they *could* 
> take lots of time.
> 
>     Tony Hansen

And there is nothing wrong with that, after all, we encourage people 
to explore the unknowns, the least untested, learn and share experiences.

Remember the KEEP-ALIVE discussions?

Well, there was experiences here.  Much exploration was done and you 
helped decide what I was proposing in regards to using continuation 
line with reply codes different from the final line as an idea keep 
clients alive for a world that would soon doing more DATA level 
processing.  Once we found there are legacy software which will not 
work here, we codified RFC 5321 to explicitly state all reply codes in 
continuation line MUST be the same as the final reply code.

But that did not stop DATA processing efforts.  In the end, I can 
safely say a 3-5 minute DATA processing time seems to be the maximum. 
Of course, it should be as fast as possible, but that would be an 
operator setup situation. Its may not be one script but a series of 
organized scripts.

The best SMTP can do here is add SHIM management logic with watch dog 
logic, hot port drop logic, etc.  We did add a GLOBAL TIMEOUT for all 
scripts as well to keep it within some level of SMTP expectations.

#-----------------------------------------------------
# v2.0
#
# Maximum Scanning Time (secs) allowed. This
# time is to help prevent SMTP timeouts in the DATA
# state.  10 mins is official wait time for DATA
# termination timeout. However, 5 mins is pretty
# common (unfortunately) for many SMTP clients. The
# default value is alittle less than 5 minutes
# (4 mins, 30 secs).  What this means is that
# all the WCX filters in the [hooks] sections MUST
# complete within this time.  To disable, set it
# too zero, however, keep in mind, SMTP clients will
# timeout and disconnect if the filters are taking
# too long. This can cause dupes and unnecessary
# retries by the client.
#-----------------------------------------------------

;GlobalTimeout = 270    # use this for 5 mins timeouts (default)
;GlobalTimeout = 570    # use this for 10 mins timeouts
;GlobalTimeout = 60     # TESTING ONLY

#-----------------------------------------------------
# v2.0
#
# GlobalCarrierCheck allows for the Wildcat! WCX processor
# to globally check for connection drops for all
# wcx run by the loader.  When FALSE, then each WCX
# is responsible to perform direct Carrier() or
# sfCheckCarrier() calls if it wants to gracefully
# check for drops and exit gracefully.
#
# RECOMMENDATION: KEEP THIS FALSE
#
# GlobalCarrierCheck is FALSE by default to allow each
# WCX to take control.  It is possible that even when
# a connection a drop, a WCX made want to get its hand
# on the email file to do something with it.  In these
# cases, you should have these WCX scripts in the
# [Event:Drop] section.
#-----------------------------------------------------

;GlobalCarrierCheck = false

So when you guys catch up, those are some of the things to consider in 
SMTP DATA shims. It is workable concept. We should not be afraid to do 
DATA shims by recommending we should use 1988 RFC 1074 guidelines 
molded in an era where hardware, OS and software data processing 
efficiency was very low compared to today. In the past, it was not 
very efficient to do SQL database queries or move large files around 
networks during a DATA state.  This is pretty much a non-issue today.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Tue, 10 Aug 2010 00:02:15 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

[email-masked] wrote:
> <csg> writes:
> 
>> As a result, duplicates were such a staggering huge problem. To mitigate
>> that, Postini computed a SHA-1 hash for every message, and if there was
>> any evidence that the receiver had sent the 250 OK, but the sender never
>> got it, the hash was saved in a database. If the sender resent the
>> message, the proxy (knowing it had already been delivered) quietly ate 
>> it.
> 
>> When that database broke, the customers noticed immediately. The effect
>> was that significant.
> 
> First of all, in the interests of full and proper disclosure, I should 
> mention that this particular trick is patented: US Patent #7080123. 
> (I'm the author, but Oracle owns the patent.)

Ridiculous!

Dupe checking method was an essential design requirement since the 80s.

Every Fidonet Mailer since the 80s and basically all online hosting 
systems, including Wildcat! all had dupe checking since the 80s.

This is a yet another case of a frivolous patent based on the new 1996 
Time Line where the existing methods and technology pre 1996 could not 
be patent was now open to patentability.  I seen it all the time and 
of course, it doesn't apply to companies and organizations that have 
used these simple methods as normal engineering requirement - Markus 
RULES!

Dupe checking was an essential SOFTWARE requirement in network 
topologies where mail dupes was an extremely high occurrence.  It 
wasn't an exception but an every minute expectation.

Fidonet use the network control lines called SEEN-BY lines which is 
technologically the same as used today by he NNTP PATH line with bang 
domain addresses.  The hidden SEEN-BY lines recorded every node 
touching the message as it passed thru the network.  When the growth 
of SEEN-BY lines got too large, Tiny SEEN-BY lines were introduced to 
reduced the syntax recording by COUNTRY, REGIONS or ZONES, NETs with 
ZONES and NODES within NETS.

With the advent of 822, the Message-ID was used at the LOCAL HOST 
level, but that was not alway reliable so a local HASH was recorded 
and there was never any retransmitted DUPES allowed.  The DUPE system 
was generic at the back end so it didn't matter what network the mail 
come from: Fidonet, UUCICO, SMTP, QWK, OPX, UTI, etc including as 
posted by a local user online!

A ridiculous patent - no patent can deny was is technologically 
fundamental and require for normal operations. DUPE CHECKING was a 
basic ingredient in ALL mail network software since the annals of 
time. I can not think of one system worth its salt that did not have a 
dupe checking concept. What? Are we suppose to stop using was so 
fundamental in all our software since the 80s?

Geez!

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: [email-masked]
Cc: "Carl S. Gutekunst" <[email-masked]>, Tony Hansen <[email-masked]>, Dave CROCKER <[email-masked]>, SMTP Interest Group <[email-masked]>
Date: Mon, 09 Aug 2010 20:37:23 -0700 (PDT)
Subject: Re: Processing after the end of DATA
To: [email-masked]
Sender: [email-masked]

P.S. There's also a funny anecdote associated with this particular patent -
when a patent issued at Sun you used to get this cute little commemorative
plaque. In this case, however, I one one plaque, and then a duplicate plaque
arrived about a month later.

				Ned

Date: Mon, 09 Aug 2010 23:25:22 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

John Levine wrote:

> I believe this was a problem on CSNET when he wrote the RFC in 1988.
> It is my impression that MTAs have gotten someone more robust in the
> ensuing two decades.  Is it still a problem now?

 From our experience and with customer support, generally only when 
scripts (and it can be more than one) all together take 3-5 minutes.

> 
> For a while I used a very aggressive greylister which soft failed
> after the dot, kept a hash of the message, and wanted to see an
> exact copy with the same hash to let the message through.  I stopped
> using it, but not because of SMTP problems.  (A surprising number of
> ESPs reconstruct the message rather than retransmitting it, so the
> hash didn't match.)
> 
> Anyone got actual experience with delays at the dot?  It wouldn't be
> hard for me to stick a module into my SMTP daemon that inserts extra
> delays and see what breaks.

Since 2003, we had a flexible DATA scripting system designed 
specifically for operators or 3rd party developed local policy p-code 
scripts.  They use an INI file to define the run order of the scripts. 
  SMTP will pass every envelope data, connection data and the file 
name of the temporary spooled file received, and the script manager 
will run each script serially.

Each script can end with:

     PASS      - continue with next script
     FAIL      - stop scripts return 45x or 55x (DATA reject)
     FAIL_KEEP - same as FAIL, copy of file saved.

We provide two key stock scripts:

   SMTPFILTER-CHECKWORDS - list of Spam words to filter message.
   SMTPFILTER-WCGREY     - performs grey listing

Both are very popular and people share there list of words file. My 
personal spamword.txt has grow to at least 5000 words.  It began as a 
very simple minded search word rejection script.  You know grey listing.

What was learned and required code changes to address the issues.

The basic issue was the increase processing time when time delays as 
high as  3 or more minutes were possible, especially with the original 
simple search words program with no smarts.

So the first thing to speed the search word scripts and it was much 
supportive of MIME messages.  What take 3-5 minutes was reduced to 
seconds.

But we also had to add socket disconnect detection to abort the 
scripts processing.  We also added logic to check the read buffer for 
the QUIT command which is used to finally accept the message.

But aborting scripts also created the potential for a loop hole where 
a client will intentional abort a delayed DATA with the idea that its 
payload will be accepted.

So now the script system has EVENTS for disconnects and timeouts that 
will always process "required" scripts.  So the operator has to design 
what order of scripts to run and which once should always regardless 
of drops and timeouts.

Finally, what I have seen is:

  - Client can drop, or

  - Clients issue QUIT, may wait and drop.

In both cases, I'm seen retransmissions but not always.

Overall, reducing the DATA processing tine is the key and keeping it 
shorter than 3 minutes (which is hardly seen now) is usually ok 
regarding client timeouts.

On a related note, in the past, we discussed KEEP-ALIVE concepts using 
continuation reply line every X time to keep the client alive. This 
was explored and proven to work with many modern clients.  However, 
there are legacy clients that do not use the last "XXXspace" line, 
they use the first line only and the RFC 5321 specs was codified to 
suggest all continuation lines use the same reply code.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Mon, 09 Aug 2010 22:52:09 -0400
From: Tony Hansen <[email-masked]>
To: Dave CROCKER <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

A big timeout doesn't mean that one can takes lots of time, but it might 
explain where someone got the idea that they *could* take lots of time.

     Tony Hansen

On 8/9/2010 8:40 PM, Dave CROCKER wrote:
>
>
>
> On 8/9/2010 5:36 PM, Tony Hansen wrote:
>> On the flip side is this:
>>
>> 4.5.3.2.6. DATA Termination: 10 Minutes.
>
> I can understand thinking that this is a 'flip' side, but as I recall, 
> there was different intent that drove the two-three rules.
>
> The core problem with doing any serious processing after crlf.crlf is 
> that it can take an unbounded amount of time, plus Internet delays can 
> get interesting.  All of which can run things up to the timeout when 
> that's not intended.
>
> A big timeout is not meant to mean that one can take lots of time...
>
> d/
>

From: [email-masked]
Cc: Tony Hansen <[email-masked]>, Dave CROCKER <[email-masked]>, SMTP Interest Group <[email-masked]>
Date: Mon, 09 Aug 2010 19:27:22 -0700 (PDT)
Subject: Re: Processing after the end of DATA
To: "Carl S. Gutekunst" <[email-masked]>
Sender: [email-masked]

I have a somewhat different take on this. First of all, I have always thought
the admonition that you MUST minimize the amount of time spent before
responding to the trailing dot to the greatest extent possible was, well, bunk.
(It's also an effectively unenforceable MUST - who can say you've done all you
can or not? - which is bad in its own right.) While it is important not to
spend too much time, the difference between a millisecond delay and a 2 second
delay is, in this situation, not worth worrying about.

What *is* worth worrying about is deferring too much processing until later. Do
too much of that and you can end up accepting messages faster than you can
process them. This may not be a big deal at low transaction rates, but when
you're operating in the many hundreds of messages a second per MTA regime, it's
can be a real killer.

It can also lead, in a variety of ways, to generating excessive amounts of
blowback.

Accordingly, my mantra has always been to do as much processing as possible
during the SMTP dialogue (not just after the trailing dot either - spread out
is better), but to avoid anything that could result in excessive (as in more
than a second or so) delays.

This approach has worked well for us. The timeout issues we've had have almost
always been the result of externalities we cannot control. For example, even if
our MTA used no resources at all, we cannot make up for someone who likes to
put the MTA's queues on an NFS mount connected over a network what appears to
be constructed of tin cans and string.

In more recent times we've mostly traded hardware performance issues for
spam/virus checker performance issues. Mind you, there are AS/AV solutions
available that are amply performant, but not everyone uses them.

<csg> writes:

> As a result, duplicates were such a staggering huge problem. To mitigate
> that, Postini computed a SHA-1 hash for every message, and if there was
> any evidence that the receiver had sent the 250 OK, but the sender never
> got it, the hash was saved in a database. If the sender resent the
> message, the proxy (knowing it had already been delivered) quietly ate it.

> When that database broke, the customers noticed immediately. The effect
> was that significant.

First of all, in the interests of full and proper disclosure, I should mention
that this particular trick is patented: US Patent #7080123. (I'm the author,
but Oracle owns the patent.)

There was a time when this trick was clearly of significant benefit at some
sites, but I honestly don't know to what extent it benefits us now. (As I type
this the database of these entries on my home system contains 979 entries, but
almost all of them are likely associated with spammers that blast and
disconnect without waiting for the dot response. Of course they're just hashes
so I have no way of knowing which ones are legit and which ones aren't.)

One final point. This is a situation where it's useful to distinguish between
submission and relay. In the case of submission, there's usually a person
sitting there waiting for the message to be sent, so a shorter timeout not only
makes sense, you're going to have very little luck convincing client authors to
use a longer one. Relays, OTOH, tend to be much more tolerant of delayed
responses. There are of course all sorts of ways to make submit operations
responsive than relay operations.

				Ned

Date: Mon, 09 Aug 2010 22:53:25 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Carl S. Gutekunst wrote:
> 
> Tony Hansen wrote:
>>
>> On the flip side is this:
>>
>> 4.5.3.2.6.  DATA Termination: 10 Minutes.
>>
>>    This is while awaiting the "250 OK" reply.
> 
> There's a world of difference between the obligation of the server to 
> emit a timely response, and the obligation of the client to not give up 
> early.
> 
> Dave, unfortunately I'm not good at the normative references, but I am 
> long on experience here. My observation at Postini was that most clients 
> give up after a much shorter time. 5 minutes was common, but some were 
> as little as 30 seconds. 

We seen an average of 3-5 minutes for the problematic reports issues.

> As a result, duplicates were such a staggering huge problem. To mitigate 
> that, Postini computed a SHA-1 hash for every message, and if there was 
> any evidence that the receiver had sent the 250 OK, but the sender never 
> got it, the hash was saved in a database. If the sender resent the 
> message, the proxy (knowing it had already been delivered) quietly ate it.

+1.

We had to do two things:

    - Socket drop detection to about the DATA scripts
    - Read the socket buffer for any QUIT command.

So if there evidence of the QUIT in biffer, the accepted payload was 
queued for delivery.  We always had a dupe checker so retransmits were 
never an issue.

But this lead to an issue regarding a 821/2821/5321 feature we enabled 
  which I call the NO-QUIT-CANCEL SMTP requirement.

Once the 250 was determined as the response for the sender, the 
message was immediately queued for delivery.  The receiver did not 
require the QUIT to cancel the message which was probably already routed.

So in view, SMTP has a silly QUIT requirement which was modeled after 
single-threaded process mentality and POST SMTP only processing.

So while the 1988 RFC 1074 touches based with the synchronization gap 
between <crlf>.<crlf> and the receiver reply code, SMTP also has a 
time gap between the receiver 250 and the sender QUIT command.

Because of the unavoidable use and growth of MTA adding DATA 
processing scripts, many of my sysops turn off the NO-QUIT-CANCEL 
requirement.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Mon, 09 Aug 2010 18:36:15 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John Levine <[email-masked]>
CC: [email-masked]
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/9/2010 6:18 PM, John Levine wrote:
> I believe this was a problem on CSNET when he wrote the RFC in 1988.

No, John, it was not just CSNet, nor did CSNet over the SMTP net display 
differential behavior.

> It is my impression that MTAs have gotten someone more robust in the
> ensuing two decades.  Is it still a problem now?

cf, Carl's note.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: 10 Aug 2010 01:18:02 -0000
From: John Levine <[email-masked]>
To: [email-masked]
Subject: Re: Processing after the end of DATA
Organization: 
Cc: [email-masked]
Sender: [email-masked]

>Tail end of section 6.1 of 5321:
>
>"To avoid receiving duplicate messages as the result of timeouts, a
>   receiver-SMTP MUST seek to minimize the time required to respond to
>   the final <CRLF>.<CRLF> end of data indicator. "

That language is taken verbatim from 2821.  In both RFCs the following
sentence refers to RFC 1047.

RFC 1047 describes a problem Craig Partridge observed when an SMTP
server did lots of work after the dot, and the client gave up and went
away in the meantime.  The client assumed the delivery failed and it
had to resend later, but the server eventually said OK, so the message
got delivered twice.

I believe this was a problem on CSNET when he wrote the RFC in 1988.
It is my impression that MTAs have gotten someone more robust in the
ensuing two decades.  Is it still a problem now?

For a while I used a very aggressive greylister which soft failed
after the dot, kept a hash of the message, and wanted to see an
exact copy with the same hash to let the message through.  I stopped
using it, but not because of SMTP problems.  (A surprising number of
ESPs reconstruct the message rather than retransmitting it, so the
hash didn't match.)

Anyone got actual experience with delays at the dot?  It wouldn't be
hard for me to stick a module into my SMTP daemon that inserts extra
delays and see what breaks.

R's,
John

Date: Mon, 09 Aug 2010 21:11:59 -0400
From: John C Klensin <[email-masked]>
To: Dave CROCKER <[email-masked]>, Tony Hansen <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

--On Monday, August 09, 2010 17:40 -0700 Dave CROCKER
<[email-masked]> wrote:

> 
> 
> 
> On 8/9/2010 5:36 PM, Tony Hansen wrote:
>> On the flip side is this:
>> 
>> 4.5.3.2.6. DATA Termination: 10 Minutes.
> 
> I can understand thinking that this is a 'flip' side, but as I
> recall, there was different intent that drove the two-three
> rules.
> 
> The core problem with doing any serious processing after
> crlf.crlf is that it can take an unbounded amount of time,
> plus Internet delays can get interesting.   All of which can
> run things up to the timeout when that's not intended.
> 
> A big timeout is not meant to mean that one can take lots of
> time...

That is certainly how I've always understood it.

   john

Date: Mon, 09 Aug 2010 17:55:01 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Tony Hansen <[email-masked]>
CC: Dave CROCKER <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

Tony Hansen wrote:
>
> On the flip side is this:
>
> 4.5.3.2.6.  DATA Termination: 10 Minutes.
>
>    This is while awaiting the "250 OK" reply.

There's a world of difference between the obligation of the server to 
emit a timely response, and the obligation of the client to not give up 
early.

Dave, unfortunately I'm not good at the normative references, but I am 
long on experience here. My observation at Postini was that most clients 
give up after a much shorter time. 5 minutes was common, but some were 
as little as 30 seconds. Postini uses a proxy, so the amount of time 
after the dot was the sum of all latencies, including the receiver and 
all of the network delays from the sender to the proxy, and from the 
proxy to the receiver.

As a result, duplicates were such a staggering huge problem. To mitigate 
that, Postini computed a SHA-1 hash for every message, and if there was 
any evidence that the receiver had sent the 250 OK, but the sender never 
got it, the hash was saved in a database. If the sender resent the 
message, the proxy (knowing it had already been delivered) quietly ate it.

When that database broke, the customers noticed immediately. The effect 
was that significant.

<csg>

Date: Mon, 09 Aug 2010 17:40:05 -0700
From: Dave CROCKER <[email-masked]>
Organization: Brandenburg InternetWorking
To: Tony Hansen <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/9/2010 5:36 PM, Tony Hansen wrote:
> On the flip side is this:
>
> 4.5.3.2.6. DATA Termination: 10 Minutes.

I can understand thinking that this is a 'flip' side, but as I recall, there was 
different intent that drove the two-three rules.

The core problem with doing any serious processing after crlf.crlf is that it 
can take an unbounded amount of time, plus Internet delays can get interesting. 
  All of which can run things up to the timeout when that's not intended.

A big timeout is not meant to mean that one can take lots of time...

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 09 Aug 2010 20:36:49 -0400
From: Tony Hansen <[email-masked]>
To: Dave CROCKER <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On the flip side is this:

4.5.3.2.6.  DATA Termination: 10 Minutes.

    This is while awaiting the "250 OK" reply.  When the receiver gets
    the final period terminating the message data, it typically performs
    processing to deliver the message to a user mailbox.  A spurious
    timeout at this point would be very wasteful and would typically
    result in delivery of multiple copies of the message, since it has
    been successfully sent and the server has accepted responsibility for
    delivery.  See Section 6.1 for additional discussion.

     Tony Hansen
     [email-masked]

On 8/9/2010 8:30 PM, Dave CROCKER wrote:
>
>
>
> On 8/9/2010 5:25 PM, Steve Atkins wrote:
>>> Can folks offer pointers to normative language?
>>
>> Tail end of section 6.1 of 5321:
>>
>> "To avoid receiving duplicate messages as the result of timeouts, a
>>     receiver-SMTP MUST seek to minimize the time required to respond to
>>     the final<CRLF>.<CRLF>  end of data indicator. "
>>
>> However the same section also says:
>
>
> Thanks!
>
> And, yeah, it might be worth reviewing both normative directives.
>
> d/
>

Date: Mon, 09 Aug 2010 17:30:18 -0700
From: Dave CROCKER <[email-masked]>
Organization: Brandenburg InternetWorking
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Processing after the end of DATA
Sender: [email-masked]

On 8/9/2010 5:25 PM, Steve Atkins wrote:
>> Can folks offer pointers to normative language?
>
> Tail end of section 6.1 of 5321:
>
> "To avoid receiving duplicate messages as the result of timeouts, a
>     receiver-SMTP MUST seek to minimize the time required to respond to
>     the final<CRLF>.<CRLF>  end of data indicator. "
>
> However the same section also says:

Thanks!

And, yeah, it might be worth reviewing both normative directives.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Subject: Re: Processing after the end of DATA
From: Steve Atkins <[email-masked]>
Date: Mon, 9 Aug 2010 17:25:37 -0700
Cc: Dave CROCKER <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Aug 9, 2010, at 5:10 PM, Dave CROCKER wrote:

> 
> G'day.
> 
> It's Monday.  I'm feeling aggressive...
> 
> Once upon a time, it was deemed important to respond after crlf.crlf as quick as possible.  Pausing for additional processing of varying complexity and delay was deemed unacceptable.
> 
> There seems to be an emerging constituency that thinks this stricture is not applicable to modern email services.
> 
> I did a 10-second scan of 5321 and didn't find the relevant text, but I'd like to resolve -- and preferably squash -- this issue quickly.
> 
> Can folks offer pointers to normative language?

Tail end of section 6.1 of 5321:

"To avoid receiving duplicate messages as the result of timeouts, a
   receiver-SMTP MUST seek to minimize the time required to respond to
   the final <CRLF>.<CRLF> end of data indicator. "

However the same section also says:

  "if there is a delivery failure after acceptance of a message, the
   receiver-SMTP MUST formulate and mail a notification message"

If you're doing any useful level of spam / virus filtering then your
only option which will comply with both of those requirements
will be to send a notification to the reverse-path - which in the case
of the vast majority of spam and virus-laden email, is an unrelated
third party.

Cheers,
  Steve

Date: Mon, 09 Aug 2010 17:10:09 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: SMTP Interest Group <[email-masked]>
Subject: Processing after the end of DATA
Sender: [email-masked]

G'day.

It's Monday.  I'm feeling aggressive...

Once upon a time, it was deemed important to respond after crlf.crlf as quick as 
possible.  Pausing for additional processing of varying complexity and delay was 
deemed unacceptable.

There seems to be an emerging constituency that thinks this stricture is not 
applicable to modern email services.

I did a 10-second scan of 5321 and didn't find the relevant text, but I'd like 
to resolve -- and preferably squash -- this issue quickly.

Can folks offer pointers to normative language?

Thanks.

d/

ps.  Yes, there is a differential to my aggressiveness.
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net