Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
From: Дилян Палаузов <[email-masked]>
To: Alessandro Vesely <[email-masked]>
Cc: IETF SMTP <[email-masked]>
Date: Mon, 30 Nov 2009 14:13:38 +0100
List-Id: <ietf-smtp.imc.org>

Hello,

> That matches my experience. However, rejection does not offer many 
> workarounds, because re-sending will likely repeat the failure, while 
> re-writing is something that apparently even spammers don't want to do. 
> Junk folders inspection can be solicited with a phone call.

A propos workarounds: When I reject emails, because of SPAM, I include 
alternative means to contact the recipients: telephone, fax, website, 
address for snail mail.

Със здраве,
   Дилян

Date: Mon, 30 Nov 2009 07:30:04 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Alessandro Vesely <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

Alessandro Vesely wrote:

> Yeah, complicated settings only work for geeks. Perhaps that's why SPF, 
> DKIM, or other anti-spam techniques' adoptions are snailing. 

IMV, SPF adoption rate has peaked - most companies support it.

 > In addition, postmasters are a category of stranded people...

They get what they pay for. :)

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Mon, 30 Nov 2009 09:50:39 +0100
From: Alessandro Vesely <[email-masked]>
To: IETF SMTP <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

Steve Atkins wrote:
>> Is there an obvious decision matrix? In particular, would reliability be better if a server rejects messages with higher spam scores and delivers to a Junk folder for medium-to-hight, or the other way around?
>>
> 
> Depends on your goals, and on your user demographic. And your definitions.
> 
> I remember a study from a few years back that said that the majority of typical (consumer or non-technical business) users will only look in a bulk folder for about two weeks after they're first exposed to it. With that sort of user base then you're more likely to lose wanted email if you deliver it to a bulk folder than if you reject it, as 9 times in 10 delivery to bulk folder is indistinguishable from silently discarding.

That matches my experience. However, rejection does not offer many 
workarounds, because re-sending will likely repeat the failure, 
while re-writing is something that apparently even spammers don't 
want to do. Junk folders inspection can be solicited with a phone call.

> On the other hand, more sophisticated users (a small minority but the sort of users who might handle role accounts, say) are used to the concept of bulk folders, and have tools like full text search available to them so delivering to a bulk folder may cause less loss of wanted email to those users. It'll likely cause delivery of more unwanted email to their bulk folders, but they have the MUA tools to deal with that tradeoff.

Yeah, complicated settings only work for geeks. Perhaps that's why 
SPF, DKIM, or other anti-spam techniques' adoptions are snailing. In 
addition, postmasters are a category of stranded people...

Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
From: Steve Atkins <[email-masked]>
Date: Sun, 29 Nov 2009 01:03:13 -0800
To: IETF SMTP <[email-masked]>
Sender: [email-masked]

On Nov 29, 2009, at 12:40 AM, Alessandro Vesely wrote:
>=20
> SA provides a certainty measure, and a server may make different =
decisions according to that score. Possible actions may include:
>=20
> * set message header (SA default, Authentication-Results),
> * set IMAP keywords (either Junk or NonJunk for TB users),
> * deliver normally,
> * deliver to a Junk folder,
> * reject with various levels of detail,
> * drop or quarantine, and possibly even
> * create a record for a class of of messages (based on envelope data) =
that the recipient can whitelist using a web form.
>=20
> Is there an obvious decision matrix? In particular, would reliability =
be better if a server rejects messages with higher spam scores and =
delivers to a Junk folder for medium-to-hight, or the other way around?
>=20

Depends on your goals, and on your user demographic. And your =
definitions.

I remember a study from a few years back that said that the majority of =
typical (consumer or non-technical business) users will only look in a =
bulk folder for about two weeks after they're first exposed to it. With =
that sort of user base then you're more likely to lose wanted email if =
you deliver it to a bulk folder than if you reject it, as 9 times in 10 =
delivery to bulk folder is indistinguishable from silently discarding.

On the other hand, more sophisticated users (a small minority but the =
sort of users who might handle role accounts, say) are used to the =
concept of bulk folders, and have tools like full text search available =
to them so delivering to a bulk folder may cause less loss of wanted =
email to those users. It'll likely cause delivery of more unwanted email =
to their bulk folders, but they have the MUA tools to deal with that =
tradeoff.

(Dealing with presentation of spam filtering decisions is a subset of =
the more interesting general problem of assisting recipients in handling =
mail volume - prioritization, tracking and so on).

Cheers,
  Steve

Date: Sun, 29 Nov 2009 09:40:48 +0100
From: Alessandro Vesely <[email-masked]>
To: John C Klensin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, [email-masked], Nate Leon <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

John C Klensin wrote:
> --On Thursday, November 26, 2009 9:20 PM +0100 Arnt Gulbrandsen 
> <[email-masked]> wrote:
>> Nate Leon writes:
>>> I agree with Hector - the less detail the better when
>>> communicating  back to spammers.
>>
>> Oh...?
>>
>> People used to say that years ago. But now, we've had 15 years
>> experience with spammer. Mine is that spammers basically don't
>> bother to learn.

That's probably because they have plenty of easier methods to hit 
their target. IOW, it is a symptom that mail admins don't deploy 
countermeasures quite effectively.

> Of course, if you know for certain that the sender is a spammer, then 
> returning "5yz FOAD" may be an appropriate response without any 
> additional information.  But the reason for that is to avoid consuming 
> resources, not because the additional information would somehow be helpful.

SA provides a certainty measure, and a server may make different 
decisions according to that score. Possible actions may include:

* set message header (SA default, Authentication-Results),
* set IMAP keywords (either Junk or NonJunk for TB users),
* deliver normally,
* deliver to a Junk folder,
* reject with various levels of detail,
* drop or quarantine, and possibly even
* create a record for a class of of messages (based on envelope 
data) that the recipient can whitelist using a web form.

Is there an obvious decision matrix? In particular, would 
reliability be better if a server rejects messages with higher spam 
scores and delivers to a Junk folder for medium-to-hight, or the 
other way around?

Date: Fri, 27 Nov 2009 20:19:36 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: John C Klensin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, [email-masked], Nate Leon <[email-masked]>, =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

John C Klensin wrote:

> So it seems to me to be of no value at all to avoid supplying useful 
> information to a legitimate user whose message might accidentally have 
> been caught in a spam trap because of some theory that doing so will 
> help the spammer.

There is value, but not to argue about.

-- 
Sincerely

Hector Santos
http://www.santronics.com

VBR-Info: md=altn.com; mc=all; mv=vbr.emailcertification.org;
Date: Fri, 27 Nov 2009 16:54:55 -0600
From: Arvel Hathcock <[email-masked]>
To: John C Klensin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, [email-masked], Nate Leon <[email-masked]>, =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

Yep.  I guess I know what I'm working on tonight!  :)

-- 
Arvel

John C Klensin wrote:
> 
> 
> --On Friday, November 27, 2009 4:36 PM -0600 Arvel Hathcock 
> <[email-masked]> wrote:
> 
>> This took the words right out of my own mouth.  I agree 100%.
>> But, it's very difficult to convince many email server admins
>> (whom are known around here by the term "customer") of what
>> John said so we don't do it in our mail server.
> 
> And, that, of course, is why one ships software with optional settings 
> and configuration arrangements so that those who are wise enough to 
> follow the argument can do what you would recommend and those pesky 
> "customers" can do what they think best.   The gains (or losses) have 
> got to be much more serious than in this case to justify telling a 
> customer (or other flavor of luser) that they can't do what they think 
> is necessary.   I think the experience of most of us is that they often 
> gradually come around, often after hearing from _their_ customers, but 
> that is no comfort in the shorter run.
> 
>    john
> 

Date: Fri, 27 Nov 2009 17:53:02 -0500
From: John C Klensin <[email-masked]>
To: Arvel Hathcock <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, [email-masked], Nate Leon <[email-masked]>, =?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?= <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

--On Friday, November 27, 2009 4:36 PM -0600 Arvel Hathcock 
<[email-masked]> wrote:

> This took the words right out of my own mouth.  I agree 100%.
> But, it's very difficult to convince many email server admins
> (whom are known around here by the term "customer") of what
> John said so we don't do it in our mail server.

And, that, of course, is why one ships software with optional 
settings and configuration arrangements so that those who are 
wise enough to follow the argument can do what you would 
recommend and those pesky "customers" can do what they think 
best.   The gains (or losses) have got to be much more serious 
than in this case to justify telling a customer (or other flavor 
of luser) that they can't do what they think is necessary.   I 
think the experience of most of us is that they often gradually 
come around, often after hearing from _their_ customers, but 
that is no comfort in the shorter run.

    john

VBR-Info: md=altn.com; mc=all; mv=vbr.emailcertification.org;
Date: Fri, 27 Nov 2009 16:36:37 -0600
From: Arvel Hathcock <[email-masked]>
To: John C Klensin <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, [email-masked], Nate Leon <[email-masked]>, =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

This took the words right out of my own mouth.  I agree 100%.  But, it's 
very difficult to convince many email server admins (whom are known 
around here by the term "customer") of what John said so we don't do it 
in our mail server.

-- 
Arvel

John C Klensin wrote:

> Let me say this a little differently.  The argument against supplying 
> the spammers with information parallels the old security argument 
> against revealing whether telling which of the user name or password 

<snipped>

Date: Fri, 27 Nov 2009 17:31:05 -0500
From: John C Klensin <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, [email-masked]
cc: Nate Leon <[email-masked]>, =?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?= <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

--On Thursday, November 26, 2009 9:20 PM +0100 Arnt Gulbrandsen 
<[email-masked]> wrote:

>
> Nate Leon writes:
>> I agree with Hector - the less detail the better when
>> communicating  back to spammers.
>
> Oh...?
>
> People used to say that years ago. But now, we've had 15 years
> experience with spammer. Mine is that spammers basically don't
> bother to learn. They learn to pick up email addresses in
> different ways, but other than that, no, basically not. So
> based on my experience i'd say that less detail is _exactly_
> as good or bad as more detail when talking to spammer. More
> detail is better when talking to some non-spammers.

Let me say this a little differently.  The argument against 
supplying the spammers with information parallels the old 
security argument against revealing whether telling which of the 
user name or password caused an authenticate failure.   The 
validity of that argument ultimately depends on whether having 
the additional information will make the attacker (or spammer) 
smarter about organizing the next attack.   In the case of 
credentials and passwords, the answer is clearly "yes" -- it is 
easier to attack the password if you know that the user name, at 
least, as valid.

But, in the case of email, the spammers know that the addresses 
they obtain are valid, or they don't care.  They (or at least 
the high-volume professionals among them) are perfectly capable 
of running Spamassassin (or equivalent), understand how the 
detection heuristics work, and take advantage of both to the 
extent that they care.  The odds of a particular message being 
sent a second time regardless of why it is rejected the first 
time is low, especially in a botnet environment, which is, by 
the way, the reason why soft-timeout delay strategies work so 
well (even though I continue to hate them).

So it seems to me to be of no value at all to avoid supplying 
useful information to a legitimate user whose message might 
accidentally have been caught in a spam trap because of some 
theory that doing so will help the spammer.

Of course, if you know for certain that the sender is a spammer, 
then returning "5yz FOAD" may be an appropriate response without 
any additional information.  But the reason for that is to avoid 
consuming resources, not because the additional information 
would somehow be helpful.

    john

Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
From: Steve Atkins <[email-masked]>
Date: Fri, 27 Nov 2009 10:25:58 -0800
To: IETF SMTP <[email-masked]>
Sender: [email-masked]

On Nov 27, 2009, at 9:56 AM, Tony Finch wrote:

> On Thu, 26 Nov 2009, Steve Atkins wrote:
>>=20
>> Some MTAs only make one line of the rejection available in their =
logs, or
>> don't make it easy to get the rest.
>=20
> They're broken.
>=20

That's not a terribly useful opinion on an operational subject, even if
it's one I tend to agree with.

It's just something that is the case. If you're wanting to communicate
with the people whose mail you are rejecting, to the extent you want
to send back screeds of multiline response, then you also want to
make sure that they can read the critical data in the (fairly common)
case that the multiline responses aren't recorded in the main logs.

>> If you're returning multiline responses, make sure that the first =
line has
>> the critical data, and use the remainder for additional info.
>=20
> IME broken delivery failure reports include the last line of the SMTP =
reply.

Putting the critical information on the first and last lines would seem
a good plan, then, and treat the remainder as fluff that may be useful
to some readers.

Current best practice seems to be to put an explicit rejection code or
(equivalently) a URL in the first line of the rejection message, perhaps
followed by more human readable information.

Cheers,
  Steve

Date: Fri, 27 Nov 2009 17:56:09 +0000
From: Tony Finch <[email-masked]>
To: Steve Atkins <[email-masked]>
cc: IETF SMTP <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

On Thu, 26 Nov 2009, Steve Atkins wrote:
>
> Some MTAs only make one line of the rejection available in their logs, or
> don't make it easy to get the rest.

They're broken.

> If you're returning multiline responses, make sure that the first line has
> the critical data, and use the remainder for additional info.

IME broken delivery failure reports include the last line of the SMTP reply.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Fri, 27 Nov 2009 04:30:34 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Finch <[email-masked]>
CC: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>, IETF SMTP <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

Tony Finch wrote:

> On Thu, 26 Nov 2009, Дилян Палаузов wrote:
>> When an email is evaluated as spam and in turn is rejected during the SMTP
>> dialog, is it reasonable to show in the SMTP response the result of the spam
>> evaluation or not?
> 
> We include the complete SpamAssassin report in our 550 responses. I don't
> buy the security argument, since spammers (or rather their software)
> alredy know how to check their messages with SpamAssassin before letting
> rip.
> 

True, assuming a network wide usage/setting.  You can also have 
per-site, per-user settings.  Relaxed scores can be targeted.
IMV, I would prefer to provide a single "Rejected by SA as Spam" 
response to provide the appearance of using network wide settings.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 26 Nov 2009 21:20:20 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Cc: Nate Leon <[email-masked]>, =?KOI8-R?b?5MnM0c4g8MHMwdXaz9c=?= <[email-masked]>
Sender: [email-masked]

Nate Leon writes:
> I agree with Hector - the less detail the better when communicating 
> back to spammers.

Oh...?

People used to say that years ago. But now, we've had 15 years 
experience with spammer. Mine is that spammers basically don't bother 
to learn. They learn to pick up email addresses in different ways, but 
other than that, no, basically not. So based on my experience i'd say 
that less detail is _exactly_ as good or bad as more detail when 
talking to spammer. More detail is better when talking to some 
non-spammers.

Is your experience different?

Arnt

Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
From: Steve Atkins <[email-masked]>
Date: Thu, 26 Nov 2009 10:31:16 -0800
To: IETF SMTP <[email-masked]>
Sender: [email-masked]

On Nov 26, 2009, at 10:16 AM, Tony Finch wrote:

> On Thu, 26 Nov 2009, =D0=94=D0=B8=D0=BB=D1=8F=D0=BD =D0=9F=D0=B0=D0=BB=D0=
=B0=D1=83=D0=B7=D0=BE=D0=B2 wrote:
>>=20
>> When an email is evaluated as spam and in turn is rejected during the =
SMTP
>> dialog, is it reasonable to show in the SMTP response the result of =
the spam
>> evaluation or not?
>=20
> We include the complete SpamAssassin report in our 550 responses. I =
don't
> buy the security argument, since spammers (or rather their software)
> alredy know how to check their messages with SpamAssassin before =
letting
> rip.

Some MTAs only make one line of the rejection available in their logs, =
or
don't make it easy to get the rest.

If you're returning multiline responses, make sure that the first line =
has
the critical data, and use the remainder for additional info.

Cheers,
  Steve

Date: Thu, 26 Nov 2009 18:16:38 +0000
From: Tony Finch <[email-masked]>
To: =?KOI8-R?B?5MnM0c4g8MHMwdXaz9c=?= <[email-masked]>
cc: IETF SMTP <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--1870869256-1968216189-1259259398=:6581
Content-Type: TEXT/PLAIN; charset=UTF-8
Content-Transfer-Encoding: QUOTED-PRINTABLE

On Thu, 26 Nov 2009, =D0=94=D0=B8=D0=BB=D1=8F=D0=BD =D0=9F=D0=B0=D0=BB=D0=
=B0=D1=83=D0=B7=D0=BE=D0=B2 wrote:
>
> When an email is evaluated as spam and in turn is rejected during the SMT=
P
> dialog, is it reasonable to show in the SMTP response the result of the s=
pam
> evaluation or not?

We include the complete SpamAssassin report in our 550 responses. I don't
buy the security argument, since spammers (or rather their software)
alredy know how to check their messages with SpamAssassin before letting
rip.

Tony.
--=20
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.
--1870869256-1968216189-1259259398=:6581--

Date: Thu, 26 Nov 2009 09:54:29 -0800
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
From: Nate Leon <[email-masked]>
To: =?windows-1251?B?xOjr/+0gz+Dr4PPn7uI=?= <[email-masked]>
Cc: IETF SMTP <[email-masked]>
Sender: [email-masked]

I agree with Hector - the less detail the better when communicating
back to spammers.

How about:
   250 Requested mail action OK

and log the details as Hector suggested.

If you are required to notify the sender of the failure due to
corporate guidelines,
make the msg even more generic:
  550  Your mail was rejected due to local policy

n8

2009/11/26 =C4=E8=EB=FF=ED =CF=E0=EB=E0=F3=E7=EE=E2 <dilyan.palauzov@aegee.=
org>:
>
> Hello,
>
> When an email is evaluated as spam and in turn is rejected during the SMT=
P
> dialog, is it reasonable to show in the SMTP response the result of the s=
pam
> evaluation or not?
>
> E.g. Using SpamAssassin to filter the messages, what response is better a=
nd
> why?
>
> 550 5.7.1 Your mail was evaluated as spam.
>
> or
>
> 550-5.7.1 Your mail was evaluated as spam (see below for details).
> 550-5.7.1
> 550-5.7.1 -0.0 SPF_PASS SPF: sender matches SPF record
> 550-5.7.1 -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
> 550-5.7.1 [score: 0.0000]
> 550 5.7.1 1.0 DKIM_VALID DKIM_VALID
>
>
> Thanks in advance for your opinion,
> =A0=C4=E8=EB=FF=ED
>
>

Date: Thu, 26 Nov 2009 09:58:48 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

My opinion:

The support side will say "the more detail to better" to assist users 
or operators allowing them to be self-reliant to resolve issues (this 
helps reduce support cost).

The security side will say "Don't give clues to bad guys." In this 
specific case, you would be allowing them to learn by trial and error 
to see how to get better scores.  For example, ESMTP AUTH, IMV it is 
bad idea to issue an specific error that says

    530 BAD USERNAME or
    530 BAD PASSWORD

as oppose to just saying

    530 Authentication required

In my view, in your scoring case, too much information.

You can probably decide this based on who you expect to read this. My 
suggestion is to log it to a file.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Дилян Палаузов wrote:

> 
> Hello,
> 
> When an email is evaluated as spam and in turn is rejected during the 
> SMTP dialog, is it reasonable to show in the SMTP response the result of 
> the spam evaluation or not?
> 
> E.g. Using SpamAssassin to filter the messages, what response is better 
> and why?
> 
> 550 5.7.1 Your mail was evaluated as spam.
> 
> or
> 
> 550-5.7.1 Your mail was evaluated as spam (see below for details).
> 550-5.7.1
> 550-5.7.1 -0.0 SPF_PASS SPF: sender matches SPF record
> 550-5.7.1 -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
> 550-5.7.1 [score: 0.0000]
> 550 5.7.1 1.0 DKIM_VALID DKIM_VALID
> 
> 
> Thanks in advance for your opinion,
>   Дилян
> 
> 
> 

X-Mail-Sent-By-AEGEE.org-Account: didopalauzov
Date: Thu, 26 Nov 2009 14:55:28 +0100
From: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
To: IETF SMTP <[email-masked]>
Subject: Returning Details about Spam Evaluation when Rejecting Spam
Sender: [email-masked]

Hello,

When an email is evaluated as spam and in turn is rejected during the 
SMTP dialog, is it reasonable to show in the SMTP response the result of 
the spam evaluation or not?

E.g. Using SpamAssassin to filter the messages, what response is better 
and why?

550 5.7.1 Your mail was evaluated as spam.

or

550-5.7.1 Your mail was evaluated as spam (see below for details).
550-5.7.1
550-5.7.1 -0.0 SPF_PASS SPF: sender matches SPF record
550-5.7.1 -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
550-5.7.1 [score: 0.0000]
550 5.7.1 1.0 DKIM_VALID DKIM_VALID

Thanks in advance for your opinion,
   Дилян

Date: Wed, 25 Nov 2009 16:06:06 +0000
From: Tony Finch <[email-masked]>
To: Mark Martinec <[email-masked]>
cc: Alexey Melnikov <[email-masked]>, IETF SMTP list <[email-masked]>, Yet Another Mail Working Group <[email-masked]>
Subject: Re: [yam] Looking for implementations of RFC 3848 (ESMTP and LMTP Transmission Types Registration)
Sender: [email-masked]

I should probably note that Exim also uses the "smtps" protocol keyword,
if the client says HELO (not EHLO) within a TLS session. This can be
either TLS-on-connect in old smtps style (traditionally port 465), or by
using STARTTLS (yes, EHLO -> STARTTLS -> HELO is a bit bizarre).

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

From: Mark Martinec <[email-masked]>
Organization: J. Stefan Institute
To: Alexey Melnikov <[email-masked]>
Subject: Re: Looking for implementations of RFC 3848          (ESMTP and LMTP Transmission Types Registration)
Date: Wed, 25 Nov 2009 16:10:53 +0100
Cc: IETF SMTP list <[email-masked]>, Yet Another Mail Working Group <[email-masked]>
Sender: [email-masked]

On Tue, 24 Nov 2009, Alexey Melnikov wrote:
> Please let me know if you know of implementations (MTAs/MDAs) that
> generate such parameters (in particular
> ESMTPA/ESMTPS/ESMTPSA/LMTPA/LMTPS/LMTPSA)

Postfix MTA since 2007-07-20:

20070720
        Support for RFC 4954 (SASL AUTH, updates RFC 2554, refines
        some reply codes and introduces DSN enhanced status codes)
        and RFC 3848 ("Received ...  with ESMTPS?A? ...). Currently,
        support for the latter is always on.  Files: smtpd/smtpd.c,
        smtpd/smtpd_sasl_proto.c, smtpd/smtpd_sasl_glue.c.

> > and clients that parse Received headers and analyze/check the
> > corresponding WITH parameter values.
> 
> SpamAssassin uses them to detect message submission hops.

Indeed, since 3.0.3, some time around April 2005.

  Mark

Date: Wed, 25 Nov 2009 14:02:08 +0000
From: Tony Finch <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: IETF SMTP list <[email-masked]>, Yet Another Mail Working Group <[email-masked]>
Subject: Re: Looking for implementations of RFC 3848          (ESMTP and LMTP Transmission Types Registration)
Sender: [email-masked]

On Tue, 24 Nov 2009, Alexey Melnikov wrote:
>
> Please let me know if you know of implementations (MTAs/MDAs) that generate
> such parameters (in particular ESMTPA/ESMTPS/ESMTPSA/LMTPA/LMTPS/LMTPSA)

Exim, since version 4.42 released on 2004-08-27.

> and clients that parse Received headers and analyze/check the
> corresponding WITH parameter values.

SpamAssassin uses them to detect message submission hops.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Wed, 25 Nov 2009 09:32:29 +0000
From: Alexey Melnikov <[email-masked]>
To: Hector Santos <[email-masked]>
CC: IETF SMTP list <[email-masked]>, Yet Another Mail Working Group <[email-masked]>
Subject: Re: Looking for implementations of RFC 3848  (ESMTP and LMTP Transmission Types Registration)
Sender: [email-masked]

Hector Santos wrote:

> Alexey Melnikov wrote:
>
>> I would like to move RFC 3848 to Draft Standard, so I am collecting 
>> information for the implementation report.
>> Please let me know if you know of implementations (MTAs/MDAs) that 
>> generate such parameters (in particular 
>> ESMTPA/ESMTPS/ESMTPSA/LMTPA/LMTPS/LMTPSA) and clients that parse 
>> Received headers and analyze/check the corresponding WITH parameter 
>> values.
>
> Just to get clarification, your interest is in CLIENTS or MAILBOTS 
> that read the Received header compliant with RFC 3848?

I am interested in knowing about both generators and consumers of the 
WITH values.
I think it would be easier to find agents that generate such values, so 
yes, I am especially interested to learn about consumers.

> Currently, our SMTP server supports RFC 1869 so the receiver currently 
> supports the stamping of the required SMTP transaction Received 
> prepending trace line "With ESMTP" (EHLO was used) or "With SMTP" 
> (HELO was used).
>
> Now that I am aware of RFC 3848, I will go ahead and jot this down for 
> a future revision consideration to include support for:
>
>    ESMTPA
>    ESMTPS
>    ESMTPSA

Good.

> However currently, we don't have any support for SMTP clients or SMTP 
> receivers that read/use that information for incoming messages. 
> Personally, this question is probably best addressed at operators 
> and/or 3rd party developers writing AVS mailbot scripts.

Feel free to forward my email to such people.

Date: Wed, 25 Nov 2009 01:32:54 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Alexey Melnikov <[email-masked]>
CC: IETF SMTP list <[email-masked]>, Yet Another Mail Working Group <[email-masked]>
Subject: Re: Looking for implementations of RFC 3848          (ESMTP and LMTP Transmission Types Registration)
Sender: [email-masked]

Alexey Melnikov wrote:

> 
> I would like to move RFC 3848 to Draft Standard, so I am collecting 
> information for the implementation report.
> Please let me know if you know of implementations (MTAs/MDAs) that 
> generate such parameters (in particular 
> ESMTPA/ESMTPS/ESMTPSA/LMTPA/LMTPS/LMTPSA) and clients that parse 
> Received headers and analyze/check the corresponding WITH parameter values.

Just to get clarification, your interest is in CLIENTS or MAILBOTS 
that read the Received header compliant with RFC 3848?

Currently, our SMTP server supports RFC 1869 so the receiver currently 
supports the stamping of the required SMTP transaction Received 
prepending trace line "With ESMTP" (EHLO was used) or "With SMTP" 
(HELO was used).

Now that I am aware of RFC 3848, I will go ahead and jot this down for 
a future revision consideration to include support for:

    ESMTPA
    ESMTPS
    ESMTPSA

However currently, we don't have any support for SMTP clients or SMTP 
receivers that read/use that information for incoming messages. 
Personally, this question is probably best addressed at operators 
and/or 3rd party developers writing AVS mailbot scripts.

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: [email-masked]
Cc: IETF SMTP list <[email-masked]>, Yet Another Mail Working Group <[email-masked]>
Date: Tue, 24 Nov 2009 15:43:40 -0800 (PST)
Subject: Re: [yam] Looking for implementations of RFC 3848 (ESMTP and LMTP Transmission Types Registration)
To: Alexey Melnikov <[email-masked]>
Sender: [email-masked]

> I would like to move RFC 3848 to Draft Standard, so I am collecting
> information for the implementation report.
> Please let me know if you know of implementations (MTAs/MDAs) that
> generate such parameters (in particular
> ESMTPA/ESMTPS/ESMTPSA/LMTPA/LMTPS/LMTPSA) and clients that parse
> Received headers and analyze/check the corresponding WITH parameter values.

Sun's MTA generates any/all of these as appropriate. 

				Ned

Date: Tue, 24 Nov 2009 23:30:09 +0000
From: Alexey Melnikov <[email-masked]>
To: IETF SMTP list <[email-masked]>, Yet Another Mail Working Group <[email-masked]>
Subject: Looking for implementations of RFC 3848  (ESMTP and LMTP Transmission Types Registration)
Sender: [email-masked]

I would like to move RFC 3848 to Draft Standard, so I am collecting 
information for the implementation report.
Please let me know if you know of implementations (MTAs/MDAs) that 
generate such parameters (in particular 
ESMTPA/ESMTPS/ESMTPSA/LMTPA/LMTPS/LMTPSA) and clients that parse 
Received headers and analyze/check the corresponding WITH parameter values.

From: [email-masked]
Cc: [email-masked], Murray S Kucherawy <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Fri, 20 Nov 2009 20:09:56 -0800 (PST)
Subject: Re: sequential processing (was Re: Abort data transfer?)
To: David MacQuigg <[email-masked]>
Sender: [email-masked]

> Dave CROCKER wrote:
> > Murray S. Kucherawy wrote:
> >>> I don't understand how running the filters in series requires that
> >>> the entire message be in a buffer.
> >>
> >> MTA buffers the entire message, sends it to filter #1.  Filter #1
> >> changes the
> >> body.  MTA sends the modified message to #2, including the new body.
> >> This
> >> can only happen if they're in series, and I can't see how it would be
> >> possible if there's not a buffer involved.
> >>
> >> Here's an even better example:  MTA buffers the entire message, sends
> >> it to
> >> filter #1.  Filter #1 orders the message to be rejected (or discarded).
> >> Filter #2 is told "nevermind", and never has to go through the
> >> processing of
> >> the body.  For a very large message, this can be a big performance
> >> win, and
> >> again can only happen if they're in series.
> >
> > Strictly speaking either fully-buffered or partial buffering allows
> > processes to
> > be staged in sequence just fine.  The only issue is whether the
> > filters are staged in sequence with early filters feeding later ones.
> >
> > What full buffering does is to allow the current filter to change an
> > earlier
> > part of the message, based on a later part.  You can't do that in a
> > partial
> > buffering (hot potato) model whether the processing is done only on
> > the current
> > chunk and is then passed back.
> >
> > Simple example would be wanting to a header field to the message,
> > based on
> > something in the body.

> Seems like we have two issues: Series/Parallel and Full/Partial
> Buffering.  Running filters in series is clearly the better strategy
> when the last filter is very time consuming, and earlier filters can
> provide a quick reject.

Not necessarily. A strategy we use is to simply discontinue processing of
additionl milters when one returns a blanket rejection. As long as processing
is discontinued before the time-consuimg step (which very often is at the end
of body), the benefits of overlapped processing can far outweigh the costs.

But an additional comment is in order here. Optimizing things is all well and
good, but there's a point past which the costs of very elaborate (and often
very fragile) configurations far outweighs tiny gains in performance.

I've previously stated that we support running multiple external filtering
agents (milter is only one of the types we support) in both series and
parallel. But I can count the number of times a parallel setup has actually
ended up being appropriate on the fingers of one hand.

				Ned

Date: Wed, 18 Nov 2009 14:56:08 -0700
From: David MacQuigg <[email-masked]>
To: [email-masked]
CC: "Murray S. Kucherawy" <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: sequential processing (was Re: Abort data transfer?)
Sender: [email-masked]

Dave CROCKER wrote:
> Murray S. Kucherawy wrote:
>>> I don't understand how running the filters in series requires that 
>>> the entire message be in a buffer.
>>
>> MTA buffers the entire message, sends it to filter #1.  Filter #1 
>> changes the
>> body.  MTA sends the modified message to #2, including the new body.  
>> This
>> can only happen if they're in series, and I can't see how it would be
>> possible if there's not a buffer involved.
>>
>> Here's an even better example:  MTA buffers the entire message, sends 
>> it to
>> filter #1.  Filter #1 orders the message to be rejected (or discarded).
>> Filter #2 is told "nevermind", and never has to go through the 
>> processing of
>> the body.  For a very large message, this can be a big performance 
>> win, and
>> again can only happen if they're in series.
>
> Strictly speaking either fully-buffered or partial buffering allows 
> processes to
> be staged in sequence just fine.  The only issue is whether the 
> filters are staged in sequence with early filters feeding later ones.
>
> What full buffering does is to allow the current filter to change an 
> earlier
> part of the message, based on a later part.  You can't do that in a 
> partial
> buffering (hot potato) model whether the processing is done only on 
> the current
> chunk and is then passed back.
>
> Simple example would be wanting to a header field to the message, 
> based on
> something in the body.

Seems like we have two issues: Series/Parallel and Full/Partial 
Buffering.  Running filters in series is clearly the better strategy 
when the last filter is very time consuming, and earlier filters can 
provide a quick reject.

The need for full buffering is a separate issue.  It depends on the 
filter process, of course, but the process Murray originally described 
(scanning the headers) could be done in a purely serial fashion, thus 
not requiring a buffer.  A statistical filter, on the other hand, 
requires the entire body in a buffer, so it can go back and do further 
scanning based on, for example, the number of times the word Viagra 
appears in the entire body.

The particular filter I had in mind when I started this thread was the 
simple count of body blocks that I run after each block is received.  
This process is purely serial.  When the count exceeds some limit, the 
rest of the body is rejected.  Until Murray pointed out how Sendmail 
actually works, I was assuming this would protect my receiver from a 
broken or abusive transmitter.

I see no reason the buffer couldn't be just ahead of the eom 
(end-of-message) processing.  This would allow the early filters to 
reject before transferring all the data, while preserving the ability of 
filters called from eom to work on the entire message at once.

    --> Filter#1 --> Filter#2 --> Buffer --> Filter#3 --> MDA

It seems like Sendmail's filter API 
(https://www.milter.org/developers/api/index) is trying to accomplish 
this, but Sendmail itself is not taking full advantage of what the API 
allows.  There are "callbacks" from Sendmail to allow running filters 
after the DATA command, after each header, at the end of all headers, 
after each body block, and at the end of all blocks (eom).  Certain 
commands (message modification) cannot be done except at eom.  Others, 
like sending a reject after too many blocks, *appear* to work before 
eom, but actually do wait for eom.

The one filter I have which *does* require full buffering is 
SpamAssassin.  I use my own buffer for that purpose, and pass it to 
SpamAssassin at eom.  Note: A filter which adds a header does *not* 
require a full buffer.  In an early filter, for example, I gather all 
the data needed to construct an authentication header.  Later, the 
authentication header is inserted using one of the commands that run at eom.

It bothers me that Sendmail does all this extra processing on messages 
that should get an early reject.  I haven't noticed the problem, because 
I don't have a large mail load, but there are plans to use this setup on 
a larger mail system, and now I worry about efficiency.  I like 
Sendmail's ability to let me control everything, but if that is only an 
illusion, I may need to look for a different MTA program.

************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

Date: Wed, 18 Nov 2009 04:58:31 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To:  [email-masked]
CC: "Murray S. Kucherawy" <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: sequential processing (was Re: Abort data transfer?)
Sender: [email-masked]

The idea of running concurrent filters was never a consideration and 
IMV, would add more loading, processing, timing and synchronization 
overhead that isn't necessary.  This is especially the fact that the 
current SMTP design does not lending itself to partial data buffering 
and processing anyway.

In our DATA filter implementation, it evolved to include events; 
drops, timeouts and what we call 'final' events.

Drop event filters handle the cases where the client dropped the line. 
Timeout events were added to help filters that were written wrong, 
i.e. taking too long.  A global timer handled these to help prevent 
DATA timeouts.  Final events are always processed.

In short, the original sequence ordered loading of filters were not 
sufficient the handle the different kinds of filters dependency or 
independency and sharing scenarios when locally written and 3rd party 
filters are part of the mix.

Example:

[Filter List]

   Filter #1 -  Check Header Policy
   Filter #2 -  Check Body Policy
   Filter #3 -  Add some trace record
   Filter #4 -  Check DKIM POLICY
   Filter #5 -  Check other Header/body Policy
   ....
   Filter #N

[Final List]  # always run this

   Filter #F1 - Always create a backup
   Filter #F2 - Act on trace record, if any
   Filter #F3 - Check for some special message.

[Drop List]

   Filter #D2 - Record this drop in some special trace

All filters have a ACCEPT, PASS, SKIP, DISCARD, KEEP results.  PASS, 
SKIP allow filters to continue.  All filters run with a default result 
of PASS.  The last filter PASSed results in ACCEPT.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Dave CROCKER wrote:

> 
> seems like the topic is a long way from the original and therefore 
> worthy of
> subject change.
> 
> 
> Murray S. Kucherawy wrote:
>>> I don't understand how running the filters in series requires that 
>>> the entire message be in a buffer.
>>
>> MTA buffers the entire message, sends it to filter #1.  Filter #1 
>> changes the
>> body.  MTA sends the modified message to #2, including the new body.  
>> This
>> can only happen if they're in series, and I can't see how it would be
>> possible if there's not a buffer involved.
>>
>> Here's an even better example:  MTA buffers the entire message, sends 
>> it to
>> filter #1.  Filter #1 orders the message to be rejected (or discarded).
>> Filter #2 is told "nevermind", and never has to go through the 
>> processing of
>> the body.  For a very large message, this can be a big performance 
>> win, and
>> again can only happen if they're in series.
> 
> Strictly speaking either fully-buffered or partial buffering allows 
> processes to
> be staged in sequence just fine.  The only issue is whether the filters 
> are staged in sequence with early filters feeding later ones.
> 
> What full buffering does is to allow the current filter to change an 
> earlier
> part of the message, based on a later part.  You can't do that in a partial
> buffering (hot potato) model whether the processing is done only on the 
> current
> chunk and is then passed back.
> 
> Simple example would be wanting to a header field to the message, based on
> something in the body.
> 
> d/
> 

Date: Wed, 18 Nov 2009 14:34:39 +0900
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Murray S. Kucherawy" <[email-masked]>
CC: IETF SMTP list <[email-masked]>
Subject: sequential processing (was Re: Abort data transfer?)
Sender: [email-masked]

seems like the topic is a long way from the original and therefore worthy of
subject change.

Murray S. Kucherawy wrote:
>> I don't understand how running the filters in series requires that the 
>> entire message be in a buffer.
> 
> MTA buffers the entire message, sends it to filter #1.  Filter #1 changes the
> body.  MTA sends the modified message to #2, including the new body.  This
> can only happen if they're in series, and I can't see how it would be
> possible if there's not a buffer involved.
> 
> Here's an even better example:  MTA buffers the entire message, sends it to
> filter #1.  Filter #1 orders the message to be rejected (or discarded).
> Filter #2 is told "nevermind", and never has to go through the processing of
> the body.  For a very large message, this can be a big performance win, and
> again can only happen if they're in series.

Strictly speaking either fully-buffered or partial buffering allows processes to
be staged in sequence just fine.  The only issue is whether the filters are 
staged in sequence with early filters feeding later ones.

What full buffering does is to allow the current filter to change an earlier
part of the message, based on a later part.  You can't do that in a partial
buffering (hot potato) model whether the processing is done only on the current
chunk and is then passed back.

Simple example would be wanting to a header field to the message, based on
something in the body.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: "Murray S. Kucherawy" <[email-masked]>
To: David MacQuigg <[email-masked]>
CC: Ned Freed <[email-masked]>, Tony Finch <[email-masked]>, "[email-masked]" <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Tue, 17 Nov 2009 16:00:12 -0800
Subject: RE: Abort data transfer?
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: David MacQuigg [mailto:[email-masked]]
> Sent: Tuesday, November 17, 2009 3:50 PM
> To: Murray S. Kucherawy
> Cc: Ned Freed; Tony Finch; [email-masked]; IETF SMTP list
> Subject: Re: Abort data transfer?
>=20
>  --> Buffer --> Filter#1 --> Filter#2 -->
>=20
> I don't understand how running the filters in series requires that the
> entire message be in a buffer.  In the example above, filter #1 scans
> the headers and decides immediately to replace the body with something
> tiny.  It could abort the transfer, saving a ton of work not just by
> filter #2, but also in the original data transfer to the buffer.

MTA buffers the entire message, sends it to filter #1.  Filter #1 changes t=
he body.  MTA sends the modified message to #2, including the new body.  Th=
is can only happen if they're in series, and I can't see how it would be po=
ssible if there's not a buffer involved.

Here's an even better example:  MTA buffers the entire message, sends it to=
 filter #1.  Filter #1 orders the message to be rejected (or discarded).  F=
ilter #2 is told "nevermind", and never has to go through the processing of=
 the body.  For a very large message, this can be a big performance win, an=
d again can only happen if they're in series.

Date: Tue, 17 Nov 2009 16:50:28 -0700
From: David MacQuigg <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: Ned Freed <[email-masked]>, Tony Finch <[email-masked]>, "[email-masked]" <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

Murray S. Kucherawy wrote:
>> -----Original Message-----
>> From: David MacQuigg [mailto:[email-masked]]
>> Sent: Tuesday, November 17, 2009 10:56 AM
>> To: Ned Freed
>> Cc: Tony Finch; [email-masked]; Murray S. Kucherawy; IETF
>> SMTP list
>> Subject: Re: Abort data transfer?
>>
>> So I'm as puzzled as Ned about the claims of efficiency.  It might make
>> sense for SpamAssassin to wait for the end of data, but I can't see how
>> buffering all the data, and not actually running each milter routine at
>> the time it appears to be called, I can't see how that does anything
>> but open a door for abuse.
>>     
>
> I believe the intent, or at least an intent, is to make it possible for an upstream filter (in the sense that there's a ordered set of filters an instance of Sendmail is using) to make changes that a downstream filter will see.  Doing them all in parallel would mean all filters get the same data, making upstream changes invisible.  So if, for example, filter #1 is a header-based content scanner and #2 is a body-based content scanner, and a gigantic message arrives, filter #1's decision to replace the body with something tiny means #2 doesn't have to do a ton of work on a body that's going to be discarded anyway.
>   

 --> Buffer --> Filter#1 --> Filter#2 -->

I don't understand how running the filters in series requires that the 
entire message be in a buffer.  In the example above, filter #1 scans 
the headers and decides immediately to replace the body with something 
tiny.  It could abort the transfer, saving a ton of work not just by 
filter #2, but also in the original data transfer to the buffer.

************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

From: [email-masked]
Cc: David MacQuigg <[email-masked]>, Ned Freed <[email-masked]>, Tony Finch <[email-masked]>, ned+ietf-smtpmrochekcom <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Tue, 17 Nov 2009 14:13:26 -0800 (PST)
Subject: RE: Abort data transfer?
To: Murray S Kucherawy <[email-masked]>
Sender: [email-masked]

> > -----Original Message-----
> > From: David MacQuigg [mailto:[email-masked]]
> > Sent: Tuesday, November 17, 2009 10:56 AM
> > To: Ned Freed
> > Cc: Tony Finch; [email-masked]; Murray S. Kucherawy; IETF
> > SMTP list
> > Subject: Re: Abort data transfer?
> >
> > So I'm as puzzled as Ned about the claims of efficiency.  It might make
> > sense for SpamAssassin to wait for the end of data, but I can't see how
> > buffering all the data, and not actually running each milter routine at
> > the time it appears to be called, I can't see how that does anything
> > but open a door for abuse.

> I believe the intent, or at least an intent, is to make it possible for an
> upstream filter (in the sense that there's a ordered set of filters an instance
> of Sendmail is using) to make changes that a downstream filter will see.  Doing
> them all in parallel would mean all filters get the same data, making upstream
> changes invisible.  So if, for example, filter #1 is a header-based content
> scanner and #2 is a body-based content scanner, and a gigantic message arrives,
> filter #1's decision to replace the body with something tiny means #2 doesn't
> have to do a ton of work on a body that's going to be discarded anyway.

OK, that's a claim that makes a lot more sense. However, I would argue that
you've made a pretty large sacrifice to get this ordering capability.

FWIW, we support both modes in our implementation. The default is to run all
milters concurrently and with as much processing concurrency between the MTA
and milters as we can eke out. If you want a strict milter cascade, it's a
simple matter to configure one more more intermediate channels and run a
different set of milters on each.

				Ned

From: "Murray S. Kucherawy" <[email-masked]>
To: David MacQuigg <[email-masked]>, Ned Freed <[email-masked]>
CC: Tony Finch <[email-masked]>, "[email-masked]" <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Tue, 17 Nov 2009 12:21:01 -0800
Subject: RE: Abort data transfer?
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: David MacQuigg [mailto:[email-masked]]
> Sent: Tuesday, November 17, 2009 10:56 AM
> To: Ned Freed
> Cc: Tony Finch; [email-masked]; Murray S. Kucherawy; IETF
> SMTP list
> Subject: Re: Abort data transfer?
>=20
> So I'm as puzzled as Ned about the claims of efficiency.  It might make
> sense for SpamAssassin to wait for the end of data, but I can't see how
> buffering all the data, and not actually running each milter routine at
> the time it appears to be called, I can't see how that does anything
> but open a door for abuse.

I believe the intent, or at least an intent, is to make it possible for an =
upstream filter (in the sense that there's a ordered set of filters an inst=
ance of Sendmail is using) to make changes that a downstream filter will se=
e.  Doing them all in parallel would mean all filters get the same data, ma=
king upstream changes invisible.  So if, for example, filter #1 is a header=
-based content scanner and #2 is a body-based content scanner, and a gigant=
ic message arrives, filter #1's decision to replace the body with something=
 tiny means #2 doesn't have to do a ton of work on a body that's going to b=
e discarded anyway.

Date: Tue, 17 Nov 2009 11:55:43 -0700
From: David MacQuigg <[email-masked]>
To: Ned Freed <[email-masked]>
CC: Tony Finch <[email-masked]>, [email-masked], Murray S Kucherawy <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

>>>> This isn't how Sendmail works.  The entire message is cached to the queue
>>>> before milter is told anything about the headers or body.  There's no "a few
>>>> seconds ahead", it's all the way ahead.  Milter has no opportunity to say
>>>> REJECT in the middle of the SMTP DATA phase because the filter doesn't even
>>>> know that's where the MTA is.
>>>>         

Murray is right.  I've confirmed this with a test on my own Sendmail setup.

>>> Interesting... Our milter implementation actually sends the message to the
>>> client as it is received. Of course any REJECTs or ACCEPTs or whatever that
>>> occur along the way are noted and further output to the milter is blocked.
>>> The status is then considered after the message is complete.
>>>
>>> Implementing it the way we did was something of a PITA; I certainly  can
>>> understand why sendmail opted for the collect then send approach.
>>>       
>
>   
>> The sendmail way has the advantage of being more efficient if you have
>> slow clients and/or heavy milter processes.
>>     
>
> How does that follow? The only difference is that in one case you're feeding it
> the data as it comes in while in the other you're buffering up the data and
> sending it as soon as the end of the message is received. Either way the milter
> is running throughout the message transfer, unless you're willing to hold off
> on even starting the milter until after the message transfer, in which
> case it can't do things like refuse specific recipients.
>
> Maybe I'm wrong, but I thought sendmail starts milters at session startup.
>   

On my system (Red Hat Linux), the milter (PyMilter) runs all the time as 
a process separate from Sendmail.  Routines within the milter process 
are called after each SMTP command, after each header, at the end of all 
headers, after each block of body data, and at the end of all data.  A 
separate SpamAssassin process is started (by the milter end of data 
routine) for each message needing filtering.  (The SpamAssassin process 
is actually a stub process, which is run for each message, and the main 
SpamAssassin process, which stays in memory all the time.)  I could be 
wrong on some details here.

So I'm as puzzled as Ned about the claims of efficiency.  It might make 
sense for SpamAssassin to wait for the end of data, but I can't see how 
buffering all the data, and not actually running each milter routine at 
the time it appears to be called, I can't see how that does anything but 
open a door for abuse.

************************************************************     *
* David MacQuigg, PhD    email: macquigg at ece.arizona.edu   *  *
* Research Associate                phone: USA 520-721-4583   *  *  *
* ECE Department, University of Arizona                       *  *  *
*                                 9320 East Mikelyn Lane       * * *
* http://purl.net/macquigg        Tucson, Arizona 85710          *
************************************************************     *

Date: Tue, 17 Nov 2009 17:45:32 +0000
From: Tony Finch <[email-masked]>
To: Ned Freed <[email-masked]>
cc: [email-masked], Murray S Kucherawy <[email-masked]>, David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: RE: Abort data transfer?
Sender: [email-masked]

On Tue, 17 Nov 2009, Ned Freed wrote:
>
> > The sendmail way has the advantage of being more efficient if you have
> > slow clients and/or heavy milter processes.
>
> How does that follow? The only difference is that in one case you're feeding it
> the data as it comes in while in the other you're buffering up the data and
> sending it as soon as the end of the message is received. Either way the milter
> is running throughout the message transfer, unless you're willing to hold off
> on even starting the milter until after the message transfer, in which
> case it can't do things like refuse specific recipients.
>
> Maybe I'm wrong, but I thought sendmail starts milters at session startup.

Er, sorry, I was thinking about data-only filters rather than the
rest of the milter functionality.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

From: [email-masked]
Cc: [email-masked], Murray S Kucherawy <[email-masked]>, David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Date: Tue, 17 Nov 2009 09:07:59 -0800 (PST)
Subject: RE: Abort data transfer?
To: Tony Finch <[email-masked]>
Sender: [email-masked]

> Sorry about the late reply...

> > > This isn't how Sendmail works.  The entire message is cached to the queue
> > > before milter is told anything about the headers or body.  There's no "a few
> > > seconds ahead", it's all the way ahead.  Milter has no opportunity to say
> > > REJECT in the middle of the SMTP DATA phase because the filter doesn't even
> > > know that's where the MTA is.
> >
> > Interesting... Our milter implementation actually sends the message to the
> > client as it is received. Of course any REJECTs or ACCEPTs or whatever that
> > occur along the way are noted and further output to the milter is blocked.
> > The status is then considered after the message is complete.
> >
> > Implementing it the way we did was something of a PITA; I certainly  can
> > understand why sendmail opted for the collect then send approach.

> The sendmail way has the advantage of being more efficient if you have
> slow clients and/or heavy milter processes.

How does that follow? The only difference is that in one case you're feeding it
the data as it comes in while in the other you're buffering up the data and
sending it as soon as the end of the message is received. Either way the milter
is running throughout the message transfer, unless you're willing to hold off
on even starting the milter until after the message transfer, in which
case it can't do things like refuse specific recipients.

Maybe I'm wrong, but I thought sendmail starts milters at session startup.

> Because it runs milters as
> fast as they will go, it doesn't require as much milter concurrency to
> deal with a given volume of mail.

Again, that would only be true if milter only starts after message transfer,
which severely limits it's ability to interact with the SMTP dialogue. I mean,
what's the point of saying you don't like the IP address or EHLO parameter
after the message has been transferred/

				Ned

Date: Sun, 15 Nov 2009 15:08:34 -0500
From: John C Klensin <[email-masked]>
To: Tony Finch <[email-masked]>
cc: Paul Smith <[email-masked]>, David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

--On Sunday, November 15, 2009 19:52 +0000 Tony Finch
<[email-masked]> wrote:

> On Thu, 22 Oct 2009, John C Klensin wrote:
>> 
>> And that analysis is almost certainly one of the reasons why
>> the multipart/partial idea in MIME has never gone much of
>> anywhere as a useful tool.
> 
> It was implemented by OUtlook Express and was certainly used
> in practice by many people - I think it might have been turned
> on by default in some situations. We started blocking messages
> containing partial mime parts in 2003 to prevent it from being
> used to bypass our email AV scanners, and a number of our
> users needed help to turn of the feature.

The fact that it has been turned off in order to make AV
scanners and other antimalware and antispam tools and to prevent
bypassing checks on total message size as seen by the recipient
is consistent with my comment.   Please note that I didn't say
"hasn't been implemented".  I said "gone much of anywhere as a
useful tool".

So I don't think we are disagreeing.

   john

Date: Sun, 15 Nov 2009 19:52:21 +0000
From: Tony Finch <[email-masked]>
To: John C Klensin <[email-masked]>
cc: Paul Smith <[email-masked]>, David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: Re: Abort data transfer?
Sender: [email-masked]

On Thu, 22 Oct 2009, John C Klensin wrote:
>
> And that analysis is almost certainly one of the reasons why the
> multipart/partial idea in MIME has never gone much of anywhere
> as a useful tool.

It was implemented by OUtlook Express and was certainly used in practice
by many people - I think it might have been turned on by default in some
situations. We started blocking messages containing partial mime parts in
2003 to prevent it from being used to bypass our email AV scanners, and a
number of our users needed help to turn of the feature.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Sun, 15 Nov 2009 19:38:20 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
cc: Murray S Kucherawy <[email-masked]>, David MacQuigg <[email-masked]>, IETF SMTP list <[email-masked]>
Subject: RE: Abort data transfer?
Sender: [email-masked]

Sorry about the late reply...

> > This isn't how Sendmail works.  The entire message is cached to the queue
> > before milter is told anything about the headers or body.  There's no "a few
> > seconds ahead", it's all the way ahead.  Milter has no opportunity to say
> > REJECT in the middle of the SMTP DATA phase because the filter doesn't even
> > know that's where the MTA is.
>
> Interesting... Our milter implementation actually sends the message to the
> client as it is received. Of course any REJECTs or ACCEPTs or whatever that
> occur along the way are noted and further output to the milter is blocked.
> The status is then considered after the message is complete.
>
> Implementing it the way we did was something of a PITA; I certainly  can
> understand why sendmail opted for the collect then send approach.

The sendmail way has the advantage of being more efficient if you have
slow clients and/or heavy milter processes. Because it runs milters as
fast as they will go, it doesn't require as much milter concurrency to
deal with a given volume of mail.

(I thought the milter protocol was designed to be implemented the way you
did it, Ned, so I'm surprised that Sendmail doesn't do it that way too.)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.