Subject: Re: draft-atkins-smtp-traffic-control
From: Steve Atkins <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Mon, 31 Oct 2011 21:59:41 -0700
List-Id: <ietf-smtp.imc.org>

On Oct 31, 2011, at 1:23 PM, Murray S. Kucherawy wrote:

> 
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Steve Atkins
>> Sent: Monday, October 31, 2011 9:24 AM
>> To: SMTP Interest Group
>> Subject: Re: draft-atkins-smtp-traffic-control
>> 
>> Yahoo! is currently handling somewhat over 300,000 delivery attempts a
>> second[1]. You don't need to increase the average cost of a mail delivery
>> (either by increasing the cost of a delivery attempt or by increasing
>> the number of delivery attempts per message) by much for it to add
>> up to significant costs.
> 
> I believe it, but to be fair, they also have not come forward to indicate that this is hurting them.  Thus, I have to conclude they can handle it just fine.

I'm not entirely sure they can, but that's not the issue I was discussing, I was simply using delivery rates to one particular ISP to show that even if the cost of a single delivery attempt is small, increasing that cost can add up to significant system resource costs when multiplied by tens of billions of messages a day.

Reducing delivery delay might be a good thing, if it can be done cheaply. Nothing huge, but short delays and messages in order are better than long delays and messages out of order.

Tony suggested that modifying SMTP clients to always retry 4xx responses at a much higher rate than is currently done would be a solution to that, and one that would be effectively free on the modern Internet. Which is a decent observation, for some SMTP servers.

But I think that there are quite a few servers, particularly those we're concerned about that are already close enough to the performance cliff that they sometimes need to shed load, where increasing retry rates by an order of magnitude or two over what's currently done would not be inconsequential.

Cheers,
  Steve

To: Douglas Otis <[email-masked]>
Cc: "Murray S. Kucherawy" <[email-masked]>, SMTP Interest Group <[email-masked]>
From: Mark Andrews <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Date: Tue, 01 Nov 2011 14:03:04 +1100
Sender: [email-masked]

Crypto is good at identifying returnees.  It does nothing in terms
of identifying whether a player is good or bad.  Unless there is a
way to map that identity into a reputation system, which can't be
gamed, it buys little in terms of spam prevention.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

Date: Mon, 31 Oct 2011 19:12:30 -0700
From: Douglas Otis <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

On 10/31/11 1:23 PM, Murray S. Kucherawy wrote:
>> On Monday, October 31, 2011 9:24 AM, Steve Atkins wrote:
>>
>> Yahoo! is currently handling somewhat over 300,000 delivery attempts a
>> second[1]. You don't need to increase the average cost of a mail delivery
>> (either by increasing the cost of a delivery attempt or by increasing
>> the number of delivery attempts per message) by much for it to add
>> up to significant costs.
> I believe it, but to be fair, they also have not come forward to indicate that this is hurting them.  Thus, I have to conclude they can handle it just fine.
As long as there is adequate network bandwidth and server resources, 
email is not likely to see problems.  When the assumption of adequate 
resources is not true, it becomes difficult to know whether there is a 
problem since there won't be evidence of a problem seen in receiving MTA 
logs.  These logs will not include the number of incomplete connections.

Those on smaller networks, such as with schools or those in rural areas, 
coping with email abuse often requires servers located in data-centers 
bringing the joy of paying at the 95 percentile of network traffic and 
the cost of server space and power consumption, in addition to the 
Internet connectivity needed to browse the Internet and to transfer 
filtered email.

We offer a strategy where abusive IP addresses are kept from competing 
with legitimate senders.  While this strategy works fairly well at 
reducing server and network loads, it seems unlikely this strategy can 
be extended to cover IPv6 which currently has 65k times the announced IP 
address space over the maximum available in IPv4.  This large space is 
also growing exponentially well beyond that of Moore's law.  Those who 
decide to not offer IPv6 connectivity will find IPv4 will impose a 
growing percentage of addresses representing a dynamic and uncategorized 
range of senders sharing the same addresses.

A hybrid approach where connections are first established using Kerberos 
could provide a good solution.  MIT has made an implementation of 
Kerberos freely available, under copyright permissions similar to BSD 
licensing.  In 2007, MIT formed the Kerberos Consortium where sponsors 
include Oracle, Apple, Google, Microsoft, and Centrify.  The IETF has 
the krb-wg supporting Kerberos.

By taking a layered approach where SMTP uses Kerberos tickets as a basis 
for acceptance, IPv6 address information can be obtained to open access 
to SMTP servers, where connections can also be validated using Kerberos 
tickets.  An extension to SMTP-Auth would be run in conjunction with the 
infrequent use of Kerberos leveraging PKI or DANE.  This would permit an 
efficient exchange of email able to avoid much of the costs associated 
with email abuse.

The many problems created by reputation systems that fail to 
authenticate the entities being rated can be avoided as well.  Kerberos 
is already being heavily used by Apple and can support the integration 
of other Internet based services.

-Doug

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Mon, 31 Oct 2011 13:23:23 -0700
Subject: RE: draft-atkins-smtp-traffic-control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Steve Atkins
> Sent: Monday, October 31, 2011 9:24 AM
> To: SMTP Interest Group
> Subject: Re: draft-atkins-smtp-traffic-control
> 
> Yahoo! is currently handling somewhat over 300,000 delivery attempts a
> second[1]. You don't need to increase the average cost of a mail delivery
> (either by increasing the cost of a delivery attempt or by increasing
> the number of delivery attempts per message) by much for it to add
> up to significant costs.

I believe it, but to be fair, they also have not come forward to indicate that this is hurting them.  Thus, I have to conclude they can handle it just fine.

Date: Mon, 31 Oct 2011 09:32:17 -0700
From: Claus Assmann <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: retrying very often (was: draft-atkins-smtp-traffic-control)
Reply-To: [email-masked]
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Mon, Oct 31, 2011, Tony Finch wrote:
> Steve Atkins <[email-masked]> wrote:

> > Each delivery attempt consumes significant system resources, both at the
> > client and the server.

> Maybe ten or twenty years ago.

Have you checked the resource usage of a TLS handshake?

> > If, for example, the client was continually deferred and retried every
> > minute,

> That would be silly. (Still, some of my users run Exchange servers that do
> this.)

I see that happening almost every day on one of my servers.
However, this kind of misconfiguration won't be stopped by
a protocol extension...

Subject: Re: draft-atkins-smtp-traffic-control
From: Steve Atkins <[email-masked]>
Date: Mon, 31 Oct 2011 09:24:12 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 31, 2011, at 9:14 AM, Tony Finch wrote:

> Steve Atkins <[email-masked]> wrote:
>> 
>> Each delivery attempt consumes significant system resources, both at the
>> client and the server.
> 
> Maybe ten or twenty years ago.

Yahoo! is currently handling somewhat over 300,000 delivery attempts a
second[1]. You don't need to increase the average cost of a mail delivery
(either by increasing the cost of a delivery attempt or by increasing
the number of delivery attempts per message) by much for it to add
up to significant costs.

>> If, for example, the client was continually deferred and retried every
>> minute,
> 
> That would be silly. (Still, some of my users run Exchange servers that do
> this.)

It's what you seemed to be suggesting as a way for poorly behaved
clients to deliver mail in a timely manner in the face of 4xx deferrals.

And it is something that some badly behaved clients do do - even when
they have thousands of messages queued for a specific server.

Cheers,
  Steve

[1] http://visualize.yahoo.com/

Date: Mon, 31 Oct 2011 16:14:15 +0000
From: Tony Finch <[email-masked]>
To: Steve Atkins <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

Steve Atkins <[email-masked]> wrote:
>
> Each delivery attempt consumes significant system resources, both at the
> client and the server.

Maybe ten or twenty years ago.

> If, for example, the client was continually deferred and retried every
> minute,

That would be silly. (Still, some of my users run Exchange servers that do
this.)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
East Dogger, Fisher, German Bight: Southerly 4 or 5, increasing 6. Moderate,
occasionally rough. Occasional rain or drizzle. Moderate, occasionally poor.

Subject: Re: draft-atkins-smtp-traffic-control
From: Steve Atkins <[email-masked]>
Date: Mon, 31 Oct 2011 09:06:00 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 31, 2011, at 8:55 AM, Tony Finch wrote:

> Steve Atkins <[email-masked]> wrote:
>> 
>> Mail that's delivered quickly, in a minute or so, vs mail that sometimes
>> takes a half hour to deliver makes a difference to the end user.
>> 
>> An ESMTP client, such as your ISP smarthost, will wait a while before
>> retrying in response to a 4xx deferral. The RFCs say they should wait at
>> least 30 minutes the first time.
> 
> Why not just ignore this suggestiona and retry more frequently?

Each delivery attempt consumes significant system resources, both at the
client and the server. It's also "unfair" behavior by the client, consuming
more server resources than better behaved clients in an attempt to get
mail delivered faster than that from other clients. 

If, for example, the client was continually deferred and retried every
minute, rather than using a reasonable delay will consume 30 times
the resources over the first half hour compared to a well-behaved client.

Cheers,
  Steve

Date: Mon, 31 Oct 2011 15:55:51 +0000
From: Tony Finch <[email-masked]>
To: Steve Atkins <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

Steve Atkins <[email-masked]> wrote:
>
> Mail that's delivered quickly, in a minute or so, vs mail that sometimes
> takes a half hour to deliver makes a difference to the end user.
>
> An ESMTP client, such as your ISP smarthost, will wait a while before
> retrying in response to a 4xx deferral. The RFCs say they should wait at
> least 30 minutes the first time.

Why not just ignore this suggestiona and retry more frequently?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
South Biscay: Southerly veering westerly 4 or 5, becoming variable 3 later.
Rough, occasionally very rough later. Rain or showers. Good, occasionally poor
later.

Date: Sun, 30 Oct 2011 17:40:35 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Putting it all together
Sender: [email-masked]

Frank Ellermann wrote:
> On 30 October 2011 11:06, Hector Santos wrote:
> 
>> I personally don't think this will work without
>> enforcement.
> 
> IBTD, it is "opt-in" on both sides.  For starters
> you can't "opt-in" if it's not yet implemented in
> your SMTP.  Some folks said that they would never
> publish the retry details, because it could help
> to defeat their GL in simple botnet clients.

That is why its OPTIONAL. :)  Measure PROS/CONS on a per site basis, 
not one's site patterns as a baseline for what should be globally done 
or not done.

> And of course clients are not forced to take the
> hint, they can stick to their own retry strategy
> based on RFC 5321.

Again, optional.  I like to think the proposal offers a SMTP compliant 
practical engineering solution to an existing issue.  No new training 
of how systems should behave.    Simply adding optional client/server 
negotiated "time data" to the words "temporary" and "try again later."

> The "enforcement" is limited to "implement the
> retry format as documented in RFC-to-be, because
> that is your best chance for interoperability in
> this (pun) grey area."

Enforcement is probably the wrong word, but protocols don't work well, 
if its based on a Honor System and little to no regulations behind it.

And in view, the examples used (down, maintenance) unfortunately 
muddles the water because its a different set of issues.  For example, 
if you are down, meaning no listening server is accepting connections, 
that provides an explicit error condition that may be already factored 
in MTA retry logics.  I don't suspect an MTA is going to continue with 
short initial retry delays for a down system.  It may try other MX/IP 
if any.

So if a response such as:

        "421 domain, Wait=3600 NextMX=none"

is provided, that could indeed help preempt secondary MX/IP attempts.

But in my opinion, it SHOULD mean what it says so that the client is 
not lied to and tries again and still finds the server is still 
issuing 421s. Another participant commented he was concern about 
servers that might lie about the hints. It would be very much similar 
issue.  Draft -02 has the follow to help address this concern:

     If a SMTP server offers a "retry=time-delay" hint which results
     in a wasted 2nd attempt and requires additional attempts, the
     SMTP client MAY begin to ignore the server's "retry=time-delay" hint
     after the 2nd wasted retry. The SMTP client implementation can 
decide
     what limits to place on honoring "retry=time-delay" hints and wasted
     attempts it provides.

That is what I mean overall with "Enforcement" semantics and its more 
appropriate IMO after the 421 and the SMTP is open for business.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sun, 30 Oct 2011 17:05:41 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Per-session SMTP extensions
Sender: [email-masked]

Peter J. Holzer wrote:
>>> 250 STRUCTUREDTEXT

> The replies are generated by the server. If the server supports
> structured text it can make sure that it never generates a message which
> can be mistaken for a keyword.
> 
> This should be possible even if the server includes error messages from
> other software (spam filter, ldap, ...) in its messages. For example, if
> the keyword:value pairs can only come between the extended status code
> and the human readable text, it is sufficient to prefix any externally
> generated message with a bit of explanatory text. Other ways of marking
> the keyword:value pairs (e.g., enclosing them in []) may need more
> complex escape mechanisms to avoid false positives.
> 
> So I don't think it is necessary that the client requests specific
> information. It may be desirable, but right now I don't see a compelling
> reason, and I lean towards keeping things simple.

+1.

That was my concern with the ideal goal of trying to generalize all. 
In my view, if anything, as shown by the different ideas for syntax, 
"tag:value" vs "tag=value", etc, I think we would benefit by 
standardizing this particular advancement for SMTP for extended 
structured text formats if only for one reason - MTA will benefit by 
having a single parser and not have to redundantly reproduce similar 
parsing code for every new extension that comes along.

Also from a plug and play, backward compatibility, minimum conflict 
design standpoint, this would be something IMEO that you would 
naturally append to the response text, or before or within the text 
string but after any extended status codes.  Adding it before any 
extended status code has a chance of conflict with an existing MTA 
looking for them to be immediately after a reply code.

We can use a JSON syntax which offer flexibility and also known 
parsers exist so a less concern regarding parsing errors. But I think 
a simple

       tag-value-pairs = *(" " tag "=" value [";"])

if I got the ABNF correct, would be easy.

We could also use HTTP format for name=value& pairs as well;

       name-value-pairs = *(name "=" value ["&"])

And there is plenty for parsers for URL parameters.

Examples:

   450 4.7.1 Temporarily restricted. Retry=300&Expire=172800

   Blocked for 5 minutes, 2 days temporary white list expiration.

   250 Message Accepted. MaxIdle=35&MaxwMail=2

   First transaction completed, server warning client they have 35 seconds
   to start a new transaction.  Two additional transactions allowed.

And so on. So if anything, the idea of needing to established IANA 
registry terms with special means and format values would help.

    retry   =  Server block time and MTA retry delay time
    expire  =  Server expiration of sender information tracking
    maxidle =  Server idle time allowed

But this again goes on to complicate these. IMO, the immediate desire 
is to teach the MTA to better retry to reduce waste and delayed mail 
delivery.  Down times and maintenance is different than all this. All 
the other stuff is already within the controls of the server.  Servers 
can force a drop for unreasonable long idleness and it can always 
issue a 550 to a new MAIL FROM transaction to limit it during a 
session.  What we don't have is giving the MTA feedback on what "try 
again later" means to the server.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sun, 30 Oct 2011 19:38:05 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Per-session SMTP extensions (was: Re: 4yz Temporary Rejections is part of the SMTP Protocol)
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--VgNP6wxEowREGVu+
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-30 09:41:42 -0400, John C Klensin wrote:
> --On Sunday, October 30, 2011 11:10 +0100 "Peter J. Holzer"
> <[email-masked]> wrote:
> > So, a new extension STRUCTUREDTEXT could be modelled after
> > ENHANCEDSTATUSCODES:
> >=20
> > 220 server.example ESMTP server
> > EHLO client.example
> > 250-server.example Hi client.example
> > 250-SIZE 10000000
> > 250-ENHANCEDSTATUSCODES
> > 250 STRUCTUREDTEXT
> > mail from:<[email-masked]> SIZE=3D543210
> > 250 2.1.0 Ok
> > rcpt to:<[email-masked]>
> > 450 4.7.0 wait:55 greylisted
> >...
>=20
> > This doesn't allow the inclusion of structuredtext in the
> > greeting (or at least there is no way to alert the client to
> > its presence).
>=20
> Right
>=20
> > Keywords also need to be defined in (standards track?) rfcs and
> > registered to prevent conflicting use. So maybe a generic
> > STRUCTUREDTEXT extension isn't a good idea after all.
>=20
> Worse, while one can work the odds, nothing prevents those
> things you want to define as keywords from showing with
> unstructured text following a 45x reply.

The replies are generated by the server. If the server supports
structured text it can make sure that it never generates a message which
can be mistaken for a keyword.

This should be possible even if the server includes error messages from
other software (spam filter, ldap, ...) in its messages. For example, if
the keyword:value pairs can only come between the extended status code
and the human readable text, it is sufficient to prefix any externally
generated message with a bit of explanatory text. Other ways of marking
the keyword:value pairs (e.g., enclosing them in []) may need more
complex escape mechanisms to avoid false positives.

So I don't think it is necessary that the client requests specific
information. It may be desirable, but right now I don't see a compelling
reason, and I lean towards keeping things simple.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--VgNP6wxEowREGVu+
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--VgNP6wxEowREGVu+--

Date: Sun, 30 Oct 2011 13:22:01 -0400
To:  [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: 4yz Temporary Rejections is part of the SMTP  Protocol
Sender: [email-masked]

At 11:56 -0400 on 10/30/2011, John C Klensin wrote about Re: 4yz 
Temporary Rejections is part of the SMTP Protocol:

>  > If the intent is know to issue a maintenance notice, then I
>>  would probably not activate AUTH. So it becomes:
>>
>>  <connection>
>>  S: 220 hi
>>  C: EHLO me
>>  S: 250 SMTPRETRY
>  >...
>Independent of the details, switching the set of extensions
>offered when the server was effectively unavailable for a while
>in the EHLO response is an option that hadn't occurred to me
>that that I think makes a lot of sense.

It is IMO also in line with the intent of the EHLO reply in that it 
documents those extensions that the server is able/willing to 
support. If the server is currently unavailable, it should only 
report those extensions that it can support in its current state - In 
this case that if requested it will report when it expects to be back 
in operation.

Date: Sun, 30 Oct 2011 13:59:05 -0400
To:  [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: Per-session SMTP extensions (was: Re: 4yz Temporary  Rejections	is part of the SMTP Protocol)
Sender: [email-masked]

At 09:41 -0400 on 10/30/2011, John C Klensin wrote about Re: 
Per-session SMTP extensions (was: Re: 4yz Temporary Rej:

>  > Keywords also need to be defined in (standards track?) rfcs and
>>  registered to prevent conflicting use. So maybe a generic
>  > STRUCTUREDTEXT extension isn't a good idea after all.
>
>Worse, while one can work the odds, nothing prevents those
>things you want to define as keywords from showing with
>unstructured text following a 45x reply.   That is the advantage
>that one gains with a client request -- independent of anything
>else, it is a client acknowledgement that syntax and keywords
>will be interpreted in accord with a particular specification
>and registry.

There is a simple solution to that issue as you note. Make the key 
words parms of the STRUCTUREDTEXT extension. The Server offers 
STRUCTUREDTEXT with those keywords it can send and the Client sends a 
STRUCTUREDTEXT command with keywords it can parse back (like AUTH 
works now). If the Client does not send back a STRUCTUREDTEXT command 
it is broken if it tries to parse the 45x looking for what looks like 
keywords. A server that supports/offers STRUCTUREDTEXT would IMO be 
broken if it send text in the 45x that can be mistaken for keywords 
UNLESS it has been requested to use these keywords by the client.

The only GOTCHA is if there were a newly registered keyword that the 
Server does not [yet?] support that occurs in the text as non-keyword 
random text and that case should be handled by the client not 
accepting it as a valid keyword in the absence of it being offered in 
the 250 EHLO reply and acknowledged by the client via its handshake 
reply.

Date: Sun, 30 Oct 2011 11:56:02 -0400
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

--On Sunday, October 30, 2011 05:25 -0400 Hector Santos
<[email-masked]> wrote:

>...
>> 220 SMTPRETRY
>> AUTH <stuff>
>> 220 OK (which may need to be faked if the real auth database
>> is not  available)
>> EHLO me again
>> 220 SMTPRETRY
>> NEEDRETRYDATA
>> 621 Server in maintenance, try later wait:600
> 
> We know you meant 250 above.  Two comments.

On not, depending on what advertising the "NEEDRETRY" extension
means.  It could open up the 6yz set of codes.

> If the intent is know to issue a maintenance notice, then I
> would probably not activate AUTH. So it becomes:
> 
> <connection>
> S: 220 hi
> C: EHLO me
> S: 250 SMTPRETRY
>...

Independent of the details, switching the set of extensions
offered when the server was effectively unavailable for a while
in the EHLO response is an option that hadn't occurred to me
that that I think makes a lot of sense.

> Also, if this is one of the purposes, then maybe a service
> keyword, like perhaps ALERT can work do this:
> 
> S: 220 hi
> C: EHLO me
> S: 250 ALERT OIL CHANGE. QUIT NOW and WAIT:600
> C: QUIT
> S: 221 see ya!

Some review of the requirements for arguments to EHLO reply
keywords is in order here, but the principle is an interesting
one.  Like all other good extensions, if the client doesn't
recognize the keyword, it can ignore it.  If it doesn't it knows
what the arguments mean.

>...

Folks, I'm seeing lots and lots of messages and design options
here: Server announcements with more or less detail, overloading
old reply codes versus using new ones, whether or not a client
request (or acknowledgment) is needed, appropriateness of
connection-time announcements, per-session versus per-message
behaviors, and so on.  My sense is that both the proposals that
have been written up have improved and that others have started
to emerge.  In the process, the questions about whether or not
one or more features of this class would really be adopted and
useful get lost -- it is lots easier and more interesting to
focus on the design details.

May I suggest that it is time that those who think this is
worthwhile get together, off this list, and see if you can shape
one or more complete and coherent proposals, with analyses of
the tradeoffs as appropriate, and then come back here or to the
ADs?

    john

Date: Sun, 30 Oct 2011 09:41:42 -0400
From: John C Klensin <[email-masked]>
To: "Peter J. Holzer" <[email-masked]>, [email-masked]
Subject: Re: Per-session SMTP extensions (was: Re: 4yz Temporary Rejections	is part of the SMTP Protocol)
Sender: [email-masked]

--On Sunday, October 30, 2011 11:10 +0100 "Peter J. Holzer"
<[email-masked]> wrote:

> On 2011-10-29 19:51:16 -0400, John C Klensin wrote:
>> In reverse order... most of the extensions so far are "server
>> offers"/"client wants".   There are exceptions, such as
>> ENHANCEDSTATUSCODES and PIPELINING (note that the former works
>> without a client request precisely because SMTP doesn't
>> specify syntax for the portion of replies that follow the
>> three-digit code).
> 
> The "structured text" responses are very similar to the
> enhanced status codes in this regard. A client which doesn't
> know about them will just see them as part of the human
> readable portion of the message and ignore them. Only a client
> which does know about them will look for them (and potentially
> confuse some random text with a structured text response).

And, of course, that possible confusion and the server knowing
whether there is any reasonable likelihood that the client will
pay attention to its requests are the argument for a client
request.  Arguably, the reason why an ENHANCEDSTATUSCODES
announcement was needed at all was so that a client could know
that digits would appear at the beginning of the reply text
actually meant something and what they meant.  It is not clear
to me that the same relationship applies here, especially if the
server, by making the announcement, doesn't _obligate_ itself to
use the structured information in the code.  A server might want
to provide the information for some senders and not others; at
the time it sends the EHLO response, it doesn't know what the
reverse paths are going to look like.  I don't know whether that
is worth the trouble or not, but it is a consideration.

That said, I was mostly talking about what would be needed if
one wanted to bind the traffic management (or whatever)
extensions to a new primary code or code, one that extends the
5321 code model.   For that, one would almost certainly want
explicit "server offers"/"client requests" to avoid the risk of
the code being interpreted in some entirely different way or,
worse, having the client abandon the transaction and bounce the
message to wherever it came from because the server did
something sufficiently unexpected.
 
> So, a new extension STRUCTUREDTEXT could be modelled after
> ENHANCEDSTATUSCODES:
> 
> 220 server.example ESMTP server
> EHLO client.example
> 250-server.example Hi client.example
> 250-SIZE 10000000
> 250-ENHANCEDSTATUSCODES
> 250 STRUCTUREDTEXT
> mail from:<[email-masked]> SIZE=543210
> 250 2.1.0 Ok
> rcpt to:<[email-masked]>
> 450 4.7.0 wait:55 greylisted
>...

> This doesn't allow the inclusion of structuredtext in the
> greeting (or at least there is no way to alert the client to
> its presence).

Right

> Keywords also need to be defined in (standards track?) rfcs and
> registered to prevent conflicting use. So maybe a generic
> STRUCTUREDTEXT extension isn't a good idea after all.

Worse, while one can work the odds, nothing prevents those
things you want to define as keywords from showing with
unstructured text following a 45x reply.   That is the advantage
that one gains with a client request -- independent of anything
else, it is a client acknowledgement that syntax and keywords
will be interpreted in accord with a particular specification
and registry.

Please understand that I'm not pushing any particular approach
here or in other notes in these threads; I'm just trying to
expose some of the design options, considerations, and tradeoffs.

    john

Date: Sun, 30 Oct 2011 09:41:03 -0400
From: John C Klensin <[email-masked]>
To: Paul Smith <[email-masked]>
cc: "Carl S. Gutekunst" <[email-masked]>, "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

--On Sunday, October 30, 2011 08:26 +0000 Paul Smith
<[email-masked]> wrote:

> On 29/10/2011 21:10, John C Klensin wrote:
>> would be perfectly adequate to permit support for, e.g., 6yz
>> or x7z codes.
>> 
>> Whether it is a good idea or not is another question, but it
>> seems to me that the protocol design is adequate.
> Of course, this assumes you don't want the 'new' response
> before the negotiation.
> 
> In the case of a 'wait' extension, it would be 'nicer' to have
> 
> <connection>
> 421 Server in maintenance, try later wait:600

With three qualifications, you can do that today:
	(i) According to RFC 5321, 421 is one of the replies
	that has a specific syntax.  It must appear as 
	   421 <domain> SP <string>
	not just 
	   421 <string>
	So the above cannot be used here at all and cannot be
	used anywhere else without a server offer and client
	acknowledgment.
	
	(ii) It depends on a slightly stretched interpretation
	of "(This may be a reply to any command if the service
	knows it must shut down)" )or the similar text in
	Section 3.8).  Precisely speaking, connection opening
	isn't a command and you are describing a slightly
	different situation from "about to shut down".  I think
	that interpretation might be reasonable, but, if it were
	to go anywhere, it needs discussion in a dedicated
	thread.
	
	(iii) The client may not interpret the string as meaning
	anything at all other than "temporary failure, retry
	sometime".

Regardless of "nicer", that is about the best you can do with
the SMTP mode.  You know that already.

> rather than
> 
> <connection>
> 220 hi
> EHLO me
> 220-AUTH LOGIN
> 220 SMTPRETRY
> AUTH <stuff>
> 220 OK (which may need to be faked if the real auth database
> is not available)
>...

Not _quite_ that bad because the statement about 421 above
clearly applies to AUTH.   So, unless I'm missing something, 
  [...]
  220 SMTPRETRY
  AUTH <stuff>
  421 <smtp.my.example> in maintenance, try later wait:600

Would be perfectly valid today without needing an "server
offers" extension (with or without a client request.  They might
still be a good idea (see other notes), but, if you are willing
to risk having the reply text treated as noise, there is no
problem.

    john

From: Frank Ellermann <[email-masked]>
Date: Sun, 30 Oct 2011 13:07:15 +0100
Subject: Re: Putting it all together
To: Hector Santos <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 30 October 2011 11:06, Hector Santos wrote:

> I personally don't think this will work without
> enforcement.

IBTD, it is "opt-in" on both sides.  For starters
you can't "opt-in" if it's not yet implemented in
your SMTP.  Some folks said that they would never
publish the retry details, because it could help
to defeat their GL in simple botnet clients.

And of course clients are not forced to take the
hint, they can stick to their own retry strategy
based on RFC 5321.

The "enforcement" is limited to "implement the
retry format as documented in RFC-to-be, because
that is your best chance for interoperability in
this (pun) grey area."

-Frank

Date: Sun, 30 Oct 2011 11:10:41 +0100
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Per-session SMTP extensions (was: Re: 4yz Temporary Rejections is part of the SMTP Protocol)
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--5tINJiUS2/Dn5Rhr
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-29 19:51:16 -0400, John C Klensin wrote:
> In reverse order... most of the extensions so far are "server
> offers"/"client wants".   There are exceptions, such as
> ENHANCEDSTATUSCODES and PIPELINING (note that the former works
> without a client request precisely because SMTP doesn't specify
> syntax for the portion of replies that follow the three-digit
> code).

The "structured text" responses are very similar to the enhanced status
codes in this regard. A client which doesn't know about them will just
see them as part of the human readable portion of the message and ignore
them. Only a client which does know about them will look for them (and
potentially confuse some random text with a structured text response).

So, a new extension STRUCTUREDTEXT could be modelled after
ENHANCEDSTATUSCODES:

220 server.example ESMTP server
EHLO client.example
250-server.example Hi client.example
250-SIZE 10000000
250-ENHANCEDSTATUSCODES
250 STRUCTUREDTEXT
mail from:<[email-masked]> SIZE=3D543210
250 2.1.0 Ok
rcpt to:<[email-masked]>
450 4.7.0 wait:55 greylisted
rcpt to:<[email-masked]>
450 4.2.2 spaceleft:123456 wait:7200 mailbox over quota
rcpt to:<[email-masked]>
250 2.1.0 Ok
=2E..

This doesn't allow the inclusion of structuredtext in the greeting (or
at least there is no way to alert the client to its presence).

Keywords also need to be defined in (standards track?) rfcs and
registered to prevent conflicting use. So maybe a generic STRUCTUREDTEXT
extension isn't a good idea after all.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--5tINJiUS2/Dn5Rhr
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--5tINJiUS2/Dn5Rhr--

Date: Sun, 30 Oct 2011 06:06:08 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: Putting it all together
Sender: [email-masked]

Paul Smith wrote:
> 
> On 30/10/2011 03:57, Hector Santos wrote:
>>
>> It seems to me ID2 and ID3 are very different since as I understand 
>> there is no enforcement in ID2 where its a critical part for ID3.
> 
> I think you're over complicating it.
> 
>  From the point of view of the client AND server, the wait:xxx response 
> doesn't care whether it's enforced or not. The enforcement is a part of 
> the greylisting system, not a part of the protocol enhancement.

Didn't I state that?

> The server should not penalise a client which ignores its wait:xxx hints 
> (eg it may choose to do so, if it sees 'wait:100000'), so, in fact, the 
> 'wait:xxx' response is not enforced separately. 

I personally don't think this will work without enforcement.  When not 
enforced, the wait proposal now becomes narrow scope for scenarios 
like down or maintenance periods when many systems already employ 
redundancy and/or the maintenance job is made to be so short in time, 
that its not feasible to go thru all this, especially when it was 
stated it is a "Best Guess" wait and very possible not to be accurate 
which means there is a whelm of possibility the server will be 
indefinitely unavailable.

> Similarly, a server which is down for an hour, implicitly 'enforces' 
> attempts to reconnect before that hour is up, but it shouldn't be part 
> of a spec that it should do so, it just happens. All the 'wait:xxx' 
> enhancement is, is a bit of helpful info for the client, if it wants to 
> use it.

Sure, like anything else. But IMV, no enforcement equates to less 
effectiveness, but then again, its a narrow scope and there are 
already solutions for down and maintenance periods.  There is a 
different between being "Down" and "Maintenance Period."  Down is 
Down. No one is answering and there is an explicit (socket) error code 
that can be tied to altering retry schedules (i.e, not part of the 
normal 4yz reasons).  For maintenance, it all depends what that means 
and not everyone uses the same package nor can we assume a protocol 
will work under the premise all software does maintenance the same 
way. Your (speaking in general) maintenance needs could will be very 
different than mine.  (We have an RPC client/server framework - 
maintenance is at the back end and the SMTP server does not need to go 
into 421 mode.)

In my experience, sites with normal and regular maintenance schedule 
needs will factor the feasibility of the amount down time vs 
implementing redundancy - i.e. there is never a receiver unavailable. 
  Certainly not for the idea of "maintenance." Some systems also have 
hot file I/O, backups and don't need to go into a maintenance mode. 
The system runs 24x7x356.

So its all very different for many.

I guess I don't see the need myself just for this 421 wait when the 
particular scenario is maintenance.  If maintenance takes a long time, 
they need to revisit the setup or make sure additional servers are 
available that will at a minimum receive the mail perhaps and keep it 
in a queue.

As I stated in the other post, what I do see is a saving in the 
connect timeout  when you have a listening server for the sole purpose 
of issuing a 421 with a wait hint. But if its just a best guess with 
the possibility it can be wrong, and still induce wasted attempts, 
then IMV just turn off the listening server and allow the MTA to get 
the connection timeout so it can increase the delay for the next retry.

Beyond the 421 and the service is open, I don't see the benefits for 
issuing a wait hint when there is no enforcement.  By benefits, it 
includes the inevitable need to deal with those that don't support it 
or ignore it making wait hints less effective.

My opinion.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sun, 30 Oct 2011 05:25:31 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

Paul Smith wrote:
> 
> On 29/10/2011 21:10, John C Klensin wrote:
>> would be perfectly adequate to permit support for, e.g., 6yz or
>> x7z codes.
>>
>> Whether it is a good idea or not is another question, but it
>> seems to me that the protocol design is adequate.
> Of course, this assumes you don't want the 'new' response before the 
> negotiation.
> 
> In the case of a 'wait' extension, it would be 'nicer' to have
> 
> <connection>
> 421 Server in maintenance, try later wait:600
> 
> rather than
> 
> <connection>
> 220 hi
> EHLO me
> 220-AUTH LOGIN
> 220 SMTPRETRY
> AUTH <stuff>
> 220 OK (which may need to be faked if the real auth database is not 
> available)
> EHLO me again
> 220 SMTPRETRY
> NEEDRETRYDATA
> 621 Server in maintenance, try later wait:600

We know you meant 250 above.  Two comments.

If the intent is know to issue a maintenance notice, then I would 
probably not activate AUTH. So it becomes:

<connection>
S: 220 hi
C: EHLO me
S: 250 SMTPRETRY
C: NEEDRETRYDATA
S: 621 Server in maintenance, try later wait:600  QUIT NOW
C: QUIT
S: 221 see ya!

Also, if this is one of the purposes, then maybe a service keyword, 
like perhaps ALERT can work do this:

S: 220 hi
C: EHLO me
S: 250 ALERT OIL CHANGE. QUIT NOW and WAIT:600
C: QUIT
S: 221 see ya!

and it can be extended to expose time left info:

S: 220 hi
C: EHLO me
S: 250 ALERT SYSTEM GOING DOWN SOON. TIMELEFT=50 WAIT=600

So now the client can decide if it can send the message(s) in 50 secs 
or hang up and try again in 5 mins. <g>

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sun, 30 Oct 2011 08:34:24 +0000
From: Paul Smith <[email-masked]>
To: Hector Santos <[email-masked]>
CC: Tim Kehres <[email-masked]>, [email-masked], John C Klensin <[email-masked]>
Subject: Re: Putting it all together
Sender: [email-masked]

On 30/10/2011 03:57, Hector Santos wrote:
>
> It seems to me ID2 and ID3 are very different since as I understand 
> there is no enforcement in ID2 where its a critical part for ID3.

I think you're over complicating it.

 From the point of view of the client AND server, the wait:xxx response 
doesn't care whether it's enforced or not. The enforcement is a part of 
the greylisting system, not a part of the protocol enhancement.

The server should not penalise a client which ignores its wait:xxx hints 
(eg it may choose to do so, if it sees 'wait:100000'), so, in fact, the 
'wait:xxx' response is not enforced separately. The enforcement is there 
already with greylisting

Similarly, a server which is down for an hour, implicitly 'enforces' 
attempts to reconnect before that hour is up, but it shouldn't be part 
of a spec that it should do so, it just happens. All the 'wait:xxx' 
enhancement is, is a bit of helpful info for the client, if it wants to 
use it.

Date: Sun, 30 Oct 2011 08:26:12 +0000
From: Paul Smith <[email-masked]>
To: John C Klensin <[email-masked]>
CC: "Carl S. Gutekunst" <[email-masked]>, "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

On 29/10/2011 21:10, John C Klensin wrote:
> would be perfectly adequate to permit support for, e.g., 6yz or
> x7z codes.
>
> Whether it is a good idea or not is another question, but it
> seems to me that the protocol design is adequate.
Of course, this assumes you don't want the 'new' response before the 
negotiation.

In the case of a 'wait' extension, it would be 'nicer' to have

<connection>
421 Server in maintenance, try later wait:600

rather than

<connection>
220 hi
EHLO me
220-AUTH LOGIN
220 SMTPRETRY
AUTH <stuff>
220 OK (which may need to be faked if the real auth database is not 
available)
EHLO me again
220 SMTPRETRY
NEEDRETRYDATA
621 Server in maintenance, try later wait:600

Date: Sat, 29 Oct 2011 23:57:14 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tim Kehres <[email-masked]>
CC: [email-masked], John C Klensin <[email-masked]>
Subject: Putting it all together
Sender: [email-masked]

Tim Kehres wrote:

>> While it might not help significantly, something more generic
>> would at least slightly and temporarily lower the odds of a
>> different problem.
> 
> A generic solution would also mean that we don't have to design the same 
> solution multiple times for different applications.   A few cases were 
> brought up earlier such as if a site needs to be unavailable for a 
> pre-determined period of time (site maintenance, etc).   I'm sure with 
> some thought we'd be able to come up with more, and as time goes on, 
> additional applications will probably also find this of use.

Here is what I think might be ideal:

   ID1 - Base definition protocol for a standard structured format
         for SMTP client/server Text Responses.

   ID2 - SMTP Extension traffic control extension defining the
         wait=time tag (based on ID1) and its extended multiple
         usages.

   ID3 - SMTP Extension for Greylisting, official IETF formulation
         of the framework, its implementation and its retry=
         and expire= tags (based on ID1). I have no problem of
         renaming retry= to wait= tag, and if it makes sense to
         reference ID2 for Greylisting usage of blocking delays.

It seems to me ID2 and ID3 are very different since as I understand 
there is no enforcement in ID2 where its a critical part for ID3. 
Please I am adding the optional "expire=time-expire" tag for those 
that need it.  ID3 is not the place for adding the suggestions for 
loading limiting, server limits. That would idea I think under a "SMTP 
Traffic Control" ID2.

Overall, with the different ideas discussed and suggestions for 
structured text information, I think it would ideal to begin a ID1 
that defines a base protocol formatting of new structured text 
information for all current considerations and the future.   I think 
this can have a very positive value to improve the dynamic SMTP 
client/server communications in an optional, backward compatible and 
very flexible and somewhat unlimited way.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sat, 29 Oct 2011 23:27:38 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

Robert A. Rosenberg wrote:
> 
> At 14:41 +0200 on 10/29/2011, Peter J. Holzer wrote about Re: 
> draft-atkins-smtp-traffic-control:
> 
>> If a 4xx wait: was received as a reply to a RCPT TO command, try to
>> resend this message to this specific recipient after the specified time.
> 
> OTOH: If this is a multi-recipient message, keep trying until you run 
> out of recipients (remembering which ones were rejected). This way if 
> one or wore were accepted these recipients will immediately receive the 
> message and it will then be requeued for a later attempt with a trimmed 
> recipient list.

Lets keep in mind that mix RCPT TO negative/positive reply codes works 
only with automated server-based/router MTAs.

With MUAs, its very possible depending on the brand, it will stop once 
the first 4yz/5yz is seen and will popup an error box indicating the 
session was not completed.

Old school businesses stuck in their ways still use an MUA as their 
"mailing list" mail sender.  I tried many times to convince one 
particular customer to install the list server they have in their CD 
but they can't move away from their old integrated marketing tools 
long paid for, and even after I got a feel in how these tools are used 
to seek an easy migration, they seem interested but its still "moving 
their cheese."

The main situation here is the need to disable the feedback to RCPT TO 
filters the automated tracking and blocking of bad emails and domains 
that never make it. When the outbound mailer begins, it finds th bad 
emails in the block list and doesn't proceed anyway.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sat, 29 Oct 2011 22:20:05 -0400
To:  SMTP Interest Group <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

At 14:41 +0200 on 10/29/2011, Peter J. Holzer wrote about Re: 
draft-atkins-smtp-traffic-control:

>If a 4xx wait: was received as a reply to a RCPT TO command, try to
>resend this message to this specific recipient after the specified time.

OTOH: If this is a multi-recipient message, keep trying until you run 
out of recipients (remembering which ones were rejected). This way if 
one or wore were accepted these recipients will immediately receive 
the message and it will then be requeued for a later attempt with a 
trimmed recipient list.

Date: Sat, 29 Oct 2011 22:04:13 -0400
To: "[email-masked]" <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

At 19:26 +0200 on 10/29/2011, Peter J. Holzer wrote about Re: SMTP 
traffic control:

>On 2011-10-24 15:24:49 +0100, Paul Smith wrote:
>>  (Of course we have bigger 'pain points' - the '10 errors and we'll drop�
>>  you' is the worst:
>>  rcpt to:<[email-masked]>
>>  4yz invalid recipient
>>  rcpt to:<[email-masked]>
>>  4yz invalid recipient
>>  ... 8 more times
>>  4yz too many syntax errors, connection dropped
>
>"Invalid recipient" should be a 5xx error, or do you expect that these
>addresses will be created within the next few days?
>
>
>>  (now a message in the outbound queue which will never go)
>>  but that's off topic)
>
>If "invalid recipient" is a 5xx error, the sender will omit the already
>rejected recipients on the next delivery attempt. Eventually all invalid
>recipients will be removed and - if any valid recipients are left - the
>message will be delivered. (And frankly, anybody who sends a message to
>tens of non-existent addresses deserves a bit of pain)

So long as at least one valid recipient has been submitted before 
that 10 (or whatever) bad address limit has been reached AND the 
rejects are done with 5XXs not 4YZs, why not just keep accepting the 
RCPTs? This way the message will be accepted and forwarded instead of 
the sender needing to remove the 10 rejected addresses and trying 
again (possibly getting rejected again for another 10 bad addresses). 
A system that tunes its rejection threshold due to having been 
supplied with valid addresses would be more responsive IMO than one 
that sends a "Go Away, clean up your list, and try again" blow-off 
reply. You can have a message with 25 good addresses get blown-off if 
10 addresses are bad in a "Drop Dead after 10 bad addresses" 
implementation when if you let all of the RCPTs be accepted and 
responded to the message will be delivered to the good addresses when 
originally submitted.

If this is a mailing list message, I can see major delays occurring 
when some of the subscribers from a domain go away but do not 
unsubscribe so this type of tuning can improve acceptance and thus 
delivery of messages with a mixture of good and bad addresses.

Date: Sat, 29 Oct 2011 21:28:29 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Sender: [email-masked]

Carl S. Gutekunst wrote:

> I'd like to think that it wouldn't matter; an accumulation at a single 
> MTA should just be the trivial case, no?

I think it depends on the MTA queuing methods, its sorting, groupings, 
limits set to not overload sites, throughput desires/needs.  But one 
of the minor points I guess is sending backed up mail to your site 
still needs to be aligned with the normal rest of the loads to other 
sites (i.e. should it be priorities). Queue residence time (age) to 
factors in, at least for our logic.

Its also the unknowns though on how your site will limit the "storm", 
even an intentionally small 100 amount example maybe good enough to 
trigger it at many sites.

For the period when your are down, I wonder how keeping a server 
active just to issue a 421 may be less helpful than just not listening 
for connections. This is not greylisting and our retry delays are 
higher for socket connection timeouts.  Though I can see where a 421 
with a wait=time will have a similar benefit and without the 
connection delay. That can a plus.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sat, 29 Oct 2011 21:24:40 -0400
From: John C Klensin <[email-masked]>
To: Paul Smith <[email-masked]>, [email-masked]
Subject: Re: SMTP traffic control
Sender: [email-masked]

--On Saturday, October 29, 2011 23:37 +0100 Paul Smith
<[email-masked]> wrote:

>> "Invalid recipient" should be a 5xx error, or do you expect
>> that these addresses will be created within the next few days?
> 
> Oh, I agree, but we *often* get ISP smarthost mail servers
> giving a 4yz error for 'invalid recipient'. We have to deal
> with this as it is our software which is failing to send the
> message, and is retrying - the user sees 'invalid recipient'
> and thinks our software shouldn't retry, so we have to try to
> explain SMTP result codes to them...
> 
> (I suspect the server is giving a 4yz because it is doing
> call-forward testing whether the remote recipient exists, and
> gets no response either from a DNS server or mail server for
> that address)

And _that_ is a case where, IMO, an extension in which the
server announced that it did call-forward testing (or other
testing that might yield uncertain results) and therefore might
return one or more reply codes with specialized meanings would
be worth exploring.  I haven't thought much about feasibility
(and won't unless someone were serious about the idea), but the
notion of a server being able to response with a reply that
unambiguously meant "can't verify, will keep trying, check back
in a few hours" feels really elegant to me (especially since I
don't think call-forward testing for recipient addresses is
rational unless the relay maintains a cache of valid addresses).
YMMD.

   john

Date: Sat, 29 Oct 2011 17:39:22 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: Per-session SMTP extensions
Sender: [email-masked]

John C Klensin wrote:
> (this is drifting off-topic enough that I changed the subject line)
>   

Thank you.

> Now, if what you really want as feature negotiation is a verb
> that would supply a client capabilities list to parallel the
> server capabilities list in EHLO, with some set of rules about,
> e.g., picking the intersection, it is certainly possible under
> the model.  You would get pushback from some of us on the
> grounds of complexity and additional complications in
> multiple-relay environments (read the introduction to RFC 1425),
> but that is another matter.

Well, to turn my own question back around at me:

It what ways does a single verb for announcing all the extensions that 
the client wants to use differ from using individual verbs for each 
extension?

Answer: it's a lot more complex, and a lot less flexible; the error 
handling alone boggles the mind.

Never mind, bad idea. Thanks John.

<csg>

From: Russ Allbery <[email-masked]>
To: "ietf-smtp\@imc.org" <[email-masked]>
Subject: Re: SMTP traffic control
Organization: The Eyrie
Date: Sat, 29 Oct 2011 17:05:31 -0700
Sender: [email-masked]

Paul Smith <[email-masked]> writes:

> (I suspect the server is giving a 4yz because it is doing call-forward
> testing whether the remote recipient exists, and gets no response either
> from a DNS server or mail server for that address)

I believe Postfix returns those sorts of 4yx errors when configured to use
LDAP-based alias lookup and the LDAP server is not responding.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Sat, 29 Oct 2011 19:51:16 -0400
From: John C Klensin <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>
cc: [email-masked]
Subject: Per-session SMTP extensions (was: Re: 4yz Temporary Rejections is part of the SMTP Protocol)
Sender: [email-masked]

(this is drifting off-topic enough that I changed the subject
line)

--On Saturday, October 29, 2011 16:21 -0700 "Carl S. Gutekunst"
<[email-masked]> wrote:

> John C Klensin wrote:
>>> Before we try anything like that, we need actual feature
>>> negotiation (not just announcements), which ESMTP lacks.   
>> 
>> Without debating where "actual feature negotiation" starts, it
>> seems to me that 
>> 
>> 	 -- Server announces that it can supply extra reply code
>> 	information for traffic flow control
>> 	
>> 	 -- Client announces, via an extension argument to MAIL
>> 	or RCPT (or, in principle, via a new verb) that is it
>> 	willing to accept such codes.
>> 	
>> 	 -- Server uses the code iff both announcements have
>> 	occurred.
>> 
>> would be perfectly adequate to permit support for, e.g., 6yz
>> or x7z codes.
>>   
> 
> Agreed. In referring to "feature negotiation," I was actually
> thinking of a new verb, since the extension would be enabled
> for the entire session, not just one message; and I'm already
> feeling some pain over MAIL FROM parsing. (Don't ask.) What
> matters most though, IMHO, is a precedent for the next
> extension that needs the same "Server Has"/"Client Wants"
> interchange. That's actually something I've wished for for a
> long time, and I suspect most of us have implemented privately
> many times over.

In reverse order... most of the extensions so far are "server
offers"/"client wants".   There are exceptions, such as
ENHANCEDSTATUSCODES and PIPELINING (note that the former works
without a client request precisely because SMTP doesn't specify
syntax for the portion of replies that follow the three-digit
code).  But the original extension design specifically provided
for "eerver offers"/"client sends new verb" precisely because we
anticipated situations in which (i) adding [more] arguments to
existing verbs would be problematic (e.g., BDAT and ETRN) and
(ii) in which the option would be effective per-session rather
than per-mail-transaction. Precedent for the latter exists
already in e.g., AUTH and its friends.  So, other than a large
collection of non-security examples, I'm not sure what you need
in terms of either mechanisms or precedents that you don't have
already.

Now, if what you really want as feature negotiation is a verb
that would supply a client capabilities list to parallel the
server capabilities list in EHLO, with some set of rules about,
e.g., picking the intersection, it is certainly possible under
the model.  You would get pushback from some of us on the
grounds of complexity and additional complications in
multiple-relay environments (read the introduction to RFC 1425),
but that is another matter.

best,
    john

Date: Sat, 29 Oct 2011 16:39:03 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Sender: [email-masked]

Hector Santos wrote:
>> Suppose I have an SMTP receiver that's been down for a small number 
>> of hours, so I'm still within the range where clients are retrying 
>> pretty aggressively. When the receiver comes back up, it gets swamped 
>> with connections, a situation was used to call a "deferral storm."
>
> Lets keep in mind not only server storms but also individual MTA's 
> storm (accumulated pressure points) created by remote down times 
> and/or 4yz failures with unknown blocking times.  Simple example, on a 
> normal basis, there are 100 messages from a site to send mail to your 
> system spread out through the day.  Now you are down, and there is 
> accumulation to send mail to your site. 

I'd like to think that it wouldn't matter; an accumulation at a single 
MTA should just be the trivial case, no?

A common issue among all the SMTP servers with which I've worked 
recently is that they have very detailed gages and knobs for the 
server's overall health, but very little is locally accessible 
per-client. Even the server's IP reputation and greylist databases are 
dirt simple; just tables of yea/nay and timestamps, respectively. It 
would be easy to craft an SMTP response that advised cooperating 
MTA-class clients when to come back based on overall load. But to craft 
different responses to different clients based on their historical 
patterns would be a much larger project.

Also, it's useful to scale this properly; for my current products, an 
expected storm might be 1,000 to 10,000 clients with anything from 1 to 
100 messages each. Obviously a big installation could go 10x that; I 
went through one of those and it was not fun.

<csg>

Date: Sat, 29 Oct 2011 16:21:29 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: John C Klensin <[email-masked]>
CC: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------020804090208020504050306
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

John C Klensin wrote:
>> Before we try anything like that, we need actual feature
>> negotiation (not just announcements), which ESMTP lacks.
>>     
>
> Without debating where "actual feature negotiation" starts, it
> seems to me that 
>
> 	 -- Server announces that it can supply extra reply code
> 	information for traffic flow control
> 	
> 	 -- Client announces, via an extension argument to MAIL
> 	or RCPT (or, in principle, via a new verb) that is it
> 	willing to accept such codes.
> 	
> 	 -- Server uses the code iff both announcements have
> 	occurred.
>
> would be perfectly adequate to permit support for, e.g., 6yz or
> x7z codes.
>   

Agreed. In referring to "feature negotiation," I was actually thinking 
of a new verb, since the extension would be enabled for the entire 
session, not just one message; and I'm already feeling some pain over 
MAIL FROM parsing. (Don't ask.) What matters most though, IMHO, is a 
precedent for the next extension that needs the same "Server 
Has"/"Client Wants" interchange. That's actually something I've wished 
for for a long time, and I suspect most of us have implemented privately 
many times over.

<csg>

--------------020804090208020504050306
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
John C Klensin wrote:<br>
<blockquote cite="mid:[email-masked]" type="cite">
  <blockquote type="cite">
    <pre wrap="">Before we try anything like that, we need actual feature
negotiation (not just announcements), which ESMTP lacks.
    </pre>
  </blockquote>
  <pre wrap=""><!---->
Without debating where "actual feature negotiation" starts, it
seems to me that 

	 -- Server announces that it can supply extra reply code
	information for traffic flow control
	
	 -- Client announces, via an extension argument to MAIL
	or RCPT (or, in principle, via a new verb) that is it
	willing to accept such codes.
	
	 -- Server uses the code iff both announcements have
	occurred.

would be perfectly adequate to permit support for, e.g., 6yz or
x7z codes.
  </pre>
</blockquote>
<br>
Agreed. In referring to "feature negotiation," I was actually thinking
of a new verb, since the extension would be enabled for the entire
session, not just one message; and I'm already feeling some pain over
MAIL FROM parsing. (Don't ask.) What matters most though, IMHO, is a
precedent for the next extension that needs the same "Server
Has"/"Client Wants" interchange. That's actually something I've wished
for for a long time, and I suspect most of us have implemented
privately many times over.<br>
<br>
&lt;csg&gt;<br>
</body>
</html>

--------------020804090208020504050306--

Date: Sat, 29 Oct 2011 23:37:51 +0100
From: Paul Smith <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 29/10/2011 18:26, Peter J. Holzer wrote:
> On 2011-10-24 15:24:49 +0100, Paul Smith wrote:
>> (Of course we have bigger 'pain points' - the '10 errors and we'll drop
>> you' is the worst:
>> rcpt to:<[email-masked]>
>> 4yz invalid recipient
>> rcpt to:<[email-masked]>
>> 4yz invalid recipient
>> ... 8 more times
>> 4yz too many syntax errors, connection dropped
> "Invalid recipient" should be a 5xx error, or do you expect that these
> addresses will be created within the next few days?

Oh, I agree, but we *often* get ISP smarthost mail servers giving a 4yz 
error for 'invalid recipient'. We have to deal with this as it is our 
software which is failing to send the message, and is retrying - the 
user sees 'invalid recipient' and thinks our software shouldn't retry, 
so we have to try to explain SMTP result codes to them...

(I suspect the server is giving a 4yz because it is doing call-forward 
testing whether the remote recipient exists, and gets no response either 
from a DNS server or mail server for that address)

>> (now a message in the outbound queue which will never go)
>> but that's off topic)
> If "invalid recipient" is a 5xx error, the sender will omit the already
> rejected recipients on the next delivery attempt. Eventually all invalid
> recipients will be removed and - if any valid recipients are left - the
> message will be delivered. (And frankly, anybody who sends a message to
> tens of non-existent addresses deserves a bit of pain)

Yes, if 'invalid recipient' was a 5yz error, all would work great - but 
when the onward server incorrectly uses 4yz, it doesn't - and the 'too 
many syntax errors' connection drop (when it isn't actually syntax 
errors at all) just makes the problem much worse.

I agree that if they send to non-existent addresses they deserve some 
problems, but these are usually collected addresses for contacts which 
no longer exist, and it would sort itself out, if either the onward 
server gave a 5yz error, or didn't drop connections after treating "4yz 
recipient doesn't exist" errors as 'syntax errors'

We tend to tell the users to contact their ISP and tell them to sort 
their ******* mail server out (to which the user usually says,they'd 
love to, but don't want to spend 2 hours in a phone queue waiting to 
speak to someone who doesn't speak English and will just say 'there are 
no problems at our end'). OTOH, it sometimes gives us the option to 
resell our own SMTP relay service... ;-)

Date: Sat, 29 Oct 2011 17:34:04 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

John C Klensin wrote:
> 
> Without debating where "actual feature negotiation" starts, it
> seems to me that 
> 
> 	 -- Server announces that it can supply extra reply code
> 	information for traffic flow control
> 	
> 	 -- Client announces, via an extension argument to MAIL
> 	or RCPT (or, in principle, via a new verb) that is it
> 	willing to accept such codes.
> 	
> 	 -- Server uses the code iff both announcements have
> 	occurred.
> 
> would be perfectly adequate to permit support for, e.g., 6yz or
> x7z codes.
> 
> Whether it is a good idea or not is another question, but it
> seems to me that the protocol design is adequate.
> 
> Of course, with that model, as with any other negotiation (or
> offer-accept) model, if the client doesn't announce that it is
> interested in whatever the server is offering, the server should
> almost certainly use standard codes and assume that the client
> won't pay any attention to anything special it is offering.

Two passing related thoughts regarding backward compatibility:

1) Support for SMTP clients who issue HELO for whatever reason
    including the idea that it may be EHLO capable and in violation of 
SMTP
    choose to use HELO for a simplistic session transport.

    Server still allowed to add its structure text response
    to 4yx for HELO clients.

2) Post SMTP session logging or the final reply code and text is
    is also passed, but presently only interested in reply code
    for any subsequence processing, including rescheduling
    (i.e. 4yz - reschedule, 5yz - bounce/delete, 2yz - delete)

Comment?

I guess my question is, does a SMTP extension need to provide  service 
keyword?  Someone offlist
indicated that SMTP extensions requires one.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sat, 29 Oct 2011 16:10:37 -0400
From: John C Klensin <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>, "Murray S. Kucherawy" <[email-masked]>
cc: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

--On Friday, October 28, 2011 23:51 -0700 "Carl S. Gutekunst"
<[email-masked]> wrote:

>...
>> John suggested a new reply code series, but that scares me a
>> bit; what would a staunch legacy implementation be likely to
>> do with that?
>>   
> 
> It would infer that it was talking to something other than an
> SMTP server, and drop the connection as rapidly as possible to
> avoid doing further damage.
> 
> Before we try anything like that, we need actual feature
> negotiation (not just announcements), which ESMTP lacks.

Without debating where "actual feature negotiation" starts, it
seems to me that 

	 -- Server announces that it can supply extra reply code
	information for traffic flow control
	
	 -- Client announces, via an extension argument to MAIL
	or RCPT (or, in principle, via a new verb) that is it
	willing to accept such codes.
	
	 -- Server uses the code iff both announcements have
	occurred.

would be perfectly adequate to permit support for, e.g., 6yz or
x7z codes.

Whether it is a good idea or not is another question, but it
seems to me that the protocol design is adequate.

Of course, with that model, as with any other negotiation (or
offer-accept) model, if the client doesn't announce that it is
interested in whatever the server is offering, the server should
almost certainly use standard codes and assume that the client
won't pay any attention to anything special it is offering.

   john

Date: Sat, 29 Oct 2011 15:55:09 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Proposal: SMTP Ext - Base protocol for structured responses
Sender: [email-masked]

SM wrote:

>> That's a pretty strong argument in favour of creating something new 
>> rather than overloading something old.  But that too runs headlong 
>> into likely interoperability concerns.  There's a bit of induced 
>> paralysis there that needs to be overcome.
> 
> Yes.
> 
> I would say that it is more about adoption than interoperability.  There 
> is also "domain of applicability" to consider.  It can help in 
> addressing interoperability concerns or even protocol "violations".
> 
>> Someone else suggested nothing parses SMTP replies beyond the three 
>> digits.  That's not true in ESMTP; the enhanced status codes (if used) 
>> and the reply to EHLO itself are structured.  It's not true in pure 
>> SMTP either, where I think some of the replies are specified beyond 
>> the three digits (the 220 greeting, the reply to HELO, and the 221 
>> message, as I recall).
> 
> The keyword is "structured text".

It seems to me that once this door is open (wider) to begin developed 
new SMTP methods, that it may be a good idea to create a base protocol 
that will define a particular text based data structure all new 
methods can use consistently.  This will allow a SMTP client to single 
source its parsing code for all applications.

Just winging it, but the I-D

May start with the ABNF

    *(tag "=" value [ ";" / SP ])

Even a consideration to use JSON syntax where solid parsers already 
exist in all major languages.

With rules such as where it can places in the text string:

     - end of the line only?
     - Anywhere, but after any Extended Codes?
     - Last line for multiple line responses?

This can really help in other areas too:

     - offer a "url=" tag for references,
     - Some already mention the idea of exposing server info/limits 
and a MX= hint

and without a doubt open the door for new great SMTP client/server 
negotiation ideas using a structured text response syntax.

Don't wish to suggest more work for Murray, but it sounds like to 
something Murray can quickly create with his experience creating other 
base format RFC protocols.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Subject: Re: draft-atkins-smtp-traffic-control
From: Steve Atkins <[email-masked]>
Date: Sat, 29 Oct 2011 11:22:27 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 29, 2011, at 10:54 AM, Peter J. Holzer wrote:

> On 2011-10-28 14:17:56 -0700, Steve Atkins wrote:
>> Hence https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/
> 
> |  This specification defines a new extended response that SMTP server
> |  can include in the text section of any SMTP response - separated by
> |  white space from the rest of the text section.
> 
> So the wait: response can occur anywhere in the text?

Yes.

> I'm a bit worried
> about the client accidentally interpreting text which just happens to
> look like such a response

Me too. But only a little.

> ("450 mailbox of user wait:34567 currently
> unavailable").

> Especially if the syntax keyword:value is picked up by
> other extensions.
> 
> Some ways to reduce the probability of confusion:
> 
> * Specify a fixed position (start or end) for the wait: response

It can't go directly at the beginning, as that'll clash with the
(fairly common) use of unannounced enhanced status codes. 
It could go directly after those, if they're there, but that still requires
quite a bit of parsing, and it leads to the ugly possibility of multiple
extensions,all believing they deserve to be at a particular point in
the response

Putting it right at the end is a bit problematic, as it's less likely to
be correctly logged anywhere (especially in the case of multii-line
responses) and it has the same issues of demanding an ordering
that putting it at a specific point near the beginning does.

> * Enclose it in some kind of delimiters.

Yes. This seems a reasonable way to make it less likely still
that the token would appear accidentally. Square brackets are
what existing implementations that add ad-hoc, machine-readable
data to ESMTP responses tend to use.

"421 [wait:42]" maybe.

> 
> * Announce the use of wait: responses in the EHLO response.

I think that just announcing them might fall between two stools. It
doesn't buy much in terms of robustness and it increases the
server implementation cost as compared to just adding a string to
existing 4xx responses.

OTOH a full ESMTP extension that requires both the announcement
in the EHLO response, a request by the client to use it (as a MAIL
FROM parameter, maybe?) and a specific 4xx response code, with
a tightly defined syntax is a very elegant option - but I think the
deployment costs of that would likely exceed the benefits.

Cheers,
  Steve

Date: Sat, 29 Oct 2011 14:21:22 -0400
From: John C Klensin <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, [email-masked]
Subject: Re: SMTP traffic control
Sender: [email-masked]

--On Saturday, October 29, 2011 14:47 +0200 Arnt Gulbrandsen
<[email-masked]> wrote:

> 
> On 10/29/2011 09:44 AM, Peter J. Holzer wrote:
>> How high is the cost of of publishing an experimental RFC?
>> Somebody has to write it, a few people have to discuss it on
>> a mailing list, and finally an editor has to do a bit of QA
>> and publish it. What do I miss?
> 
> About two thirds.
> 
> There's a document shepherd and a draft writeup (with which
> the author helps, but it's real work even so), a responsible
> AD, last call, IESG discussion including a phone meeting, IANA
> vetting/feedback, and the editor does a little more than you
> appear to think.
> 
> The potential shepherds typically have too much to do, the
> responsible AD and other IESG members much too much. Don't
> know about IANA these days.
> 
> There's also a minor money problem. The RFC editor is paid by
> the IETF, and the IETF is anything but rich.

And still probably an underestimate.  Perhaps a better question
might have been "how much less does it cost to get an
experimental RFC published as compared to a standards-track
one?"   The answer is "little or nothing".   The IESG seems to
take every document it has to process seriously and, while I
sometimes question priorities (especially about nit-picking that
could probably be left to the RFF Editor), it is hard to
complain about their being serious and diligent.

   john

Date: Sat, 29 Oct 2011 19:54:50 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--ZaW/dtY/7oMe/vLp
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 14:17:56 -0700, Steve Atkins wrote:
> Hence https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/

|  This specification defines a new extended response that SMTP server
|  can include in the text section of any SMTP response - separated by
|  white space from the rest of the text section.

So the wait: response can occur anywhere in the text? I'm a bit worried
about the client accidentally interpreting text which just happens to
look like such a response ("450 mailbox of user wait:34567 currently
unavailable"). Especially if the syntax keyword:value is picked up by
other extensions.

Some ways to reduce the probability of confusion:

* Specify a fixed position (start or end) for the wait: response

* Enclose it in some kind of delimiters.

* Announce the use of wait: responses in the EHLO response.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--ZaW/dtY/7oMe/vLp
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--ZaW/dtY/7oMe/vLp--

Date: Sat, 29 Oct 2011 19:38:23 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--6tPipYVl+OcoAvSh
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 14:17:56 -0700, Steve Atkins wrote:
> Hence
> https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/
> It's a very lightweight approach that allows a server to tell a
> cooperating client how long it would like the client to wait before
> retrying.

It also allows a server to tell a client how long to wait when it is not
retrying (i.e. in a 2xx or 5xx response). I am sceptical about this, and
think that the wait: respnse should be limited to 4xx replies.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--6tPipYVl+OcoAvSh
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--6tPipYVl+OcoAvSh--

Date: Sat, 29 Oct 2011 19:32:31 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--/DkgDsqJFFamjKX0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-29 10:30:03 -0400, Hector Santos wrote:
> Peter J. Holzer wrote:
>> On 2011-10-28 14:17:56 -0700, Steve Atkins wrote:
>
>> A somewhat simpler strategy could be to always apply the delay to the
>> message.
>
> That is what we do at the DATA level and once a 451 greylist is issued,=
=20
> no further transactions are allowed.

You are at the wrong end of the connection. The paragraph I quoted and
replied to are about what the client should do when it receives a
"4xx wait:" response.=20

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--/DkgDsqJFFamjKX0
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--/DkgDsqJFFamjKX0--

Date: Sat, 29 Oct 2011 19:26:52 +0200
From: "Peter J. Holzer" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Mail-Followup-To: "[email-masked]" <[email-masked]>
Sender: [email-masked]

--hXth9cGL35Nvpk4x
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-24 15:24:49 +0100, Paul Smith wrote:
> (Of course we have bigger 'pain points' - the '10 errors and we'll drop =
=20
> you' is the worst:
> rcpt to:<[email-masked]>
> 4yz invalid recipient
> rcpt to:<[email-masked]>
> 4yz invalid recipient
> ... 8 more times
> 4yz too many syntax errors, connection dropped

"Invalid recipient" should be a 5xx error, or do you expect that these
addresses will be created within the next few days?

> (now a message in the outbound queue which will never go)
> but that's off topic)

If "invalid recipient" is a 5xx error, the sender will omit the already
rejected recipients on the next delivery attempt. Eventually all invalid
recipients will be removed and - if any valid recipients are left - the
message will be delivered. (And frankly, anybody who sends a message to
tens of non-existent addresses deserves a bit of pain)

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--hXth9cGL35Nvpk4x
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--hXth9cGL35Nvpk4x--

Date: Sat, 29 Oct 2011 19:20:03 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--5FetH82qe0Z6y/zI
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 13:49:49 -0700, Carl S. Gutekunst wrote:
>> That's the end-user visible part of the thought behind
>> https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/
>> anyway.
>>  =20
>
> I'd support that proposal. I might even implement it.
>
> Unfortunately, the dominant conversation seems to be around an SMTP
> response that only applies to greylisting.

Probably because excessive delays due to 4xx errors are most visible or
at least most annoying (the server could accept the mail, but it doesn't
want to!) with greylisting.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--5FetH82qe0Z6y/zI
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--5FetH82qe0Z6y/zI--

Date: Sat, 29 Oct 2011 19:17:03 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--BHMq0+oGliEfCAkH
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-29 04:14:45 -0400, Hector Santos wrote:
> Peter J. Holzer wrote:
>> On 2011-10-28 23:33:09 -0400, Hector Santos wrote:
>>> But I am sure there would be a concern with a no enforcement model in=
=20
>>>  the open market since the tendency will be "why bother?" Even within=
=20
>>> the Honor System, good intention clients who feel they must get a=20
>>> message delivered, or simply has no incentive to bother with code=20
>>> changes, will cross their fingers and just retry again. After all,=20
>>> the new please "wait=3D" idea is for the *other guy" and not them.
>>
>> For good guys the reason to bother is that they will usually get the
>> mail through *faster*, because the wait time announced by the server is
>> *less* than their usual retry time.
>
> I agree if there would be a cost awareness factor associated with known=
=20
> policed controls. I don't see that when there isn't any server =20
> enforcement. Remember, in this alternative model where there is no =20
> regulation controls on the "wait=3D" time.  Its an honor system control. =
=20

There is no need to "police" or "enforce" the delay. The server is
currently not ready to accept the mail (because of greylisting, high
load, planned downtime, or whatever) and provides a best guess when it
will be ready. If we assume that the guess is accurate (in case of
greylisting it probably is very accurate, in case of high load not so
much), then a client which retries early will find that the server is
still not ready and has to retry again.=20

Also consider that at least initially almost all clients will ignore the
hint simply because they don't understand it. "Punishing" them for
malicious behaviour is counter-productive.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--BHMq0+oGliEfCAkH
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--BHMq0+oGliEfCAkH--

From: "Rosenwald, Jordan" <[email-masked]>
To: Steve Atkins <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Date: Sat, 29 Oct 2011 15:26:12 +0000
Accept-Language: en-US
Sender: [email-masked]

--_000_4AAE28F96FC24870936AADC066946BDAcablecomcastcom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Yea,  I get the idea,  Steve. I'm saying the execution is...  Doubtful.

----- Reply message -----
From: "Steve Atkins" <[email-masked]>
To: "SMTP Interest Group" <[email-masked]>
Subject: SMTP traffic control
Date: Fri, Oct 28, 2011 4:46 pm

On Oct 28, 2011, at 12:28 PM, Rosenwald, Jordan wrote:

> Perhaps I missed something (this has been a long thread), but I'm complet=
ely missing how this will solve the problem of long, unpredictable delays f=
or users. Everything I've read says these are return codes for server consu=
mption, not to be returned to users.
>
> As best I can tell this proposed idea does nothing for the end user.

Mail that's delivered quickly, in a minute or so, vs mail that sometimes ta=
kes a half hour to deliver makes a difference to the end user. Both due to =
the noticeable delay, and the annoyance of conversations arriving out of or=
der.

Sometimes mailservers want to shed load, or prioritize one source of traffi=
c over another. The only way to do that within SMTP that doesn't require yo=
u to keep a session open is to respond with a 4xx deferral and wait for the=
 client to retry.

An ESMTP client, such as your ISP smarthost, will wait a while before retry=
ing in response to a 4xx deferral. The RFCs say they should wait at least 3=
0 minutes the first time.

If Yahoo!, uh, I mean the receivers MX, only needs to shed load for a short=
 time period, or just wants you to drop your open TCP session and go back t=
o the TCP connect queue so that someone else gets a chance then all they ca=
n do is send that 4xx response and delay that mail delivery by 30 minutes o=
r so.

It would be nice if the server could say "give me a minute to get my stuff =
together and come back", have the client retry in 60 seconds and get the ma=
il accepted for delivery - with a delay of a minute or so, rather than a de=
lay of 30 minutes. The server gets to shed load, the client gets to dump a =
message out of it's queue and the human correspondents see fast, in-order m=
ail delivery rather than slow, out-of-order delivery.

That's the end-user visible part of the thought behind https://datatracker.=
ietf.org/doc/draft-atkins-smtp-traffic-control/ anyway.

Cheers,
  Steve

--_000_4AAE28F96FC24870936AADC066946BDAcablecomcastcom_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; pad=
ding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<style>
<!--
.x_EmailQuote
	{margin-left:1pt;
	padding-left:4pt;
	border-left:#800000 2px solid}
-->
</style>
<div>Yea, &nbsp;I get the idea, &nbsp;Steve. I'm saying the execution is...=
 &nbsp;Doubtful. <br>
<br>
<div id=3D"x_htc_header" style=3D"">----- Reply message -----<br>
From: &quot;Steve Atkins&quot; &lt;[email-masked]&gt;<br>
To: &quot;SMTP Interest Group&quot; &lt;[email-masked]&gt;<br>
Subject: SMTP traffic control<br>
Date: Fri, Oct 28, 2011 4:46 pm<br>
<br>
</div>
<br>
<br>
</div>
<font size=3D"2"><span style=3D"font-size:10pt;">
<div class=3D"PlainText"><br>
<br>
On Oct 28, 2011, at 12:28 PM, Rosenwald, Jordan wrote:<br>
<br>
&gt; Perhaps I missed something (this has been a long thread), but I'm comp=
letely missing how this will solve the problem of long, unpredictable delay=
s for users. Everything I've read says these are return codes for server co=
nsumption, not to be returned to users.
<br>
&gt; <br>
&gt; As best I can tell this proposed idea does nothing for the end user. <=
br>
<br>
<br>
Mail that's delivered quickly, in a minute or so, vs mail that sometimes ta=
kes a half hour to deliver makes a difference to the end user. Both due to =
the noticeable delay, and the annoyance of conversations arriving out of or=
der.<br>
<br>
Sometimes mailservers want to shed load, or prioritize one source of traffi=
c over another. The only way to do that within SMTP that doesn't require yo=
u to keep a session open is to respond with a 4xx deferral and wait for the=
 client to retry.<br>
<br>
An ESMTP client, such as your ISP smarthost, will wait a while before retry=
ing in response to a 4xx deferral. The RFCs say they should wait at least 3=
0 minutes the first time.
<br>
<br>
If Yahoo!, uh, I mean the receivers MX, only needs to shed load for a short=
 time period, or just wants you to drop your open TCP session and go back t=
o the TCP connect queue so that someone else gets a chance then all they ca=
n do is send that 4xx response and
 delay that mail delivery by 30 minutes or so. <br>
<br>
It would be nice if the server could say &quot;give me a minute to get my s=
tuff together and come back&quot;, have the client retry in 60 seconds and =
get the mail accepted for delivery - with a delay of a minute or so, rather=
 than a delay of 30 minutes. The server gets
 to shed load, the client gets to dump a message out of it's queue and the =
human correspondents see fast, in-order mail delivery rather than slow, out=
-of-order delivery.<br>
<br>
That's the end-user visible part of the thought behind <a href=3D"https://d=
atatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/">
https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/</a> any=
way.<br>
<br>
Cheers,<br>
&nbsp; Steve<br>
<br>
<br>
</div>
</span></font>
</body>
</html>

--_000_4AAE28F96FC24870936AADC066946BDAcablecomcastcom_--

Date: Sat, 29 Oct 2011 10:30:03 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

Peter J. Holzer wrote:
> On 2011-10-28 14:17:56 -0700, Steve Atkins wrote:

> A somewhat simpler strategy could be to always apply the delay to the
> message.

That is what we do at the DATA level and once a 451 greylist is 
issued, no further transactions are allowed. In fact, we our small 55 
secs, I had found some spammers will hold the connection and try 
again, thus finding a loophole.  I have working evidence of this 
closing this loophole. See the "451 Transactions disabled due to 
policy" response below:

**************************************************************************
Wildcat! ESMTP Server v6.4.454.1
SMTP log started at Fri, 28 Oct 2011  12:07:31
Connection Time: 20111028 12:07:31  cid: 000069F9 tid: 00000A64
SSL-Enabled=NO No-Quit-Cancel=OFF Receiver-Bin=ON
Client IP: 76.73.19.178:64948 (unknown) Host IP: 208.247.131.9:25
12:07:31.612 S: 220 winserver.com Wildcat! ESMTP Server v6.4.454.1 ready
12:07:32.408 C: EHLO host.germdefenderproducts.com
12:07:32.408 S: 250-winserver.com, Pleased to meet you.
12:07:32.408 S: 250-SIZE 10240000
12:07:32.408 S: 250-8BITMIME
12:07:32.408 S: 250-SUBMITTER
12:07:32.408 S: 250-ETRN
12:07:32.408 S: 250-AUTH CRAM-MD5 LOGIN PLAIN PLAIN-MD5 SHA-1
12:07:32.408 S: 250-AUTH=LOGIN
12:07:32.408 S: 250 HELP
12:07:34.280 C: MAIL FROM:<[email-masked]>
12:07:34.280 S: 250 <[email-masked]>... Sender 
validation pending. Continue.
12:07:35.153 C: RCPT TO:<[email-masked]>
12:07:35.497 ** WCX Process: wcsap ret: -1 (344 msecs)
12:07:35.497 S: 250 <[email-masked]>... Recipient ok
12:07:36.323 C: DATA
12:07:36.323 S: 354 Start mail input; end with <CRLF>.<CRLF>
12:07:38.913 ** end of data received. (bytes: 10103) (3.6 K/s)
12:07:39.007 ** WCX Process: SmtpFilterHookLoader  ret: 0 (94 msecs)
12:07:39.007 S: 451 Greylisted. Please try again later (TempFail) 
retry=00:00:55
12:07:39.849 C: RSET
12:07:39.849 S: 250 Reset State #1
12:07:40.613 C: MAIL FROM:<[email-masked]>
12:07:40.613 S: 451 Transactions disabled due to policy.
12:07:41.378 C: RSET
12:07:41.378 S: 250 Reset State #2
12:07:42.142 ** connection drop - error: 10054 state: tDefault 
lastcmd: RSET
12:07:42.142 ** Completed. Elapsed Time: 10530 msecs

But I also believe there is Connection Sharing/Holding clients as many 
will hold and then try to start a new transactions.  Maybe it would be 
nice to suggest to the client to hangup but we right now instead I'm 
exploring the 5 mins IDLE time drop to 1 minute after the first 
transaction attempt.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sat, 29 Oct 2011 15:04:04 +0200
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: new draft: draft-santos-smtpgrey-01
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--vSsTm1kUtxIHoa7M
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 04:01:55 -0400, Hector Santos wrote:
> Tony Finch wrote:
>> The original motivation for this draft was to work around problems=20
>> caused by silly retry or greylisting configurations. Fixing the silly=20
>> configurations is the right way to deal with he problem.
>
> How do we fix the silly configs?  With suggestions for values?
>
> Ok, I bite, what are your recommendations for:
>
> Server:
>
>    Blocking Time:  ___ secs/mins

Something between ca. 1 minute and ca. 5 minutes to minimize delays for
legitimate senders.

If you want to use greylisting to delay unknown senders long enough for
them to get into a blacklist you probably have to block longer. 1 hour
seems about right for that but I don't have any experience with that.

>    Record (sender recording) Expiration Time:  ____ secs/mins/hours/days

Significantly longer than the 4 hours suggested by Harris. 9 to 24 hours
works well IME.

>
> Client:
>
>    Retry Frequencies
>
>    - initial try  ___ secs/mins

A few minutes. Probably more than 1 minute and less than 10 minutes.

>    - 2nd try      ___ secs/mins
>    - 3rd try      ___ secs/mins
>    - 4th try      ___ secs/mins

Double delay with each unsuccessful attempt until the delay reaches an
upper limit of 1 to 3 hours. Then retry at that rate for 5 to 7 days.

> and I presume the fix means that EVERYONE has to use the same values at=
=20
> all servers and clients?

No, it means that there is a range of values where you can be reasonably
sure that all mails will get through in a reasonable time. If you retry
only once every 24 hours, you have must expect that your mails will
never be accepted by greylisting receivers. Similary, if you set your
timeouts too long or too short at the receiver side, you must expect to
lose mail.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--vSsTm1kUtxIHoa7M
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--vSsTm1kUtxIHoa7M--

Date: Sat, 29 Oct 2011 06:00:54 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 10/29/2011 2:30 AM, Murray S. Kucherawy wrote:
> Please don't take these costs for granted.

Indeed.

About 15 years ago, I tried to estimate the minimum cost of an RFC.  The IETF 
tended to have working groups with a larger core of active participants than 
now, but engineer salaries have gone up.  Don't know whether these balance each 
other out.

Anyhow, the aggregate start-to-finish cost of producting an entry-level IETF 
standard was around US$ 1 million.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Sat, 29 Oct 2011 14:47:34 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 10/29/2011 09:44 AM, Peter J. Holzer wrote:
> How high is the cost of of publishing an experimental RFC? Somebody has
> to write it, a few people have to discuss it on a mailing list, and
> finally an editor has to do a bit of QA and publish it. What do I miss?

About two thirds.

There's a document shepherd and a draft writeup (with which the author 
helps, but it's real work even so), a responsible AD, last call, IESG 
discussion including a phone meeting, IANA vetting/feedback, and the 
editor does a little more than you appear to think.

The potential shepherds typically have too much to do, the responsible 
AD and other IESG members much too much. Don't know about IANA these days.

There's also a minor money problem. The RFC editor is paid by the IETF, 
and the IETF is anything but rich.

Arnt

Date: Sat, 29 Oct 2011 14:41:12 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--Pql/uPZNXIm1JCle
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 14:17:56 -0700, Steve Atkins wrote:
> https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/

|  If a client sees this additional response in any SMTP response from a
|  server then it SHOULD gracefully close the SMTP connection by sending
|  a QUIT command as soon as possible.  It SHOULD NOT make any new
|  connection to the same server for the duration of the delay.  If
|  possible it SHOULD gracefully close any other connections to the same
|  server.  If the implementation of the client makes it difficult to
|  implement some of these behaviours, it can still usefully implement
|  the other requirements.

I think this interacts badly with two use cases I have in mind:

1) Greylisting (yes, I know you don't care much about that, but it's
   what triggered this discussion and people will want to use it for that):

   Assume a site which uses greylisting as originally described by Evan
   Harris: Greylisting on the tupel (sender, rcpt, ip), initial delay of
   a previously unknown triplet: 1 Hour lifetime of triplets that have
   not yet allowed a mail to pass: 4 hours.

   Now you want to send emails to 5 users at this site which you have
   not emailed before.

   Your MTA will open a connection to the MX of the site and get=20
   a "450 wait:3600" response to "RCPT TO:<[email-masked]>".
   It will close the connection, put the message back into
   the queue and not contact the MX again for an hour.

   An hour later it will try to deliver the next message. Not it will
   get a "450 wait:3600" response for user2.=20

   Repeat for users 3, 4, and 5.

   When it finally gets back to user1, 5 hours have passed. Not only is
   this an unacceptably long delay, by now the triplet for this user has
   expired (after 4 hours), so it will get a "450 wait:3600" response
   again, and the pattern repeats. The mails will never be delivered.

2) Too many connections:

    Assume that you have your MTA configured to attempt at most 10
    concurrent connections to an MX, and that you want to send lots of
    mails to an MX which accepts only 5 concurrent connections from a
    single IP address.=20

    So your MTA opens the first 5 connections and starts sending
    messages. When it opens the 6th connection it gets a=20
    "4xx wait:" reply with a small delay (maybe 0, maybe a few seconds).
    This will cause it to abort the already established connections
    (hopefully after finishing the current transaction) and then start
    new connections after the specified delay (which may be
    immediately). This seems wasteful to me.

I think it would be better if the delay applied only to the entity which
triggered it:

If a 4xx wait: reply is received at connection startup (either in the
greeting or as a reply to EHLO), then try to reconnect to that host
after the specified time.

If a 4xx wait: was received as a reply to a MAIL FROM or DATA command,
try to resend this message after the specified time.

If a 4xx wait: was received as a reply to a RCPT TO command, try to
resend this message to this specific recipient after the specified time.

A somewhat simpler strategy could be to always apply the delay to the
message.

	hp

[1] http://projects.puremagic.com/greylisting/whitepaper.html

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--Pql/uPZNXIm1JCle
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--Pql/uPZNXIm1JCle--

Date: Sat, 29 Oct 2011 02:28:49 -0700
To: "Peter J. Holzer" <[email-masked]>
From: SM <[email-masked]>
Subject: Re: SMTP traffic control
Cc: [email-masked]
Sender: [email-masked]

Hi Peter,
At 00:44 29-10-2011, Peter J. Holzer wrote:
>How high is the cost of of publishing an experimental RFC? Somebody has
>to write it, a few people have to discuss it on a mailing list, and
>finally an editor has to do a bit of QA and publish it. What do I miss?

The cost is lower or about the same depending on what the 
specification is about.  The reviews and discussions are expensive as 
you have to find people to do that.  I'll put the question from another angle:

If you want to publish a RFC, provide the following:

  1. Five reviews of the document

  2. One review from each of the other IETF areas

  3. Someone to deal with all the process details

Regards,
-sm 

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Sat, 29 Oct 2011 02:30:07 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter J. Holzer
> Sent: Saturday, October 29, 2011 12:44 AM
> To: [email-masked]
> Subject: Re: SMTP traffic control
> 
> How high is the cost of of publishing an experimental RFC? Somebody has
> to write it, a few people have to discuss it on a mailing list, and
> finally an editor has to do a bit of QA and publish it. What do I miss?

That's one of the most amazing trivializations of the IETF process I've ever seen.  Even a slam-dunk of a document takes months to get through the process, and invariably requires dozens of people's time.

It's true that Experimental is "cheaper" than other status options, but that doesn't mean it's cheap overall.  And draft-santos-smtpgrey is seeking Standards Track status, the most intensive option in terms of requiring demonstrable consensus, layers of evaluation, interoperability considerations, security evaluation, privacy evaluation, specification rigor, etc.

Please don't take these costs for granted.

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Sat, 29 Oct 2011 02:21:30 -0700
Subject: RE: 4yz Temporary Rejections is part of the SMTP Protocol
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: SM [mailto:[email-masked]]
> Sent: Saturday, October 29, 2011 1:59 AM
> To: Murray S. Kucherawy
> Cc: [email-masked]
> Subject: RE: 4yz Temporary Rejections is part of the SMTP Protocol
> 
> >John suggested a new reply code series, but that scares me a bit;
> >what would a staunch legacy implementation be likely to do with that?
> 
> The legacy implementation could ignore it.  In practice, it may cause
> code path issues if there isn't any protocol negotiation.

A well-planned specification will presume the latter.

Date: Sat, 29 Oct 2011 01:59:26 -0700
To: "Murray S. Kucherawy" <[email-masked]>
From: SM <[email-masked]>
Subject: RE: 4yz Temporary Rejections is part of the SMTP Protocol
Cc: [email-masked]
Sender: [email-masked]

Hi Murray,
At 23:27 28-10-2011, Murray S. Kucherawy wrote:
>That's a pretty strong argument in favour of creating something new 
>rather than overloading something old.  But that too runs headlong 
>into likely interoperability concerns.  There's a bit of induced 
>paralysis there that needs to be overcome.

Yes.

I would say that it is more about adoption than 
interoperability.  There is also "domain of applicability" to 
consider.  It can help in addressing interoperability concerns or 
even protocol "violations".

>Someone else suggested nothing parses SMTP replies beyond the three 
>digits.  That's not true in ESMTP; the enhanced status codes (if 
>used) and the reply to EHLO itself are structured.  It's not true in 
>pure SMTP either, where I think some of the replies are specified 
>beyond the three digits (the 220 greeting, the reply to HELO, and 
>the 221 message, as I recall).

The keyword is "structured text".

BTW, you can even get into trouble with assumptions about the three 
digits being what the specification say they should be as some 
implementation diverge in their interpretation.  The careful reader 
will take note of the discussion about that.

>John suggested a new reply code series, but that scares me a bit; 
>what would a staunch legacy implementation be likely to do with that?

The legacy implementation could ignore it.  In practice, it may cause 
code path issues if there isn't any protocol negotiation.

Regards,
-sm 

Date: Sat, 29 Oct 2011 10:04:13 +0100
From: Paul Smith <[email-masked]>
To: Hector Santos <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

On 29/10/2011 01:02, Hector Santos wrote:
> Focus is lost when you attempt to remove greylisting realities from 
> the "temporary rejection time delays" issues.
>
> In other words, if you remove this, your proposal boils down to two 
> reasons to exist:
>
>   #1 Real Server Controls at Load Limiting
>   #2 Wishful Server Controls at Connection Sharing/Holding clients
>
> For #2, you have no control - its wishful and that only way to deal 
> violators is drop the connection.  For #1,  when these server issues a 
> "wait=" time, it will be behaving like a Greylisting server because 
> the specification will now need to define how the server will  enforce 
> the "wait=" time.

Well, for #2, you can say 'I'm too busy now, try again in 15 minutes' - 
it may be 'wishful thinking', but the thing is the server effort here is 
minimal.

The server effort for #1 is much more complex than for #2, so while #2 
may be wishful thinking, for the effort involved it may be worth doing.

#1 may be 'behaving like a greylisting server', but it may not be one - 
eg the server may know it's very busy between 9:00 and 10:00 local time, 
so if it receives a message that it can't handle at 9:45, it may say 
'421 too busy try later - wait:1200', but if another server sends a 
message at 10:30, it will just accept it without requiring a delay. This 
is regardless of where the message is coming from, just based on server 
load or other factors.

Date: Sat, 29 Oct 2011 10:07:15 +0100
From: Paul Smith <[email-masked]>
To: Hector Santos <[email-masked]>
CC: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

On 29/10/2011 09:43, Hector Santos wrote:
>
> Murray S. Kucherawy wrote:
>
>> John suggested a new reply code series, but that scares me a bit;
>>  what would a staunch legacy implementation be likely to do with that?
>
> I seriously looked at something that would fit with the current 
> system, even something like 6yz or somehow leveraging extended status 
> codes.  The common issue was how to get the "time issue" hint in there 
> if text parsing was not part of it.
>
> However, if text parsing was part of it than backward compatibility 
> would suggest the current status code would suffice.

Yes, I can't see a problem with having some simplistic text parsing of a 
4yz response if the server says it may be appropriate.

Date: Sat, 29 Oct 2011 04:43:28 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: "[email-masked]" <[email-masked]>
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
To: [email-masked]
Sender: [email-masked]

Murray S. Kucherawy wrote:

> John suggested a new reply code series, but that scares me a bit;
>  what would a staunch legacy implementation be likely to do with that?

I seriously looked at something that would fit with the current 
system, even something like 6yz or somehow leveraging extended status 
codes.  The common issue was how to get the "time issue" hint in there 
if text parsing was not part of it.

However, if text parsing was part of it than backward compatibility 
would suggest the current status code would suffice.

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: [email-masked]

Date: Sat, 29 Oct 2011 04:14:45 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Sender: [email-masked]

Peter J. Holzer wrote:
> On 2011-10-28 23:33:09 -0400, Hector Santos wrote:
>> But I am sure there would be a concern with a no enforcement model in  
>> the open market since the tendency will be "why bother?" Even within the 
>> Honor System, good intention clients who feel they must get a message 
>> delivered, or simply has no incentive to bother with code changes, will 
>> cross their fingers and just retry again. After all, the new please 
>> "wait=" idea is for the *other guy" and not them.
> 
> For good guys the reason to bother is that they will usually get the
> mail through *faster*, because the wait time announced by the server is
> *less* than their usual retry time.

I agree if there would be a cost awareness factor associated with 
known policed controls. I don't see that when there isn't any server 
enforcement. Remember, in this alternative model where there is no 
regulation controls on the "wait=" time.  Its an honor system control. 
  Additional premature attempts will not be policed..... hmmm, if it 
does, its a greylister :)

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sat, 29 Oct 2011 04:03:44 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Sender: [email-masked]

Carl S. Gutekunst wrote:
> There's another common problem I have, though, that may be more of what 
> Hector was thinking of. Suppose I have an SMTP receiver that's been down 
> for a small number of hours, so I'm still within the range where clients 
> are retrying pretty aggressively. When the receiver comes back up, it 
> gets swamped with connections, a situation was used to call a "deferral 
> storm." In this situation, some number of the clients will get 4xy 
> responses again, while others would be accepted, and the rate of 
> accept/defer will be likely be tied to the connection rate of the 
> clients, whether I intended it to be that way or not. Here, what would 
> be useful is a way to apply controlled back-pressure to the storm.

Lets keep in mind not only server storms but also individual MTA's 
storm (accumulated pressure points) created by remote down times 
and/or 4yz failures with unknown blocking times.  Simple example, on a 
normal basis, there are 100 messages from a site to send mail to your 
system spread out through the day.  Now you are down, and there is 
accumulation to send mail to your site.

How should that be done?

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sat, 29 Oct 2011 09:52:07 +0200
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--ryJZkp9/svQ58syV
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 22:02:48 -0400, Hector Santos wrote:
> John C Klensin wrote:
>>> I considered this and even a special extended status code but
>>> stopped because the key part of the SMTP extension is parsing
>>> the response text, which as far I am aware of, there are no
>>> known SMTP practice to do so (outside of proprietary or
>>> localized) due to the idea we can't rely on it to be
>>> persistent.
>>
>> I'm not sure what you mean.  Certainly the responses to VRFY and
>> EXPN have to be parsed and 5321 is moderately specific about
>> their syntax. ...
>
> Ah yes. Forgot about these and of course, others like AUTH, CHUNKING
> are SMTP extension examples of *structured* response text parsing for =20
> client/server negotiated concepts.

Don't forget EHLO. The client has to parse the response to know which
extensions are available.

And then there are the 251 and 551 replies although I don't know any
client which can use them.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--ryJZkp9/svQ58syV
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--ryJZkp9/svQ58syV--

Date: Sat, 29 Oct 2011 09:44:10 +0200
From: "Peter J. Holzer" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Mail-Followup-To: "[email-masked]" <[email-masked]>
Sender: [email-masked]

--cHMo6Wbp1wrKhbfi
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 23:09:26 -0700, Murray S. Kucherawy wrote:
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]=
] On Behalf Of Peter J. Holzer
> > Sent: Friday, October 28, 2011 12:10 PM
> > To: [email-masked]
> > Subject: Re: SMTP traffic control
> >=20
> > I really don't understand the resistance against the idea. Some servers
> > and some clients will implement it and benefit from it (if they are
> > talking to each other), others won't and for them everything will be as
> > before. How would a standardized way of conveying the same information
> > that some servers already convey informally hurt the mail system?
>=20
> The issue is not one of potential harm, it's a measure of the cost of
> specifying this through the IETF (which is high)

How high is the cost of of publishing an experimental RFC? Somebody has
to write it, a few people have to discuss it on a mailing list, and
finally an editor has to do a bit of QA and publish it. What do I miss?

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--cHMo6Wbp1wrKhbfi
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--cHMo6Wbp1wrKhbfi--

Date: Sat, 29 Oct 2011 09:36:07 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--dkEUBIird37B8yKS
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 23:33:09 -0400, Hector Santos wrote:
> But I am sure there would be a concern with a no enforcement model in =20
> the open market since the tendency will be "why bother?" Even within the=
=20
> Honor System, good intention clients who feel they must get a message=20
> delivered, or simply has no incentive to bother with code changes, will=
=20
> cross their fingers and just retry again. After all, the new please=20
> "wait=3D" idea is for the *other guy" and not them.

For good guys the reason to bother is that they will usually get the
mail through *faster*, because the wait time announced by the server is
*less* than their usual retry time.

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--dkEUBIird37B8yKS
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--dkEUBIird37B8yKS--

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Fri, 28 Oct 2011 23:54:36 -0700
Subject: RE: 4yz Temporary Rejections is part of the SMTP Protocol
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Carl S. Gutekunst [mailto:[email-masked]]
> Sent: Friday, October 28, 2011 11:51 PM
> To: Murray S. Kucherawy
> Cc: [email-masked]
> Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
> 
> Before we try anything like that, we need actual feature negotiation
> (not just announcements), which ESMTP lacks.

The closest thing is ENHANCEDSTATUSCODES, which doesn't seem to have been much of a problem so far.  That's probably the most palatable path.

Date: Fri, 28 Oct 2011 23:51:21 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

Murray S. Kucherawy wrote:
> Someone else suggested nothing parses SMTP replies beyond the three digits.  That's not true in ESMTP; the enhanced status codes (if used) and the reply to EHLO itself are structured.
>   

Not to keep piling on, but RFC 2920 (pipelining) says the server SHOULD 
structure the text responses so that the client can pair up commands and 
responses.

> John suggested a new reply code series, but that scares me a bit; what would a staunch legacy implementation be likely to do with that?
>   

It would infer that it was talking to something other than an SMTP 
server, and drop the connection as rapidly as possible to avoid doing 
further damage.

Before we try anything like that, we need actual feature negotiation 
(not just announcements), which ESMTP lacks.

<csg>

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Fri, 28 Oct 2011 23:27:32 -0700
Subject: RE: 4yz Temporary Rejections is part of the SMTP Protocol
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of SM
> Sent: Thursday, October 27, 2011 6:36 PM
> To: Hector Santos
> Cc: [email-masked]
> Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
> 
>  From RFC 5321:
> 
>    "Unfortunately, variations, creative interpretations, and outright
>     violations of Internet mail protocols do occur; some would suggest
>     that they occur quite frequently."

That's a pretty strong argument in favour of creating something new rather than overloading something old.  But that too runs headlong into likely interoperability concerns.  There's a bit of induced paralysis there that needs to be overcome.

Someone else suggested nothing parses SMTP replies beyond the three digits.  That's not true in ESMTP; the enhanced status codes (if used) and the reply to EHLO itself are structured.  It's not true in pure SMTP either, where I think some of the replies are specified beyond the three digits (the 220 greeting, the reply to HELO, and the 221 message, as I recall).

John suggested a new reply code series, but that scares me a bit; what would a staunch legacy implementation be likely to do with that?

-MSK

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Fri, 28 Oct 2011 23:16:21 -0700
Subject: RE: Server Enforcement of Time Blocks (wait=)
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Steve Atkins
> Sent: Friday, October 28, 2011 7:28 PM
> To: SMTP Interest Group
> Subject: Re: Server Enforcement of Time Blocks (wait=)
> 
> >>> My point is if like Atkins proposal and might implement it, once
> >>> you do, you will be behaving like a Greylisted server.
> >>
> >> That's simply not true. Please stop stating that.
> >
> > Are you saying your proposal offer a server issuing of a "wait=" time
> > with no server tracking and enforcement?
> 
> Yes. It's purely a protocol for the server to communicate to
> cooperative clients. I foresee the primary usage to be not enforced by
> the server, certainly not on a real time basis, though it's something
> that would be quite usable if the server were tracking clients in that
> way.

Moreover, IMHO, a protocol extension that imposes storage requirements of that sort on the server is dead before it gets started.

Any protocol on this topic should present the mechanism for reporting a requested wait time, and maybe a reason, and maybe with some reasonable constraints, and that's it.  Server-side implementation of the logic regarding what it indicates in either of those places is entirely implementation-specific, has nothing to do with interoperability, and thus doesn't belong in an RFC.

-MSK

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Fri, 28 Oct 2011 23:09:26 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter J. Holzer
> Sent: Friday, October 28, 2011 12:10 PM
> To: [email-masked]
> Subject: Re: SMTP traffic control
> 
> I really don't understand the resistance against the idea. Some servers
> and some clients will implement it and benefit from it (if they are
> talking to each other), others won't and for them everything will be as
> before. How would a standardized way of conveying the same information
> that some servers already convey informally hurt the mail system?

The issue is not one of potential harm, it's a measure of the cost of specifying this through the IETF (which is high) versus the potential gain (which, so far, doesn't seem likely to be high).

If people are convinced this is a big win, go off and build it, show how much of an improvement it is and at low cost, and include that in the evidence that this is a worthwhile investment for the IETF.  That would increase the likelihood of success dramatically.

So far, though, it doesn't sound like this addresses a pain point big enough for widespread adoption to be likely, so the cost of processing a standards track RFC would be wasted.  That's the cause of the resistance.

Date: Sat, 29 Oct 2011 00:29:28 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "[email-masked]'" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Peter J. Holzer wrote:

> Half an hour is ok if the reason for the temporary failure is a problem
> which requires human intervention. It is already questionable if the
> reason is that server is too busy (in my experience load spikes usually
> last only a few minutes, but YMMV). 

My data shows that too. They come in random waves with no pattern 
(that I can eyeball measure) when they come.

> It is definitely too long for greylisting: 

Exactly!

> In contrast, consider that case that Hectors server (which uses an
> initial blocking time of 55 seconds, as he wrote) can tell the client
> "try again in 56 seconds" and that the client can do this. Then the
> greylisted mail will get through after 56 seconds (+ probably a bit of
> jitter added by the queue runner), and Hectors users will be happy
> because they get the information they requested on the phone while they
> are still talking instead of half an hour later. It also allows mail
> admins to keep the default retry interval at a relatively high value and
> avoid pounding servers which are already overloaded.

Right. And yet, even with the 55 secs delay, we don't know what the 
customer's mail system MTA 2nd retry time will be. They could be using 
the SMTP recommendations or did they adjust to the greylisting market 
and have a 1-5 minutes for at least the 2nd retry?  We don't know.

> I also see some applications outside of greylisting, too:
> 
> For example, I've occassionally have to return a 4xx error to some
> recipients of a multi-recipient message, because they couldn't be
> processed together in the same transaction. In this case it would be
> nice if I could tell the client that it is ok to immediately retry for
> those recipients.

Interesting.

> Another reason for unexpected mail delays is if you try to open more
> connections than the receiving MX allows. The mails on some connections
> get rejected with a temporary error and go back into the queue (possibly
> several times). They could almost certainly be sent immediately over one
> of the existing connections, but the client doesn't know that. If the
> server could convey that information to the client, a wasteful delay
> could be avoided.

I found it to be rare, but I have logs showing remote servers issuing 
a 421 with greylisting information in the text, including informal 
time hints as oppose to just saying "too many connections."

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 21:21:27 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Sender: [email-masked]

Steve Atkins wrote:
> (Greylisting requires keeping track of remote clients in some way, so as to treat them differently the first time you see mail from them. As a concrete example of a server enforcing delays that's not greylisting consider a mailserver with a database backend, where you need to take the backend down for maintenance for a few minutes. While the database is down, the server may respond to any transaction with a 4xx (or even with a 4xx wait=<number of seconds until the database server comes back online>). Attempted redeliveries before that will be deferred again.)

The specific case I had in mind was an SMTP server that comes up and is 
able to respond to connections long before it's actually able to process 
messages. (The backend process that filter the mail take a long time to 
start.) It would be useful to be able to say when the server will 
actually be available. In that scenario the server is oblivious to 
client behavior.

There's another common problem I have, though, that may be more of what 
Hector was thinking of. Suppose I have an SMTP receiver that's been down 
for a small number of hours, so I'm still within the range where clients 
are retrying pretty aggressively. When the receiver comes back up, it 
gets swamped with connections, a situation was used to call a "deferral 
storm." In this situation, some number of the clients will get 4xy 
responses again, while others would be accepted, and the rate of 
accept/defer will be likely be tied to the connection rate of the 
clients, whether I intended it to be that way or not. Here, what would 
be useful is a way to apply controlled back-pressure to the storm.

It's not "greylisting" as Levine described, but it seems like it fits 
Hector's characterization of a "greylist behavior."

<csg>

Date: Fri, 28 Oct 2011 23:33:09 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: Server Enforcement of Time Blocks (wait=)
Sender: [email-masked]

Steve, thanks for the clarification.

Indeed, if the proposal does not offer a server enforcement 
methodology, then its not going to be modeled on a Greylisting concept 
with its key idea of a blocking time enforcement, hence I can see 
value when the HONOR system is followed.  This is good for private 
networks  or those with working relationships.

But I am sure there would be a concern with a no enforcement model in 
the open market since the tendency will be "why bother?" Even within 
the Honor System, good intention clients who feel they must get a 
message delivered, or simply has no incentive to bother with code 
changes, will cross their fingers and just retry again. After all, the 
new please "wait=" idea is for the *other guy" and not them.

Yes, I understand completely the maintenance and perhaps for loading 
balancing/limiting.

The latter is more complex. GL at 421 is rare but it does exist. In 
general its just to control loads.  But for the specific example of a 
down system, the feasibility of offering a wait probably is lower than 
just having it down for a short fast maintenance period. Most systems 
will have redundancy in place.  But I can see smaller system using 
some TinySMTP program, like many use the popular small footprint 
TinyWeb for these informative "Temporarily Out of Service" single page 
web announcements.  With TinySMTP, it can provide the 421 with the 
"wait=" hint so compliant MTAs don't have waste time trying again. 
And for these scenarios, if a system opts to do this, then they really 
cares any MTA retrying anyway.  Its only going to be done for a very 
short period anyway.

Anyway, I just think the no enforcement approach, to me, provides no 
incentive to put in the time and money to support it.  While I could 
support it as a client, as a server, to spit out this information, it 
has to have some weight behind it. My opinion.

--
HLS

Steve Atkins wrote:
> 
> On Oct 28, 2011, at 7:11 PM, Hector Santos wrote:
> 
>> Steve Atkins wrote:
>>
>>>> My point is if like Atkins proposal and might implement it, once you do, you will be behaving like a Greylisted server.
>>> That's simply not true. Please stop stating that.
>> Are you saying your proposal offer a server issuing of a "wait=" time with no server tracking and enforcement?
> 
> Yes. It's purely a protocol for the server to communicate to cooperative clients. I foresee the primary usage to be not enforced by the server, certainly not on a real time basis, though it's something that would be quite usable if the server were tracking clients in that way.
> 
>> Its not be meant to be a negative and it doesn't help with ongoing attempts to create division.
>>
>> Any proposal that includes a "time delay" for a client to use, and more importantly the server will enforce, is 100% *behaving* (keyword) like a Greylisting Server model.  I can't see how you can avoid it.
> 
> Actually it's not, but that's a different issue. (Greylisting requires keeping track of remote clients in some way, so as to treat them differently the first time you see mail from them. As a concrete example of a server enforcing delays that's not greylisting consider a mailserver with a database backend, where you need to take the backend down for maintenance for a few minutes. While the database is down, the server may respond to any transaction with a 4xx (or even with a 4xx wait=<number of seconds until the database server comes back online>). Attempted redeliveries before that will be deferred again.)
> 
>> The client is doing this blinding.  If it encounters a server with this "wait=" in a 4yz response, its going to have the same MTA software rescheduling code for supporting any other kind a "retry=" or parsing of the informal existing greylisting servers with their time hints.
>>
>> So unless the proposal is not offering server enforcement, it is very much a Greylisting Model.
> 
> It's a traffic management enhancement. It could certainly benefit a greylister, but it's of broader value than that narrow niche.
> 
> Cheers,
>   Steve
> 
> 
> 

-- 
Sincerely

Hector Santos
http://www.santronics.com

Subject: Re: Server Enforcement of Time Blocks (wait=)
From: Steve Atkins <[email-masked]>
Date: Fri, 28 Oct 2011 19:27:36 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 28, 2011, at 7:11 PM, Hector Santos wrote:

> 
> Steve Atkins wrote:
> 
>>> My point is if like Atkins proposal and might implement it, once you do, you will be behaving like a Greylisted server.
>> That's simply not true. Please stop stating that.
> 
> Are you saying your proposal offer a server issuing of a "wait=" time with no server tracking and enforcement?

Yes. It's purely a protocol for the server to communicate to cooperative clients. I foresee the primary usage to be not enforced by the server, certainly not on a real time basis, though it's something that would be quite usable if the server were tracking clients in that way.

> Its not be meant to be a negative and it doesn't help with ongoing attempts to create division.
> 
> Any proposal that includes a "time delay" for a client to use, and more importantly the server will enforce, is 100% *behaving* (keyword) like a Greylisting Server model.  I can't see how you can avoid it.

Actually it's not, but that's a different issue. (Greylisting requires keeping track of remote clients in some way, so as to treat them differently the first time you see mail from them. As a concrete example of a server enforcing delays that's not greylisting consider a mailserver with a database backend, where you need to take the backend down for maintenance for a few minutes. While the database is down, the server may respond to any transaction with a 4xx (or even with a 4xx wait=<number of seconds until the database server comes back online>). Attempted redeliveries before that will be deferred again.)

> The client is doing this blinding.  If it encounters a server with this "wait=" in a 4yz response, its going to have the same MTA software rescheduling code for supporting any other kind a "retry=" or parsing of the informal existing greylisting servers with their time hints.
> 
> So unless the proposal is not offering server enforcement, it is very much a Greylisting Model.

It's a traffic management enhancement. It could certainly benefit a greylister, but it's of broader value than that narrow niche.

Cheers,
  Steve

Date: Fri, 28 Oct 2011 22:11:53 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: SMTP Interest Group <[email-masked]>
Subject: Server Enforcement of Time Blocks (wait=)
To: [email-masked]
Sender: [email-masked]

Steve Atkins wrote:

>> My point is if like Atkins proposal and might implement it, once you do, 
>> you will be behaving like a Greylisted server.
> 
> That's simply not true. Please stop stating that.

Are you saying your proposal offer a server issuing of a "wait=" time 
with no server tracking and enforcement?

Its not be meant to be a negative and it doesn't help with ongoing 
attempts to create division.

Any proposal that includes a "time delay" for a client to use, and 
more importantly the server will enforce, is 100% *behaving* (keyword) 
like a Greylisting Server model.  I can't see how you can avoid it. 
The client is doing this blinding.  If it encounters a server with 
this "wait=" in a 4yz response, its going to have the same MTA 
software rescheduling code for supporting any other kind a "retry=" or 
parsing of the informal existing greylisting servers with their time 
hints.

So unless the proposal is not offering server enforcement, it is very 
much a Greylisting Model.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 22:02:48 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
To: [email-masked]
Sender: [email-masked]

John C Klensin wrote:

>> I considered this and even a special extended status code but
>> stopped because the key part of the SMTP extension is parsing
>> the response text, which as far I am aware of, there are no
>> known SMTP practice to do so (outside of proprietary or
>> localized) due to the idea we can't rely on it to be
>> persistent.
> 
> I'm not sure what you mean.  Certainly the responses to VRFY and
> EXPN have to be parsed and 5321 is moderately specific about
> their syntax. ... The SMTP extension model clearly allows for 
> doing just about anything.  

Ah yes. Forgot about these and of course, others like AUTH, CHUNKING
are SMTP extension examples of *structured* response text parsing for 
client/server negotiated concepts.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 18:18:21 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------010808030108020705020907
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Steve Atkins wrote:
>> My point is if like Atkins proposal and might implement it, once you do, you will be behaving like a Greylisted server.
>>     
>
> That's simply not true. Please stop stating that.
>   

+1

<csg>

--------------010808030108020705020907
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
</head>
<body bgcolor="#ffffff" text="#000000">
Steve Atkins wrote:<br>
<blockquote class="cite"
 id="mid_E3030477-AC4E-4858-82B5-BF3824054D98_blighty_com"
 cite="mid:[email-masked]" type="cite">
  <blockquote class="cite" id="StationeryCiteGenerated_8" type="cite">
    <pre wrap="">My point is if like Atkins proposal and might implement it, once you do, you will be behaving like a Greylisted server.
    </pre>
  </blockquote>
  <pre wrap=""><!---->
That's simply not true. Please stop stating that.
  </pre>
</blockquote>
<br>
+1<br>
<br>
&lt;csg&gt;<br>
</body>
</html>

--------------010808030108020705020907--

Subject: Re: SMTP traffic control
From: Steve Atkins <[email-masked]>
Date: Fri, 28 Oct 2011 18:09:15 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 28, 2011, at 5:42 PM, Hector Santos wrote:

> 
> Carl S. Gutekunst wrote:
>> Hector Santos wrote:
>>>> Unfortunately, the dominant conversation seems to be around an SMTP response that only applies to greylisting. I'd never use that or support it, it's such a teeny-tiny corner case, especially in the B-to-B world where I live.
>>> 
>>> Does this suggests that your B2B market is oblivious to the outside world remote server behaviors employing 4yz temporary rejections?
>> They're highly conscious of 4xx temporary rejections associated with resource exhaustion and down systems. But in 20 years of supporting B-to-B E-mail, I can only remember one issue with someone else's greylisting -- and that was because it was broken.
> 
> Ok, and we should|must all respect that.  Nevertheless, its not the case for all though, nor should it be reason for not considering this.  It just means it doesn't apply to your market area.
> 
> My point is if like Atkins proposal and might implement it, once you do, you will be behaving like a Greylisted server.

That's simply not true. Please stop stating that.

Cheers,
  Steve

Date: Fri, 28 Oct 2011 20:42:34 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Carl S. Gutekunst wrote:
> Hector Santos wrote:
>>> Unfortunately, the dominant conversation seems to be around an SMTP 
>>> response that only applies to greylisting. I'd never use that or 
>>> support it, it's such a teeny-tiny corner case, especially in the 
>>> B-to-B world where I live.
>>
>> Does this suggests that your B2B market is oblivious to the outside 
>> world remote server behaviors employing 4yz temporary rejections?
> 
> They're highly conscious of 4xx temporary rejections associated with 
> resource exhaustion and down systems. But in 20 years of supporting 
> B-to-B E-mail, I can only remember one issue with someone else's 
> greylisting -- and that was because it was broken.

Ok, and we should|must all respect that.  Nevertheless, its not the 
case for all though, nor should it be reason for not considering this. 
  It just means it doesn't apply to your market area.

My point is if like Atkins proposal and might implement it, once you 
do, you will be behaving like a Greylisted server.

Most better SMTP servers already deal with DoS related issues and 
employ a concept for loading limiting/restrictions.  What has been 
seen with some real servers 421 responses today, and now also being 
suggested is adding "time delay" hints to this load limiting 421 
rejections because it is also behaving like a Greylisting server.

Again, for lack of a better term, Greylisting, is really two parts:

  - Using 4yz for unknown, non-whitelisted, unrecorded senders 
immediately,
  - Blocking them for a specific, implementation based time length.

If we remove the blocking time from the equation, i.e. it was never 
part of the logic, then I don't believe we would be having this 
discussion.  All 2nd tries will work and delays will be limited to the 
MTA's 2nd attempt time.

But thats not the case. The issue is that is there are blocking times 
and in comes in varying degrees and implementing Atkins proposals 
pushing you into something you indicated you don't wish to support.

Look, I think both proposals are ok.  We do need something that 
documents Greylisting because its more than just issuing a "time 
delay", it about monitoring and tracking and using blocking times.

I could change the SMTPGREY proposal to use "wait=" which would 
reference a more "Generic" proposal on Server/client waiting times.

I would rather that we work together to seek a common solution to all 
the issues and needs, rather than begin starting and playing this 
partisanship games once again.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 17:23:25 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Hector Santos wrote:
>> Unfortunately, the dominant conversation seems to be around an SMTP 
>> response that only applies to greylisting. I'd never use that or 
>> support it, it's such a teeny-tiny corner case, especially in the 
>> B-to-B world where I live.
>
> Does this suggests that your B2B market is oblivious to the outside 
> world remote server behaviors employing 4yz temporary rejections?

They're highly conscious of 4xx temporary rejections associated with 
resource exhaustion and down systems. But in 20 years of supporting 
B-to-B E-mail, I can only remember one issue with someone else's 
greylisting -- and that was because it was broken.

<csg>

Date: Fri, 28 Oct 2011 20:15:25 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Carl S. Gutekunst wrote:
> 
> Steve Atkins wrote:
>> It would be nice if the server could say "give me a minute to get my 
>> stuff together and come back", have the client retry in 60 seconds and 
>> get the mail accepted for delivery - with a delay of a minute or so, 
>> rather than a delay of 30 minutes. The server gets to shed load, the 
>> client gets to dump a message out of it's queue and the human 
>> correspondents see fast, in-order mail delivery rather than slow, 
>> out-of-order delivery.
>>
>> That's the end-user visible part of the thought behind 
>> https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/ 
>> anyway.
>>   
> 
> I'd support that proposal. I might even implement it.
> 
> Unfortunately, the dominant conversation seems to be around an SMTP 
> response that only applies to greylisting. I'd never use that or support 
> it, it's such a teeny-tiny corner case, especially in the B-to-B world 
> where I live.

Does this suggests that your B2B market is oblivious to the outside 
world remote server behaviors employing 4yz temporary rejections?

Keep in mind, whether its that or this proposal, implementing a 
structure "wait=" or "retry=" time delay syntax inherently means two 
things:

Client:

Common to all proposals, leveraging the server suggested delay in retries.

Server:

Common to all proposals, adding logic to monitor, track the original 
4yz restricted client so that it does not violate the wait/retry= 
delay or blocking time.

That my friend, for lack of a better term, is called a "Greylisting" 
methodology in contemporary times.

Even is the issue of putting all this under a different, more 
appeasing IETF quality terminology, you still need to (re)design the 
same "Greylisting" ideas.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 20:02:11 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: SMTP Interest Group <[email-masked]>
Subject: Re: draft-atkins-smtp-traffic-control
Sender: [email-masked]

Steve Atkins wrote:

> Hence https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/  It's a very lightweight approach that allows a server to tell a cooperating client how long it would like the client to wait before retrying. I suspect it has some issues that the IETF process might object to and I'm not sure that extending the idea to 5xx and 2xx responses is as good an idea as it seemed at first, but I think it's a useful concrete example to use as a basis for discussion.

Applying it to 5yz and 2yz unquestionably deals with SMTP protocol 
standard changes, expectations in protocol behavior.

> A server that's doing greylisting could certainly benefit from this, but 
> I'm not particularly interested in either focusing on the implications for 
> greylisting nor expanding it to do other things that greylisters might find 
> useful in this thread.

Focus is lost when you attempt to remove greylisting realities from 
the "temporary rejection time delays" issues.

In other words, if you remove this, your proposal boils down to two 
reasons to exist:

   #1 Real Server Controls at Load Limiting
   #2 Wishful Server Controls at Connection Sharing/Holding clients

For #2, you have no control - its wishful and that only way to deal 
violators is drop the connection.  For #1,  when these server issues a 
"wait=" time, it will be behaving like a Greylisting server because 
the specification will now need to define how the server will  enforce 
the "wait=" time.

How will the server track the original sender to make sure it doesn't 
try again before the wait= time expires?

Just saying, Good luck in trying to get a generic SMTP traffic control 
using a "wait=" method without touching base with the whole, for lack 
of a better term, "Greylisting" concept.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 16:23:41 -0700
From: Douglas Otis <[email-masked]>
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 10/28/11 2:07 PM, Steve Atkins wrote: On Oct 28, 2011, at 1:49 PM, 
Carl S. Gutekunst wrote:
>> Steve Atkins wrote:
>>> It would be nice if the server could say "give me a minute to get my stuff together and come back", have the client retry in 60 seconds and get the mail accepted for delivery - with a delay of a minute or so, rather than a delay of 30 minutes. The server gets to shed load, the client gets to dump a message out of it's queue and the human correspondents see fast, in-order mail delivery rather than slow, out-of-order delivery.
>>>
>>> That's the end-user visible part of the thought behind https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/ anyway.
>> I'd support that proposal. I might even implement it.
>>
>> Unfortunately, the dominant conversation seems to be around an SMTP response that only applies to greylisting. I'd never use that or support it, it's such a teeny-tiny corner case, especially in the B-to-B world where I live.
> Me too.
>
> But that's the thing with an open mailing list - people who are really interested in one niche aspect of a subject or a broad generalization of one can easily dominate the conversation. And a broad, general blue-sky idea will always provoke more discussion than a trivially simple one. I'd suggest changing the title and spinning off a sub-thread to discuss just the "SMTP traffic control" aspect, but that didn't work too well last time�
>
> Let me spin up a new thread, with an implicit "no, I don't mean greylisting" subtext. :)
In the past, when there was an unexpected growth of incoming traffic, 
delays occurred when clients were unable to establish connections where 
the event was not noted in receiving SMTP logs.  One solution was to 
deploy more servers or impose more restrictive firewalls.

What does this mean for IPv6?

Geoff Huston offers a perspective:
http://bgp.potaroo.net/v6/as6447/
The graph clearly indicates IPv6 is seeing geometric growth.
What happens when IPv6 interfaces represent the new acceptance granularity?
http://bgp.potaroo.net/v6/as6447/report.txt

Total address space spanned by table (/64s): 246,300,931,457,031 or 
246,301 billion.  When divided by 3.8 billion of unicast IPv4 addresses, 
the result is 65k greater in size.  It also seems growth will see a six 
fold near term increase.  Expecting IPv6 CIDRs to remain larger than /64 
also seems doubtful since there will be business demands for allowing 
/128 exceptions. :^(

Attempting to use DKIM as a basis for acceptance is not practical.  DKIM 
mandates inclusion of invalid signatures.  Even when a valid signature 
is found, it is not intended to verify the entity issuing the message 
nor their intended recipients.  As such, DKIM fails to offer enough 
information to establish accountability.  Attempting to establish 
accountability based upon message content devoid of who sent a message 
and to whom would not permit consistent policies.

The alternative of depending upon IP address authorization in the face 
of RFC6333, RFC5969, and shared MTAs needed in the IPv4 to IPv6 
transition is illogical.  Especially when prevalent authorization 
schemes attempt to resolve all IPv4 and IPv6 hosts for an entire base 
domain using macro expansions that defeat DNS caching at each SMTP 
connection.

A desire to establish new reputation systems as a solution should first 
impose authentication as a baseline requirement.  Otherwise, the defense 
of a domain's reputation is impossible.

Our system makes use of dynamic temporary errors as a triage strategy, 
not as a method to determine whether a sender is able to retain state.  
Sources currently seen emitting spam are more likely to be temporarily 
blocked.  The duration depends entirely on the source.

Should we add a comment to please stop spamming before attempting a 
retry?  How would that help?

A reasonable control strategy might follow the structure reviewed in 
RFC6281.  Perhaps leveraging Kerberos and DNS/DANE services that offers 
long term tickets for SMTP senders.  Such a strategy could quickly 
eliminate the vast amount of abuse currently responsible for much of 
today's delays.  It would also ensure delivery through any Internet 
infrastructure.

Perhaps restoring accountability to SMTP would reduce a desire to use 
proprietary social networking where often the client software affords 
poor security.  Just as it is no longer safe to use FTP, SMTP suffers 
from openness that can no longer be sustained.  Rather than attempting 
to track billions upon billions of potential IP sources that are 
increasingly shared, a vetted authentication scheme can easily handle 
even a high estimate of 130 million legitimate sources.

The upside of this added complexity that is not really more complex when 
considering the scale needed to resolve accountability at the IP 
address.  SMTP could become dramatically safer and easier to deploy with 
much better throughput.  By focusing on outbound MTAs, users are more 
likely to be notified when their account or system is compromised.

-Doug

**

From: "Tim Kehres" <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>, "Steve Atkins" <[email-masked]>
Cc: "SMTP Interest Group" <[email-masked]>
Subject: Re: SMTP traffic control
Date: Sat, 29 Oct 2011 06:53:08 +0800
Sender: [email-masked]

----- Original Message ----- 
From: "Carl S. Gutekunst" <[email-masked]>
To: "Steve Atkins" <[email-masked]>
Cc: "SMTP Interest Group" <[email-masked]>
Sent: Saturday, October 29, 2011 4:49 AM
Subject: Re: SMTP traffic control

>
> Steve Atkins wrote:
>> It would be nice if the server could say "give me a minute to get my 
>> stuff together and come back", have the client retry in 60 seconds and 
>> get the mail accepted for delivery - with a delay of a minute or so, 
>> rather than a delay of 30 minutes. The server gets to shed load, the 
>> client gets to dump a message out of it's queue and the human 
>> correspondents see fast, in-order mail delivery rather than slow, 
>> out-of-order delivery.
>>
>> That's the end-user visible part of the thought behind 
>> https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/ 
>> anyway.
>>
>
> I'd support that proposal. I might even implement it.
>
> Unfortunately, the dominant conversation seems to be around an SMTP 
> response that only applies to greylisting. I'd never use that or support 
> it, it's such a teeny-tiny corner case, especially in the B-to-B world 
> where I live.

+1 

Date: Fri, 28 Oct 2011 17:28:07 -0400
To:  "[email-masked]'"	<[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

At 19:28 +0000 on 10/28/2011, Rosenwald, Jordan wrote about Re: SMTP 
traffic control:

>Perhaps I missed something (this has been a long thread), but I'm 
>completely missing how this will solve the problem of long, 
>unpredictable delays for users. Everything I've read says these are 
>return codes for server consumption, not to be returned to users.
>
>As best I can tell this proposed idea does nothing for the end user.

Once the end user has handed their message over to the MSA they are 
out of the loop in any case - The MSA has accepted the message for 
delivery/forwarding and there is nothing more that the user can/needs 
to do. The only time that the user's submission attempt is impacted 
is if the MSA is Graylisting their MUA Client. Once the MSA has 
accepted responsibility for the message, the user assumes that it 
will rapidly be delivered to its ultimate destination. The user 
benefits since the MSA (and intermediate forwarding MTAs) are given 
the information needed to pass the message on as rapidly as possible 
(ie: Pass it on as soon as the next MTA in the chain is willing/able 
to accept responsibility for the message).

From: Steve Atkins <[email-masked]>
Subject: draft-atkins-smtp-traffic-control
Date: Fri, 28 Oct 2011 14:17:56 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

(moved over from the current "SMTP traffic control" thread to stand out from the greylisting discussion a little)

Mail that's delivered quickly, in a minute or so, vs mail that sometimes takes a half hour to deliver makes a difference to the end user. Both due to the noticeable delay, and the annoyance of conversations arriving out of order.

Sometimes mailservers want to shed load, or prioritize one source of traffic over another. The only way to do that within SMTP that doesn't require you to keep a session open is to respond with a 4xx deferral and wait for the client to retry.

An ESMTP client, such as your ISP smarthost, will wait a while before retrying in response to a 4xx deferral. The RFCs say they should wait at least 30 minutes the first time. 

Hence https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/  It's a very lightweight approach that allows a server to tell a cooperating client how long it would like the client to wait before retrying. I suspect it has some issues that the IETF process might object to and I'm not sure that extending the idea to 5xx and 2xx responses is as good an idea as it seemed at first, but I think it's a useful concrete example to use as a basis for discussion.

A server that's doing greylisting could certainly benefit from this, but I'm not particularly interested in either focusing on the implications for greylisting nor expanding it to do other things that greylisters might find useful in this thread. 

Cheers,
  Steve

Subject: Re: SMTP traffic control
From: Steve Atkins <[email-masked]>
Date: Fri, 28 Oct 2011 14:07:06 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 28, 2011, at 1:49 PM, Carl S. Gutekunst wrote:

> Steve Atkins wrote:
>> It would be nice if the server could say "give me a minute to get my stuff together and come back", have the client retry in 60 seconds and get the mail accepted for delivery - with a delay of a minute or so, rather than a delay of 30 minutes. The server gets to shed load, the client gets to dump a message out of it's queue and the human correspondents see fast, in-order mail delivery rather than slow, out-of-order delivery.
>> 
>> That's the end-user visible part of the thought behind https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/ anyway.
>>  
> 
> I'd support that proposal. I might even implement it.
> 
> Unfortunately, the dominant conversation seems to be around an SMTP response that only applies to greylisting. I'd never use that or support it, it's such a teeny-tiny corner case, especially in the B-to-B world where I live.

Me too.

But that's the thing with an open mailing list - people who are really interested in one niche aspect of a subject or a broad generalization of one can easily dominate the conversation. And a broad, general blue-sky idea will always provoke more discussion than a trivially simple one. I'd suggest changing the title and spinning off a sub-thread to discuss just the "SMTP traffic control" aspect, but that didn't work too well last time�

Let me spin up a new thread, with an implicit "no, I don't mean greylisting" subtext. :)

Cheers,
  Steve

Date: Fri, 28 Oct 2011 14:02:08 -0700
From: Claus Assmann <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Reply-To: SMTP Interest Group <[email-masked]>
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Fri, Oct 28, 2011, Carl S. Gutekunst wrote:
> Steve Atkins wrote:

> >That's the end-user visible part of the thought behind https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/ anyway.

> I'd support that proposal. I might even implement it.

Yes, we should use this as basis for further discussion.  I would
like to see this extended by more explanations of temporary failures
beyond "wait", e.g., session and transaction limits (see an earlier
mail from me in this subthread).

Date: Fri, 28 Oct 2011 13:49:49 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Steve Atkins wrote:
> It would be nice if the server could say "give me a minute to get my stuff together and come back", have the client retry in 60 seconds and get the mail accepted for delivery - with a delay of a minute or so, rather than a delay of 30 minutes. The server gets to shed load, the client gets to dump a message out of it's queue and the human correspondents see fast, in-order mail delivery rather than slow, out-of-order delivery.
>
> That's the end-user visible part of the thought behind https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/ anyway.
>   

I'd support that proposal. I might even implement it.

Unfortunately, the dominant conversation seems to be around an SMTP 
response that only applies to greylisting. I'd never use that or support 
it, it's such a teeny-tiny corner case, especially in the B-to-B world 
where I live.

<csg>

Date: Fri, 28 Oct 2011 22:46:58 +0200
From: "Peter J. Holzer" <[email-masked]>
To: "[email-masked]'" <[email-masked]>
Subject: Re: SMTP traffic control
Mail-Followup-To: "[email-masked]'" <[email-masked]>
Sender: [email-masked]

--FkmkrVfFsRoUs1wW
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-28 19:28:05 +0000, Rosenwald, Jordan wrote:
>=20
> Perhaps I missed something (this has been a long thread), but I'm
> completely missing how this will solve the problem of long,
> unpredictable delays for users. Everything I've read says these are
> return codes for server consumption, not to be returned to users.=20

You mean client, not server, but otherwise you are correct: They are for
communication between MTAs, not between an MTA and a user.

Currently, when a client gets a 4xx response code, it has no idea when
to retry. RFC 5321 gives some advice:

   The sender MUST delay retrying a particular destination after one
   attempt has failed.  In general, the retry interval SHOULD be at
   least 30 minutes; however, more sophisticated and variable strategies
   will be beneficial when the SMTP client can determine the reason for
   non-delivery.

   Retries continue until the message is transmitted or the sender gives
   up; the give-up time generally needs to be at least 4-5 days.  It MAY
   be appropriate to set a shorter maximum number of retries for non-
   delivery notifications and equivalent error messages than for
   standard messages.  The parameters to the retry algorithm MUST be
   configurable.

Half an hour is ok if the reason for the temporary failure is a problem
which requires human intervention. It is already questionable if the
reason is that server is too busy (in my experience load spikes usually
last only a few minutes, but YMMV). It is definitely too long for
greylisting: Most servers which use greylisting use a much shorter
initial blocking time - a few minutes seems to be normal, I've even seen
times as short as 5 seconds. So, the client could successfully retry
after a few minutes, but it doesn't know that, and heeding the advice in
the RFC, waits for half an hour.=20

Of course people already noticed that and started to ignore the RFC in
this regard: Today MTAs often are configured for much shorter (initial)
retry times. This helps, but it is rather crude, a bit wasteful, and you
risk being blocked for "excessive number of connections" if you overdo
it. So the mail may still sit in the queue for a relatively long and
variable time (because that depends on the server configuration, the
clients initial delay, the clients backoff algorithm, the schedule of
queue runs and the precise moment when the mail was queued).

In contrast, consider that case that Hectors server (which uses an
initial blocking time of 55 seconds, as he wrote) can tell the client
"try again in 56 seconds" and that the client can do this. Then the
greylisted mail will get through after 56 seconds (+ probably a bit of
jitter added by the queue runner), and Hectors users will be happy
because they get the information they requested on the phone while they
are still talking instead of half an hour later. It also allows mail
admins to keep the default retry interval at a relatively high value and
avoid pounding servers which are already overloaded.

(It must be emphasized that this only works if both the server and the
client use the protocol extension. If the server doesn't provide the
retry hint or the client doesn't use it, the situation is just as it is
now. So to have any noticable effect it needs to be implemented by at
least some popular MTAs)

I also see some applications outside of greylisting, too:

For example, I've occassionally have to return a 4xx error to some
recipients of a multi-recipient message, because they couldn't be
processed together in the same transaction. In this case it would be
nice if I could tell the client that it is ok to immediately retry for
those recipients.

Another reason for unexpected mail delays is if you try to open more
connections than the receiving MX allows. The mails on some connections
get rejected with a temporary error and go back into the queue (possibly
several times). They could almost certainly be sent immediately over one
of the existing connections, but the client doesn't know that. If the
server could convey that information to the client, a wasteful delay
could be avoided.

> As best I can tell this proposed idea does nothing for the end user.=20

It can reduce the delays in case of 4xx errors. End users do care about
that (not always, maybe not even most of the time, but often enough).

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--FkmkrVfFsRoUs1wW
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--FkmkrVfFsRoUs1wW--

Subject: Re: SMTP traffic control
From: Steve Atkins <[email-masked]>
Date: Fri, 28 Oct 2011 13:39:26 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 28, 2011, at 12:28 PM, Rosenwald, Jordan wrote:

> Perhaps I missed something (this has been a long thread), but I'm completely missing how this will solve the problem of long, unpredictable delays for users. Everything I've read says these are return codes for server consumption, not to be returned to users. 
> 
> As best I can tell this proposed idea does nothing for the end user. 

Mail that's delivered quickly, in a minute or so, vs mail that sometimes takes a half hour to deliver makes a difference to the end user. Both due to the noticeable delay, and the annoyance of conversations arriving out of order.

Sometimes mailservers want to shed load, or prioritize one source of traffic over another. The only way to do that within SMTP that doesn't require you to keep a session open is to respond with a 4xx deferral and wait for the client to retry.

An ESMTP client, such as your ISP smarthost, will wait a while before retrying in response to a 4xx deferral. The RFCs say they should wait at least 30 minutes the first time. 

If Yahoo!, uh, I mean the receivers MX, only needs to shed load for a short time period, or just wants you to drop your open TCP session and go back to the TCP connect queue so that someone else gets a chance then all they can do is send that 4xx response and delay that mail delivery by 30 minutes or so. 

It would be nice if the server could say "give me a minute to get my stuff together and come back", have the client retry in 60 seconds and get the mail accepted for delivery - with a delay of a minute or so, rather than a delay of 30 minutes. The server gets to shed load, the client gets to dump a message out of it's queue and the human correspondents see fast, in-order mail delivery rather than slow, out-of-order delivery.

That's the end-user visible part of the thought behind https://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/ anyway.

Cheers,
  Steve

From: "Rosenwald, Jordan" <[email-masked]>
To: "[email-masked]'" <[email-masked]>, "[email-masked]'" <[email-masked]>
Subject: Re: SMTP traffic control
Date: Fri, 28 Oct 2011 19:28:05 +0000
Accept-Language: en-US
Sender: [email-masked]

Perhaps I missed something (this has been a long thread), but I'm completely missing how this will solve the problem of long, unpredictable delays for users. Everything I've read says these are return codes for server consumption, not to be returned to users. 

As best I can tell this proposed idea does nothing for the end user. 

----- Original Message -----
From: Peter J. Holzer [mailto:[email-masked]]
Sent: Friday, October 28, 2011 03:10 PM
To: [email-masked] <[email-masked]>
Subject: Re: SMTP traffic control

On 2011-10-24 15:38:16 -0700, Carl S. Gutekunst wrote:
> Hector wrote:
>>> Personally I think this is a solution looking for a problem. 
>>
>> The cliche is nice, but there is a recognized problem and a  
>> reproducible proof of concept solution possible to help minimize the  
>> impact.
>
> Hector, I've seen nothing in the past two of weeks of discussion that  
> identify a "problem."

It may not be a problem for you. For some (many?) users long and
unpredictable delays in mail delivery are a problem, especially if the
mail is sent during a phone call.

Such delays can be reduced and regularized if the server can provide
information about when to retry (and possibly other info, like the
number of allowed concurrent connections) to the client.

I really don't understand the resistance against the idea. Some servers
and some clients will implement it and benefit from it (if they are
talking to each other), others won't and for them everything will be as
before. How would a standardized way of conveying the same information
that some servers already convey informally hurt the mail system?

	hp

-- 
   _  | Peter J. Holzer    | Web 2.0 k�nnte man also auch �bersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         | 
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

Date: Fri, 28 Oct 2011 21:10:01 +0200
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: SMTP traffic control
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--PEIAKu/WMn1b1Hv9
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2011-10-24 15:38:16 -0700, Carl S. Gutekunst wrote:
> Hector wrote:
>>> Personally I think this is a solution looking for a problem.=20
>>
>> The cliche is nice, but there is a recognized problem and a =20
>> reproducible proof of concept solution possible to help minimize the =20
>> impact.
>
> Hector, I've seen nothing in the past two of weeks of discussion that =20
> identify a "problem."

It may not be a problem for you. For some (many?) users long and
unpredictable delays in mail delivery are a problem, especially if the
mail is sent during a phone call.

Such delays can be reduced and regularized if the server can provide
information about when to retry (and possibly other info, like the
number of allowed concurrent connections) to the client.

I really don't understand the resistance against the idea. Some servers
and some clients will implement it and benefit from it (if they are
talking to each other), others won't and for them everything will be as
before. How would a standardized way of conveying the same information
that some servers already convey informally hurt the mail system?

	hp

--=20
   _  | Peter J. Holzer    | Web 2.0 k=F6nnte man also auch =FCbersetzen als
|_|_) | Sysadmin WSR       | "Netz der kleinen Geister".
| |   | [email-masked]         |=20
__/   | http://www.hjp.at/ |  -- Oliver Cromm in desd

--PEIAKu/WMn1b1Hv9
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--PEIAKu/WMn1b1Hv9--

Date: Fri, 28 Oct 2011 09:34:50 -0400
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

--On Thursday, October 27, 2011 19:25 -0400 Hector Santos
<[email-masked]> wrote:

>...
>> As I tried to say in an earlier note, whether use of 4yz codes
>> in an attempt to block spam, or even as a traffic control
>> measure, is a clever extension of SMTP to use an existing
>> feature in a new way or a serious conception abuse of the base
>> protocol is mostly in the mind of the beholder.  But I don't
>> think it moves things forward to pretend that this sort of
>> usage was there all along.  
> 
> I agree, but it did evolve to include local policy user and/or
> system wide considerations.   You can see this with the change
> in 450:
> 
> RFC2821:
> 
>     450 Requested mail action not taken: mailbox unavailable
>         (e.g., mailbox busy)
> 
> RFC5321:
> 
>     450  Requested mail action not taken: mailbox unavailable
> (e.g.,
>          mailbox busy or temporarily blocked for policy
> reasons)

Yes, definitely.

>> I'm not suggesting it, but, to the extent to which SMTP
>> extension are involved here (rather than a description of how
>> to use the base protocol in a particular way and improve
>> interoperability if it is used that way), one could even
>> thinks about a new series of reply codes for traffic
>> management of various types (6yx?  9yz?) rather than trying
>> to do tricky things with 4yz.  
> 
> I consider this and even a special extended status code but
> stopped because the key part of the SMTP extension is parsing
> the response text, which as far I am aware of, there are no
> known SMTP practice to do so (outside of proprietary or
> localized) due to the idea we can't rely on it to be
> persistent.

I'm not sure what you mean.  Certainly the responses to VRFY and
EXPN have to be parsed and 5321 is moderately specific about
their syntax.    For other commands, yes, the pattern has been
to try to be sure there is no need for information in the
response text that isn't in the codes themselves.  If nothing
else, that facilitates both machine processing and localization.

> Honestly, I was surprise I have didn't get any resistance
> based on that concept (parsing of text) alone because frankly,
> it would be very hard to argue against and might have been a
> show stopper for me.
 
> But the atmosphere did not reject the idea, hence I went
> forward with the "reply=time-delay."  It helped that there
> were already GL servers out that already supply an informal
> time in in their responses.

The SMTP extension model clearly allows for doing just about
anything.  Either coming up with a new first-digit reply code
and associated text model or modifying an existing code would be
consistent with "just about anything" as long as the conditions
for advertising and accepting the extension are well-documented
and the interactions with legacy (un-extended) systems, if any,
are well-understood.

By contrast, taking a reply code that is specified in the SMTP
specs and in wide use without clients needing to interpret text
and hoping that some specific text will be given particular
treatment would almost certainly be a terrible idea.  But I
didn't understand you to be suggesting that.

Whether it is a good idea or not is a separate issue, for which
the comments of others that the discussion hasn't really started
are relevant.
 
>...
> What do you think?  or do we have enough history, including
> amony the other internet client/server text based protocols
> that provides some protocol for a format text structure
> command response output?
>...

Again, we do it already with the VRFY and EXPN responses, so it
is nothing new in that regard.  But, while the syntax of those
responses has gotten more specific over the years, the intent of
the reply code and information returned as its text really is
not.  Other protocols built on the FTP model are, IMO, less
interesting because each has its own rules.  And, again, if one
were planning a clean extension rather than a trick, moving to
either a new code structure or, e.g., a 46z code (with x6z
defined as "server traffic or connection management requests")
would have a certain amount of appeal. 

>...

   john

Date: Fri, 28 Oct 2011 07:03:13 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "[email-masked]" <[email-masked]>
Subject: Re: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

Tony Finch wrote:

>> How do we fix the silly configs?  With suggestions for values?
> 
> Firstly by contacting the postmaster at the other site with which 
> you are having interop problems. 

You are serious? Contacting people en masse?

> If you can point to an rfc like Murray's that may help.

What RFC?

> I don't see how your draft reduces the need for this kind 
> of liaison or how it can help with your example of a 
> silly one day greylist blocking period.

It eliminates contacting anyone because there is automated solution 
which doesn't required alerting the protocol cops on anyone.  Here is 
real proof of its working:

Attempt #1:

**************************************************************************
Wildcat! ESMTP Server v6.4.454.1
SMTP log started at Wed, 26 Oct 2011  20:00:14
Connection Time: 20111026 20:00:14  cid: 00000000 tid: 0000031C
SSL Enabled: NO
Message Queue: \spool\santronics\smtp\43285W
Destination: [email-masked]
Mail Host IP: 216.229.32.113:25 (smtp1.pcisys.net)
Attempt #1 LastAttempt: n/a
20:00:14.433 ** Opening Connection to host: smtp1.pcisys.net ip: 
216.229.32.113:25
20:00:16.321 S: 220 smtp1.pcisys.net ESMTP Sendmail 8.13.6/8.13.6; 
Wed, 26 Oct 2011 18:00:15 -0600 (MDT)
20:00:16.321 C: EHLO mail.santronics.com
20:00:16.399 S: 250-smtp1.pcisys.net Hello groups.winserver.com 
[208.247.131.9], pleased to meet you
20:00:16.399 S: 250-ENHANCEDSTATUSCODES
20:00:16.399 S: 250-PIPELINING
20:00:16.399 S: 250-8BITMIME
20:00:16.399 S: 250-SIZE
20:00:16.399 S: 250-DSN
20:00:16.399 S: 250-ETRN
20:00:16.399 S: 250-AUTH LOGIN PLAIN
20:00:16.399 S: 250-DELIVERBY
20:00:16.399 S: 250 HELP
20:00:16.399 C: MAIL FROM:<[email-masked]>
20:00:16.633 S: 250 2.1.0 <[email-masked]>... Sender ok
20:00:16.633 C: RCPT TO:<[email-masked]>
20:00:16.883 S: 451 4.7.1 Greylisting in action, please come back in 
00:10:00
20:00:16.883 C: QUIT
20:00:16.961 S: 221 2.0.0 smtp1.pcisys.net closing connection
20:00:16.961 ** Completed. Elapsed Time: 2589 msecs

Under normal circumstance, our MTA has a retry frequency with:

    2nd attempt 1 min
    3rd attempt 4 mins
    4th attempt 15 mins
    all the rest omitted

and without reading the hint from the server, our MTA will deliver the 
mail at the 4th try and in 20 minutes.

Here is the real result, at the 2nd attempt 10 minutes later:

**************************************************************************
Wildcat! ESMTP Server v6.4.454.1
SMTP log started at Wed, 26 Oct 2011  20:10:25
Connection Time: 20111026 20:10:25  cid: 00000000 tid: 00000D18
SSL Enabled: NO
Message Queue: \spool\santronics\smtp\43285W
Destination: [email-masked]
Mail Host IP: 216.229.32.234:25 (mx4.pcisys.net)
Attempt #2 LastAttempt: 10/26/2011 20:00:16
20:10:25.840 ** Opening Connection to host: mx4.pcisys.net ip: 
216.229.32.234:25
20:10:29.023 S: 220 mx4.pcisys.net ESMTP Sendmail 8.13.6/8.13.6; Wed, 
26 Oct 2011 18:10:25 -0600 (MDT)
20:10:29.023 C: EHLO mail.santronics.com
20:10:29.085 S: 250-mx4.pcisys.net Hello groups.winserver.com 
[208.247.131.9], pleased to meet you
20:10:29.085 S: 250-ENHANCEDSTATUSCODES
20:10:29.085 S: 250-PIPELINING
20:10:29.085 S: 250-8BITMIME
20:10:29.085 S: 250-SIZE 52500000
20:10:29.085 S: 250-DSN
20:10:29.085 S: 250-ETRN
20:10:29.085 S: 250-AUTH LOGIN PLAIN
20:10:29.085 S: 250-DELIVERBY
20:10:29.085 S: 250 HELP
20:10:29.085 C: MAIL FROM:<[email-masked]>
20:10:29.475 S: 250 2.1.0 <[email-masked]>... Sender ok
20:10:29.475 C: RCPT TO:<[email-masked]>
20:10:29.631 S: 250 2.1.5 <[email-masked]>... Recipient ok
20:10:29.631 C: DATA
20:10:29.709 S: 354 Enter mail, end with "." on a line by itself
20:10:29.912 S: 250 2.0.0 p9R0APdB024107 Message accepted for delivery
20:10:29.912 C: QUIT
20:10:29.990 S: 221 2.0.0 mx4.pcisys.net closing connection
20:10:29.990 ** Completed. Elapsed Time: 4244 msecs

This is clear proof of concept and potential for significant reduction 
in overhead and faster delivery. It works without calling anyone, 
yelling or cursing anyone out. Totally surprise you expect people to 
be contacting all these sites, You know it isn't realistic.

2 tries, 10 minutes. Not 4 tries, 20 minutes and before you say 
anything about my retry frequencies, they work fine against against 
sites with shorter delays.  But we expect to be mandating that across 
the board and throwing Murray's RFC at them.

>> Ok, I bite, what are your recommendations for:
>>
>> Server:
>>
>>   Blocking Time:  ___ secs/mins
> 
> Short. There's little point in sending more than one 450, to prove the sender retries.

How short?  30 secs, 1 minute? 5 minutes? 10 minutes?

>>   Record (sender recording) Expiration Time:  ____ secs/mins/hours/days
> 
> Something like a week.

Ok.

>> Client:  Retry Frequencies
> 
> RFC 5321 has guidelines. You need to try many more than four times!

I wasn't going to write 60-72 lines to fill you. You got the idea. 
But even then, going over 2 or 4 times contradicts the short and only 
blocking once. No?

We have no idea what the blocking times are.  The MTA retry tables 
people use vary and they are working blindly against the GL servers.

>> and I presume the fix means that EVERYONE has to use the same values at 
>> all servers and clients?
> 
> Of course not.

So it wouldn't be consistent nor a fix when the world is not running a 
single setup like yours or mines or the one modeled on Murray's RFC. 
Wouldn't it?

Less traffic and faster delivery. A BCP can't offer that. It can try 
but the MTA will still be working in a blind.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Cc: "[email-masked]" <[email-masked]>
From: Tony Finch <[email-masked]>
Subject: Re: new draft: draft-santos-smtpgrey-01
Date: Fri, 28 Oct 2011 11:20:56 +0100
To: Hector Santos <[email-masked]>
Sender: [email-masked]

On 28 Oct 2011, at 09:01, Hector Santos <[email-masked]> wrote:
> Tony Finch wrote:
>> The original motivation for this draft was to work around problems caused by silly retry or greylisting configurations. Fixing the silly configurations is the right way to deal with the problem.
> 
> How do we fix the silly configs?  With suggestions for values?

Firstly by contacting the postmaster at the other site with which you are having interop problems. If you can point to an rfc like Murray's that may help.

I don't see how your draft reduces the need for this kind of liaison or how it can help with your example of a silly one day greylist blocking period.

> Ok, I bite, what are your recommendations for:
> 
> Server:
> 
>   Blocking Time:  ___ secs/mins

Short. There's little point in sending more than one 450, to prove the sender retries.

>   Record (sender recording) Expiration Time:  ____ secs/mins/hours/days

Something like a week.

> Client:  Retry Frequencies

RFC 5321 has guidelines. You need to try many more than four times!

> and I presume the fix means that EVERYONE has to use the same values at all servers and clients?

Of course not.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

Date: Fri, 28 Oct 2011 04:01:55 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: "[email-masked]" <[email-masked]>
Subject: Re: new draft: draft-santos-smtpgrey-01
To: [email-masked]
Sender: [email-masked]

Tony Finch wrote:
> 
> The original motivation for this draft was to work around problems caused by 
> silly retry or greylisting configurations. Fixing the silly configurations is 
> the right way to deal with he problem.

How do we fix the silly configs?  With suggestions for values?

Ok, I bite, what are your recommendations for:

Server:

    Blocking Time:  ___ secs/mins
    Record (sender recording) Expiration Time:  ____ secs/mins/hours/days

Client:

    Retry Frequencies

    - initial try  ___ secs/mins
    - 2nd try      ___ secs/mins
    - 3rd try      ___ secs/mins
    - 4th try      ___ secs/mins

and I presume the fix means that EVERYONE has to use the same values 
at all servers and clients?

-- 
Sincerely

Hector Santos
http://www.santronics.com

Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
From: Tony Finch <[email-masked]>
Subject: Re: new draft: draft-santos-smtpgrey-01
Date: Fri, 28 Oct 2011 08:36:31 +0100
To: Keith Moore <[email-masked]>
Sender: [email-masked]

On 26 Oct 2011, at 16:47, Keith Moore <[email-masked]> wrote:
> 
> So... not a good idea, IMO.

The original motivation for this draft was to work around problems caused by silly retry or greylisting configurations. Fixing the silly configurations is the right way to deal with he problem.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

Date: Fri, 28 Oct 2011 02:39:08 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

Robert A. Rosenberg wrote:
> 
> At 19:25 -0400 on 10/27/2011, Hector Santos wrote about Re: 4yz 
> Temporary Rejections is part of the SMTP Protocol:
> 
>> But we don't any think in SMTP that comes comes to parsing text for
>> anything, as far I   think of right now.
> 
> I beg to differ. It is my impression that when you attempt to send a 
> message you can send an optional message size. This size information is 
> scanned and used to short circuit the upload attempt if this will exceed 
> the max attachment size or (possibly) if this will trigger a mailbox 
> size exceeded condition. I may be wrong but it seems that this is an 
> area where some scanning may be being done to improve processing 
> (sending a too large attachment can thus be detected BEFORE DATA not at 
> end of DATA time).

First please allow me to correct my statement:

    With the exception of extended status codes, I don't think SMTP comes
    close to parsing text for anything else, as far I am aware of.

You're right. We have logic for reading extended text information in 
MAIL FROM and I believe RCPT TO in some areas, but this is the client 
command.  Not the server response.

Do we have methods out there, outside of the reading optional server 
numeric extended status codes, to parse the remaining text in server 
responses?

--
HLS

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 02:22:06 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

IME, most standard protocols (at least the goods one) where designed 
with the idea of flexibility and growth.  If being creative is part of 
that process, then it was successful in its intent.

By your definition, writing SMTP extensions is a creative 
interpretation of the standard SMTP process which was meant to be 
locked in to any one particular method of usage.  Nes Pas! And I 
believe you meant that.

We can only work with a common backbone mechanism that can not 
deviate.  The SMTPGREY extension does not alter that. The questionable 
new issue, the "moving cheese" so to speak is using the text part to 
add more server information for the client to read because SMTP is 
currently is incapable of providing more response information outside 
the reply code. With the exception of Extended Status Code, its 
client/server negotiation capabilities are limiting and not backward 
compatible.

But it does not mean we can't use the text part because the Extended 
Status Codes standard proves we were willing to read more beyond the 
just the standard SMTP reply codes.  Compliant clients can use the 
EXTENDEDSTATUSCODE to definitively get that "signal" that parsing text 
is possible for these special extended sections.

So what is different that a GREYLIST SMTP Extension Keyword offering? 
   Really not that much difference idea.

One can make a note that the special reading is only possible when the 
GREYLIST service keyword is presence.  Lets get that RFC2019 lingo 
correct.

I currently have it that its all optional because I did not wish to 
limit it to just EHLO sessions but also clients who issue HELO.  The 
only MUST is when the server issues:

    250-GREYLIST RETRY

that a MUST was appropriate for it to add a "retry=time-delay" when 
its 4zy rejections are greylist related.

--
HLS

SM wrote:
> 
> Hi Hector,
> At 16:25 27-10-2011, Hector Santos wrote:
>>  considerations.   You can see this with the change in 450:
> 
> The subject line says "temporary rejections".  The term used in the SMTP 
> specification is "temporary error status code".
> 
>> Honestly, I was surprise I have didn't get any resistance based on 
>> that concept (parsing of text) alone because frankly, it would be very 
>> hard to argue against and might have been a show stopper for me.
> 
> The hunting season is not open yet. :-)
> 
>  From RFC 5321:
> 
>   "Unfortunately, variations, creative interpretations, and outright
>    violations of Internet mail protocols do occur; some would suggest
>    that they occur quite frequently."
> 
> The draft discusses about a creative interpretation of SMTP.  Dave 
> Crocker commented on why greylisting works [1].
> 
> BTW, RFC 1651 has been obsoleted by RFC 5321.  The correct reference for 
> DKIM is RFC 6376.
> 
> Regards,
> -sm
> 
> 1. http://www.imc.org/ietf-smtp/mail-archive/msg06860.html
> 
> 

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 28 Oct 2011 01:57:34 -0400
To: Hector Santos <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Cc: [email-masked]
Sender: [email-masked]

At 19:25 -0400 on 10/27/2011, Hector Santos wrote about Re: 4yz 
Temporary Rejections is part of the SMTP Protocol:

>But we don't any think in SMTP that comes comes to parsing text for
>anything, as far I   think of right now.

I beg to differ. It is my impression that when you attempt to send a 
message you can send an optional message size. This size information 
is scanned and used to short circuit the upload attempt if this will 
exceed the max attachment size or (possibly) if this will trigger a 
mailbox size exceeded condition. I may be wrong but it seems that 
this is an area where some scanning may be being done to improve 
processing (sending a too large attachment can thus be detected 
BEFORE DATA not at end of DATA time).

Date: Fri, 28 Oct 2011 00:53:16 -0500
From: Pete Resnick <[email-masked]>
To: <[email-masked]>
CC: Dave CROCKER <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

Interrupting my work on the reply to your previous message, but I can 
answer this one more quickly:

On 10/28/11 12:13 AM, Dave CROCKER wrote:
> Pete,
>
> On 10/28/2011 6:28 AM, Pete Resnick wrote:
>
>> If a document
>> defines conventions and particular methods of using a technical 
>> specification,
>> then there can be implementations of the document, and therefore 
>> incremental
>> experience with it, and it therefore should be on the standards track.
>
>> If a document defines ways of being interoperable with a technical 
>> specification,
>> then there can be implementations of the document, and therefore 
>> incremental
>> experience with it, and it therefore should be on the standards track.
>
> You think BCP 23, 24, 28, 34, ... involve no software and do not 
> change the behavior of protocol engines?
>
> By your personal definitions, these seem to have been mis-assigned and 
> ought to be required to be standards track.

Probably. And often to the repeated consternation of the community. We 
*repeatedly* get into arguments on the IETF list (and in working group 
lists) about what should be BCP or Informational and what should be 
standards track. I know chairs and ADs who get asked time and again for 
guidance on this topic, and they get different answers depending on who 
they ask (and the way the wind is blowing.) There is *not* community 
consensus on this point. We are haphazard in our assignment to each of 
the categories.

> There is a core problem:  you appear to be using your personal 
> terminology and definitions, rather than reflecting community practice.

To the contrary, I am constantly reflecting on community practice. That 
practice is inconsistent. It varies by WG, by AD, by IESG, and by the 
entire community over time. Sometimes in waves.

My push has been that *I* will try to have WGs in the Apps Area for 
which I am the stuckee be as consistent in their practice as I can 
convince them to be. So I'm trying.

> In all likelihood, your terms and definitions are more reasonable, 
> consistent and practical than the community's, but that's irrelevant.
>
> What is relevant is established practice.

If there were an established practice, I would agree with you 
completely. There's not.

> It's fine to seek a change in established practice, but not as a sole 
> voice in a management position, invoking personal and distinct 
> language and requirements.  That's not supposed to be the way things 
> work around here.
>
> My query about applicability statements was not about documents that 
> effectively serve that role but about documents that are formally 
> assigned that status AND are effective.  Established practice, not 
> personal assessment.
[...]
> Please get the consensus of the community about this.

I absolutely am trying to get said consensus. One document, one WG at a 
time. I have asked WGs (in other areas as well as Apps) why some 
documents are Informational or BCP instead of standards track. Sometimes 
they tell me to blow. Sometimes they agree and switch the documents to 
standards track. I'm hoping that if I repeat this exercise enough times, 
we can get to a consensus on this.

Now, maybe you are complaining that, given my position as AD (and its 
unfortunate perception as authoritative), my style of trying to achieve 
that consensus is actually "coercing" rather than "getting" that 
consensus. (Stories of missing fire hydrants come to mind.) I've 
certainly misjudged such things before, and if I'm doing it here, I'll 
try to change tack. But my intentions are as pure as such things ever get.

> Please then re-label the documents that have been misassigned.

One windmill at a time.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Fri, 28 Oct 2011 07:13:17 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Pete Resnick <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

Pete,

On 10/28/2011 6:28 AM, Pete Resnick wrote:
> > Herein lies my confusion. What is a Best Current Practices document if not one
>> that talks about conventions, particular methods of using the thing a
>> specification defines, [improved] interoperability, or relating of operational
>> experience?
>
> I say it just below: A BCP document gives guidelines to operators and
> administrators, gives statements of architectural principle, or documents
> procedures and operations of the IETF itself. BCPs ought not have conventions
> and particular methods of using a technical specification, etc.

 >    If a document is
> defines conventions and particular methods of using a technical specification,
> then there can be implementations of the document, and therefore incremental
> experience with it, and it therefore should be on the standards track.

 >     If a
> document defines ways of being interoperable with a technical specification,
> then there can be implementations of the document, and therefore incremental
> experience with it, and it therefore should be on the standards track.

You think BCP 23, 24, 28, 34, ... involve no software and do not change the 
behavior of protocol engines?

By your personal definitions, these seem to have been mis-assigned and ought to 
be required to be standards track.

 >    All of
> these kinds of statements (and the additional ones I cite above) are
> applicability statements, and those statements can be implemented, and we can
> get incremental experience with those statements, and therefore documents with
> those statements belong on the standards track.

There is a core problem:  you appear to be using your personal terminology and 
definitions, rather than reflecting community practice.  In all likelihood, your 
terms and definitions are more reasonable, consistent and practical than the 
community's, but that's irrelevant.

What is relevant is established practice.  It's fine to seek a change in 
established practice, but not as a sole voice in a management position, invoking 
personal and distinct language and requirements.  That's not supposed to be the 
way things work around here.

My query about applicability statements was not about documents that effectively 
serve that role but about documents that are formally assigned that status AND 
are effective.  Established practice, not personal assessment.

> The distinction I want to make is: If you're giving implementation
> specifications or guidance for the protocol, those belongs as part of a
> standards track document, whether they are statements in the protocol document
> itself or statements in a standalone document (which I've been calling an AS).

Please get the consensus of the community about this.  Please then re-label the 
documents that have been misassigned.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Thu, 27 Oct 2011 23:28:36 -0500
From: Pete Resnick <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

On 10/26/11 11:37 PM, Murray S. Kucherawy wrote:

>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Pete Resnick
>> Sent: Wednesday, October 26, 2011 8:08 AM
>> To: Keith Moore
>> Cc: [email-masked]; [email-masked]
>> Subject: Re: draft-kucherawy-greylisting-bcp
>>
>> I think Keith has it pretty spot-on. BCPs are for documents where it
>> makes no sense to talk about the concepts of (to quote 2026) "protocol,
>> service, procedure, convention, or format", "particular methods of using
>> a [technical specification]", "specify particular values or ranges, of
>> TS parameters or subfunctions of a TS protocol that must be
>> implemented", "interoperability", and "implementation and/or operational
>> experience".
>>      
> Herein lies my confusion.  What is a Best Current Practices document if not one that talks about conventions, particular methods of using the thing a specification defines, [improved] interoperability, or relating of operational experience?
>    

I say it just below: A BCP document gives guidelines to operators and 
administrators, gives statements of architectural principle, or 
documents procedures and operations of the IETF itself. BCPs ought not 
have conventions and particular methods of using a technical 
specification, etc. If a document is defines conventions and particular 
methods of using a technical specification, then there can be 
implementations of the document, and therefore incremental experience 
with it, and it therefore should be on the standards track. If a 
document defines ways of being interoperable with a technical 
specification, then there can be implementations of the document, and 
therefore incremental experience with it, and it therefore should be on 
the standards track. All of these kinds of statements (and the 
additional ones I cite above) are applicability statements, and those 
statements can be implemented, and we can get incremental experience 
with those statements, and therefore documents with those statements 
belong on the standards track.

>> My personal take is
>> that BCPs should be reserved to guidelines for operators and
>> administrators, statements of architectural principles, and
>> documentation of procedures and operations of the IETF itself.
>>      
> That sounds right, but some of it seems (to me) to conflict with the citations you made.  I suspect there's some (pardon the allusion) grey area in there that makes it hard, for me at least, to see clearly when something should be a BCP and something should be an AS.
>    

The only grey area would be the guidelines for operators. But when I say 
guidelines for operators, I'm thinking of things like the amount of 
available disk space required to run a typical SMTP server getting 10's 
of messages per second or the recommendation to get an SMTP server with 
a robust logging facility because of the need to check error reports, 
things that really aren't part of the implementation of the protocol, 
but are local conventions for the use of implementations. Those usually 
belong in BCPs, though sometimes we see such statements in standards 
track documents too.

The distinction I want to make is: If you're giving implementation 
specifications or guidance for the protocol, those belongs as part of a 
standards track document, whether they are statements in the protocol 
document itself or statements in a standalone document (which I've been 
calling an AS). We make these statements all of the time as part of 
standards track documents, and the act of separating them into a 
standalone document apart from the technical specification doesn't mean 
they don't belong on the standards track. BCPs should only be used when 
putting something on the standards track makes no sense.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Thu, 27 Oct 2011 18:35:46 -0700
To: Hector Santos <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Cc: [email-masked]
Sender: [email-masked]

Hi Hector,
At 16:25 27-10-2011, Hector Santos wrote:
>  considerations.   You can see this with the change in 450:

The subject line says "temporary rejections".  The term used in the 
SMTP specification is "temporary error status code".

>Honestly, I was surprise I have didn't get any resistance based on 
>that concept (parsing of text) alone because frankly, it would be 
>very hard to argue against and might have been a show stopper for me.

The hunting season is not open yet. :-)

 From RFC 5321:

   "Unfortunately, variations, creative interpretations, and outright
    violations of Internet mail protocols do occur; some would suggest
    that they occur quite frequently."

The draft discusses about a creative interpretation of SMTP.  Dave 
Crocker commented on why greylisting works [1].

BTW, RFC 1651 has been obsoleted by RFC 5321.  The correct reference 
for DKIM is RFC 6376.

Regards,
-sm

1. http://www.imc.org/ietf-smtp/mail-archive/msg06860.html 

Date: Thu, 27 Oct 2011 19:25:10 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

John C Klensin wrote:

> Hector Santos wrote:
>> One may suggest SMTP did document the basic idea of
>> greylisting by using the term "temporary" for a 4yz reply
>> code.  It just [kept] it open ended and did not get into ideas
>> of what is "temporary" and/or how it relates to "time."
> 
> ..
>
> As I tried to say in an earlier note, whether use of 4yz codes
> in an attempt to block spam, or even as a traffic control
> measure, is a clever extension of SMTP to use an existing
> feature in a new way or a serious conception abuse of the base
> protocol is mostly in the mind of the beholder.  But I don't
> think it moves things forward to pretend that this sort of usage
> was there all along.  

I agree, but it did evolve to include local policy user and/or system 
wide considerations.   You can see this with the change in 450:

RFC2821:

    450 Requested mail action not taken: mailbox unavailable
        (e.g., mailbox busy)

RFC5321:

    450  Requested mail action not taken: mailbox unavailable (e.g.,
         mailbox busy or temporarily blocked for policy reasons)

> I'm not suggesting it, but, to the extent to which SMTP
> extension are involved here (rather than a description of how to
> use the base protocol in a particular way and improve
> interoperability if it is used that way), one could even thinks
> about a new series of reply codes for traffic management of
> various types (6yx?  9yz?) rather than trying to do tricky
> things with 4yz.  

I consider this and even a special extended status code but stopped 
because the key part of the SMTP extension is parsing the response 
text, which as far I am aware of, there are no known SMTP practice to 
do so (outside of proprietary or localized) due to the idea we can't 
rely on it to be persistent.

Honestly, I was surprise I have didn't get any resistance based on 
that concept (parsing of text) alone because frankly, it would be very 
hard to argue against and might have been a show stopper for me.

But the atmosphere did not reject the idea, hence I went forward with 
the "reply=time-delay."  It helped that there were already GL servers 
out that already supply an informal time in in their responses.

But I am still concern about the idea of parsing text.  No one here 
rejected it but it may come up under IETF/IAB review if it got that 
far - I know if I was part of the review board I would raise that 
question.

What do you think?  or do we have enough history, including amony the 
other internet client/server text based protocols that provides some 
protocol for a format text structure command response output?

Off hand:

- FTP has the parsing for the data port establishment.
- POP3 as the parsing for the totals, indices, etc.
- NNTP also has similar parsing of numbers.

But we don't any think in SMTP that comes comes to parsing text for 
anything, as far I   think of right now.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 27 Oct 2011 18:12:38 -0400
From: John C Klensin <[email-masked]>
To: Hector Santos <[email-masked]>, [email-masked]
Subject: Re: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

--On Thursday, October 27, 2011 17:01 -0400 Hector Santos
<[email-masked]> wrote:

>...
> One may suggest SMTP did document the basic idea of
> greylisting by using the term "temporary" for a 4yz reply
> code.  It just keep it open ended and did not get into ideas
> of what is "temporary" and/or how it relates to "time."

Without trying to split this set of hairs further, I suggest
that a key property of the original SMTP use of 4yz codes was
that they were reporting an inability to continue that was
outside the control of the SMTP server itself.  "Connection
going down", "temporary lack of storage to add to user mailbox",
"System shutting down", etc., are all examples of that
situation.  "I'm going to deliberately put stumbling blocks in
your path so that maybe, if you are a bad guy, you will just go
away" was certainly not part of that model.  

As I tried to say in an earlier note, whether use of 4yz codes
in an attempt to block spam, or even as a traffic control
measure, is a clever extension of SMTP to use an existing
feature in a new way or a serious conception abuse of the base
protocol is mostly in the mind of the beholder.  But I don't
think it moves things forward to pretend that this sort of usage
was there all along.  

I'm not suggesting it, but, to the extent to which SMTP
extension are involved here (rather than a description of how to
use the base protocol in a particular way and improve
interoperability if it is used that way), one could even thinks
about a new series of reply codes for traffic management of
various types (6yx?  9yz?) rather than trying to do tricky
things with 4yz.  

   john

Date: Thu, 27 Oct 2011 17:01:27 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "[email-masked]" <[email-masked]>
Subject: 4yz Temporary Rejections is part of the SMTP Protocol
Sender: [email-masked]

Dave CROCKER wrote:

>>>> Greylisting is far more obviously a protocol with testable 
>>>> interoperability than is, say, RFC 5234 or 5322.

>> Pete stated:
>> [To clarify: I am not committed to greylisting being a protocol. I'm 
>> just saying it looks more like a protocol than do 5234 or 5322. It is 
>> the claim that there is no way to test interoperability that I reject..]
> 
> 5234 and 5322 define things that need direct processing by whoever 
> receives the data.  There is no intermediary.  The interaction between 
> data generation and data parsing/processing is exactly that, an 
> interaction.  In addition, some of the data with 5322 specify actual, 
> honest to goodness protocol behaviors. Reply-to directs the construction 
> of a portion of a new email message, for example.
> 
> The effect of the proposed capability, in contrast, is entirely 
> indirect.  It does not define a new format or a new protocol command or 
> a new protocol response.  Rather, it uses /existing/ ones in a 
> particular way.  As such, sorry, but those other specs are "more like a 
> protocol" than this one is.

I view the idea of SMTP server's intentional usage of Temporary 
Rejection with protocol expectation and conformity of a SMTP client 
Retry standard to use 4yz response, a method, procedure and/or a 
protocol concept in SMTP.

While the term "GreyListing" is probably poor and dislike by the SMTP 
purist, replacing it with

        "Intentional Temporary Rejections with a Retry Expectation"

which is part of the SMTP protocol, inherently makes the idea we call 
Greylisting as part of the "protocol" as well.

In other words, the protocol is the idea that a 4yz response is 100% 
designed to drive and allow a client to retry the same transactional 
information at some later point in time. A 5yz is designed to drive 
the client to not retry again the same transactional information - ever.

What is not part of the SMTP protocol, but part of the GreyList 
Protocol is the concept of "blocking time."

One may suggest SMTP did document the basic idea of greylisting by 
using the term "temporary" for a 4yz reply code.  It just keep it open 
ended and did not get into ideas of what is "temporary" and/or how it 
relates to "time."

That is what this SMTP Extensions hopes to offer - by virtue of an 
Extension, to optionally add a time factor to the existing 4yz 
temporary rejection protocol concept in SMTP.

At best, the reason we call "Greylisting" for a temporary rejection 
falling with a intentional delay may not be part of the SMTP protocol, 
but that does not make it any less a protocol extended idea to the 
current SMTP framework.

The difference is random MTA retrying vs a client/server negotiated 
controlled one - thats a protocol.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 27 Oct 2011 22:14:13 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Pete Resnick <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

Pete,

On 10/26/2011 6:16 PM, Pete Resnick wrote:
>> You have just defined an interoperability criterion that includes all sources
>> of software misbehavior, daemon sleep times, and anything else that can cause
>> delays.
>
> I notice you conveniently snipped my example of using a status code that caused

I clipped enough to provide context for my reply.  No other intent was present. 
  If I snipped too much, please add it back and explain its relevant.  (Your 
observational text about the snipping didn't explain the relevance.)

> trouble. But yes, timeouts are commonly put into standards track documents, and
> reasonably so. Implementation parameters do cause interoperability problems and
> many of those are observable from the net. I have no idea how you can claim
> these aren't "interoperability criteria".

The new work does not change the protocol definition, including timeouts.  It 
works in the grey area of operational choice.  Operations, not protocol. 
Operations = practice.

> Let's go back to my initial claims: If it implementation experience would cause
> changes to the instructions we give in a document to implementers, then it
> deserves to be on the standards track. If it's a "protocol, service, procedure,
> convention, or format" [2026, 3.1], it belongs in a technical specification. If

We have a very consistent tendency in the IETF to argue only one side of an 
issue and to quite literally ignore or flatly deny counter-points.  To remedy 
that, in this case:

RFC 2026, Section 5:

    "The BCP subseries of the RFC series is designed to be a way to
     standardize practices...community's best current thinking on a statement of
     principle or on what is believed to be the best way to perform some
     operations or IETF process function."

Anyone who thinks that the IETF's use of the words procedure, convention and 
practices are completely precise and completely orthogonal should please let me 
know what their repertoire of drugs is, because I really would like to see the 
world that way.  Absent the drugs, the world we are living in is rather more 
messy and ambiguous than your line of argument would suggest.

Equally, I suggest that the draft under consideration is a "way to perform" SMTP 
processing.  It does not change SMTP at all, but pertains to internal 
operational issues in HOW the protocol is used, not in WHAT is does.

> it gives implementation instructions (values for parameters, how the protocol
> ought be implemented, etc.), then it belongs in an applicability statement. Both
> TSs and ASs go on the standards track.

Now is the time for noting that the suggestion to use Applicability Statement is 
rubbish.  ASs are not used and have no practical meaning in the current IETF. 
They were a fine idea.  I wish they had succeeded.  But they didn't.

You are encouraged to demonstrate that my perception is incorrect, such as by 
citing a /pattern/ of AS assignments over the last 5 years and by describing 
their real-world utility.  To the extent that you wish to change the reality of 
their non-use, please do it through a collaborative process that gets people to 
sign up for it before risking their work with a process experiment, rather than 
by suddenly foisting it on a document not intended for the process experiment.

>>> Greylisting is far more obviously a protocol with testable interoperability
>>> than is, say, RFC 5234 or 5322.
>
> [To clarify: I am not committed to greylisting being a protocol. I'm just saying
> it looks more like a protocol than do 5234 or 5322. It is the claim that there
> is no way to test interoperability that I reject..]

5234 and 5322 define things that need direct processing by whoever receives the 
data.  There is no intermediary.  The interaction between data generation and 
data parsing/processing is exactly that, an interaction.  In addition, some of 
the data with 5322 specify actual, honest to goodness protocol behaviors. 
Reply-to directs the construction of a portion of a new email message, for example.

The effect of the proposed capability, in contrast, is entirely indirect.  It 
does not define a new format or a new protocol command or a new protocol 
response.  Rather, it uses /existing/ ones in a particular way.  As such, sorry, 
but those other specs are "more like a protocol" than this one is.

>> 5322 and other format specs explicitly entail parsing and processing (and in
>> the case of From: and Reply-to:, replying) by the remote engine. That's
>> essential interoperability.
>
> How does one "test interoperability" in the case of parsing that is going on in
> a local implementation like 5322? How is that different than greylisting? And
> for ANBF...?

You think that the inability to produce a successful parse of data does not 
demonstrate an interoperability failure?

>> In other words, formats are inherently bilateral. Greylisting is unilateral.
>
> Formats are bilateral, but only in the sense that both sides have a common
> understanding of the semantics of the parsed format.

That 'only' is pretty substantial, since its the essence of interoperation.

>  Greylisting is bilateral
> for those who actually participate: That is, if I receive 4xx status code, I
> know to wait a while and resend.

That's a requirement for processing SMTP.  The client SMTP is not required to 
know or care that greylisting is happening.  As such, the client SMTP engine is 
not "participating" in greylisting.  It is participating in SMTP.

>  Successful delivery happens if we both do our
> parts correctly. It is certainly the case that if I am a poorly implemented SMTP
> client, I will not try again and my mail will not be delivered (the point of the

That's an SMTP failure, whether greylisting is being done by the server SMTP or not.

Note the key point:  The success and failure conditions you cite apply to ANY 
SMTP interaction, independent of greylisting.  As such, they do not test 
greylisting "interoperability".

>> Again: since my view is rubbish, please define an interoperability test for it.
>
> (I don't think I am making any statement about *your* *view*. I apologize for
> this coming across as a personal attack. I used "rubbish" only to mean, "I

Well, I was the one who introduced the personal language in that last round but 
it wasn't intentional.  I am concerned with calling the substance rubbish, not 
the speaker.  (I am pretty sure that I was vetted as rubbish long enough ago and 
publicly enough to make it a waste of everyone's time to repeat it now, so it 
didn't occur to me that that might have been what you intended...)

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Thu, 27 Oct 2011 16:06:21 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: [email-masked]
Subject: Re: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

Paul Smith wrote:
> 
> On 26/10/2011 16:22, John C Klensin wrote:
>> Independent of what advantages it offers people who are trying to 
>> cooperate in sending legitimate mail, to paraphrase Paul Vixie, giving 
>> spammers the information they need to become smarter and the 
>> incentives to do so are really not in our long-term best interests.
> 
> The reason why I don't think this is an issue, is that spammers already 
> have the information they need to get around greylisting. 

Right, the 4yz retry code.

> The 
> 'retry=<time>' data won't give them any information they need to be able 
> to get their messages through. However, the fact is that, for now at 
> least, many are not doing anything about countering greylisting. When 
> they do decide to do something about it, the 'retry=<time>' data won't 
> be vital to them, or even make their job significantly easier.

+1.  There is basically two categories of senders GL addresses:

1) Primarily those that will never retry for whatever reason that may 
be. They just don't. However, one can reasonable suggest Spamming a 
low margin, high volume market business and success is measured in low 
acceptance rates such as 1% or even less.  So randomly blasting 1 
million harvested email addresses and getting 1% (10,000) is often 
deemed successful in this market.  I still see 1990's address for our 
usage base when we offered Free accounts to customers.  We also see 
many CIS (CompuServe) email addresses still being tried.  And a huge 
amount try obscure variations user-ids in addresses.

2) Secondary are those that do retry, but they retry again maybe twice 
more immediately and often using a different IP.  Even with a low GL 
blocking time such as ours with 55 seconds, I see these getting 
blocked again and stop after the 2 to 3 fast retries.

With #2, it is also apparent many that do have fast retry will back 
off and retry later and GL can not stop any client that retries after 
the blocking time.

But it is #1 which is by far the major benefit of using an intentional 
4yz temporary rejection methodology for mail filtering anonymous senders.

> Having said that, I'd prefer the 'retry=<time>' extension to be generic, 
> and not linked to greylisting, except by coincidence.

The problem with going generic will depend on what we mean by 
"Generic" and how broad we want that to be.

Ideally, in my technical opinion, from a pure mechanical client/server 
state machine protocol framework, it can be a very simple generic SMTP 
concept or generic SMTP extension to the 4yz response.  We could 
called it:

       SMTP Extension for Handling 4yz Temporary Rejections

The abstract may say:

       SMTPRETRY is SMTP extension to provide retry time delay
       information in negative temporary rejections 4yz responses
       to help address issues related to wasted attempts and
       delayed mail delivery.

With a very simple implementation description with no subjective 
background reasoning for why the SMTP 4yz exist in the standard or why 
servers use it, it can simple define:

       A server MAY add a retry=time-delay in in 4yz responses and
       clients MAY leverage the server time hint in the client
       retry framework employed in their implementation of SMTP.

So this would be simple generic idea for extending the SMTP 4yx 
description.

But if we begin to talk above and beyond that with SMTP Traffic 
Control and Management ideas then IMO, it gets more complex and begins 
to introduce a whole range of things that is not closer to modifying 
SMTP protocol and definitely will require more code changes to 
implement the more elaborate proposal.  What will happen IMV is that 
people will pick and choose what they need and chances are good it 
will be for "Greylisting" reasons systems are facing today.

SMTPGREY tries to keep a real focus to realistically deliver this 
extension with specific existing practical purposes and things that we 
can measure today. It is not experimental or a BCP although 
documenting the basic GL framework does lends itself to making some 
"suggestions."  I approached that by simply indicating to keep delays 
as low as possible in the name of keeping with timely deliveries and 
lowering the impact of intention temporary rejections. I provided some 
empirical average values for blocking times.  But overall, the 
extension is not intended to change the standard SMTP protocol,  the 
pseudo-standard GL "protocol" and begin to mandate any idea that 
intends to change how systems concurrently behave.  Its an extension 
and by its definition, its all optional and does not alter anything 
but to allow a server to tell a client how to handle a 4yz response in 
terms of "Time."

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 27 Oct 2011 12:48:17 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: Trusted agency (was: We need an IETF BCP  for GREY LISTING)
Sender: [email-masked]

This is an aside directed at one limited point, and is probably
not relevant to the larger discussion.

On Tue, Oct 18, 2011 at 12:20:17AM -0700, SM wrote:
> Social networks is where a lot of spam come from but we cannot call it that.

I don't see why not.  I receive unsolicited bulk email from social networks
on a regular basis, and because it's UBE, it's unquestionably spam.

Let me note in passing that much of that spam also happens to consist
of forgeries, no doubt an attempt to leverage automated and human filters
in favor of delivery.

---rsk

Date: Thu, 27 Oct 2011 12:43:16 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On Mon, Oct 17, 2011 at 07:51:47AM -0400, John C Klensin wrote:
> And even more strongly agree if "delivery of all legitimate mail
> without forcing senders to use a small number of senders with
> bilateral arrangements" is added to the list.

I strongly concur with this as well, and would welcome any insights
as to how we might collaboratively work toward this goal.

---rsk

Date: Thu, 27 Oct 2011 06:54:47 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

On 10/27/2011 6:38 AM, Murray S. Kucherawy wrote:
>> Greylisting is not a protocol, according to any definition I know.
>
> ...as currently defined.  It seems this fledgling effort seeks to change
> that, and would introduce a new part of SMTP that could be so tested.

Please note the subject line of this thread.  The SMTP extension is not the 
document being discussed.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Wed, 26 Oct 2011 21:38:37 -0700
Subject: RE: draft-kucherawy-greylisting-bcp
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave CROCKER
> Sent: Wednesday, October 26, 2011 8:18 AM
> To: Pete Resnick
> Cc: [email-masked]
> Subject: Re: draft-kucherawy-greylisting-bcp
> 
> Greylisting is not a protocol, according to any definition I know.

...as currently defined.  It seems this fledgling effort seeks to change that, and would introduce a new part of SMTP that could be so tested.

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Wed, 26 Oct 2011 21:37:02 -0700
Subject: RE: draft-kucherawy-greylisting-bcp
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Pete Resnick
> Sent: Wednesday, October 26, 2011 8:08 AM
> To: Keith Moore
> Cc: [email-masked]; [email-masked]
> Subject: Re: draft-kucherawy-greylisting-bcp
> 
> I think Keith has it pretty spot-on. BCPs are for documents where it
> makes no sense to talk about the concepts of (to quote 2026) "protocol,
> service, procedure, convention, or format", "particular methods of using
> a [technical specification]", "specify particular values or ranges, of
> TS parameters or subfunctions of a TS protocol that must be
> implemented", "interoperability", and "implementation and/or operational
> experience". 

Herein lies my confusion.  What is a Best Current Practices document if not one that talks about conventions, particular methods of using the thing a specification defines, [improved] interoperability, or relating of operational experience?

> My personal take is
> that BCPs should be reserved to guidelines for operators and
> administrators, statements of architectural principles, and
> documentation of procedures and operations of the IETF itself.

That sounds right, but some of it seems (to me) to conflict with the citations you made.  I suspect there's some (pardon the allusion) grey area in there that makes it hard, for me at least, to see clearly when something should be a BCP and something should be an AS.

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Wed, 26 Oct 2011 21:22:32 -0700
Subject: RE: new draft: draft-santos-smtpgrey-01
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of John C Klensin
> Sent: Wednesday, October 26, 2011 8:23 AM
> To: Tim Kehres; Hector; [email-masked]
> Subject: Re: new draft: draft-santos-smtpgrey-01
> 
> While it might not help significantly, something more generic
> would at least slightly and temporarily lower the odds of a
> different problem.   I really worry about saying to the spammer
> "this site does greylisting" and, if that happens enough, having
> the spammer respond (internally) "ah, it is greylisting and not
> some random temporary server unavailability, we know what to do
> about that".
> [...]

Fully agree.  I think the only possible mitigation would be a whitelist of domains that are known to be legitimate and want this information, and then the server returns these more informative replies since there's been some trust established.  In all other cases, a generic 4xy is returned when greylisting.  But that only adds further complexity.

Date: Wed, 26 Oct 2011 22:46:56 -0400
From: Hector <[email-masked]>
CC: [email-masked]
Subject: Re: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

John C Klensin wrote:

> tim kehres wrote:
> 
>> I've mentioned this to Hector directly, but my feeling is that
>> while an SMTP extension can be useful, I think tying it to
>> Greylisting is a bad idea.  If what we're looking for is a way
>> within a SMTP response to indicate a retry time for the
>> sending MTA, this can and SHOULD be a generic specification
>> and not one tied to a single application.   Something along
>> the lines of what is in the Atkins proposal seems more
>> reasonable to me
>> (http://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-con
>> trol/).
> 
> While it might not help significantly, something more generic
> would at least slightly and temporarily lower the odds of a
> different problem.   I really worry about saying to the spammer
> "this site does greylisting" and, if that happens enough, having
> the spammer respond (internally) "ah, it is greylisting and not
> some random temporary server unavailability, we know what to do
> about that".  
> 
> Independent of what advantages it offers people who are trying
> to cooperate in sending legitimate mail, to paraphrase Paul
> Vixie, giving spammers the information they need to become
> smarter and the incentives to do so are really not in our
> long-term best interests.
> 
> I have other doubts about both proposals but, if we are going to
> move forward with either, I think we need to understand and
> consider the tradeoffs very carefully.

I can only make one point here.

We already had a standard SMTP client/server framework where by a 
temporary negative reply code (4yz) is by design meant to drive 
clients to retry.  I don't believe it (the server) any correlations to 
when a MTA will retry or if it every will and can only base time would 
be related to the SMTP standard recommendations for retrying - hence 
an expectation there will be new attempts.  If we did not have the 
confidence that a 4yz will do what it meant to do, I don't believe we 
would have had the success over the years as a working mail system 
short of always accepting a payload and have a much different 
client/server paradigm of mail transport, acceptability and delivery.

My point is that adaptation has already taking place and any "bad guy" 
that wanted to get around the idea of "Greylisting" long had the 
ability to do so already simply by following the 4yz concept 
independent of any server reason for the rejection.

The real impact of Greylisting is the how its attribute of a "Blocking 
delay" has an affect on the theoretical and empirical protocol impact 
on the standard SMTP retry framework.

Greylisting address the market problem of senders that seem to be 
using a bulk volume of random addressing blasting with the goal of 
achieving by some measure of success, a 1% rate of accepted mail the 
first time.  This is a low margin high volume market and I venture due 
to cost reasons, it doesn't pay to waste too much time on transactions 
that are rejected.   But I am sure the  more professional outfits have 
the better eMarketing tools and these spammers  can not be controlled 
using temporary rejections simply because they do use standard SMTP 
software, probably the top free ones out there that already have all 
the intelligence built in.

Yet, lets suppose that the highlighting of a new standard extension 
exposing how clients can lower the impact of temporary rejection 
filtering methods, the question is what are the pros and cons here:

    PRO:  Good Guys Benefit
    CON:  Bad Guys Benefit

I just find it hard to believe that an "Standard' to exposed server 
hints is going to have any negative effect with bad guys adapting 
since it has already taken into account GL is ineffective against any 
client that will retry.

There is no illusion to the idea that the "secret entry ticket" is to 
retry.  Bad guys already know this.

This I believe the PRO of helping the good guys will outweigh any CONS 
with BAD GUYS suddenly getting an flash of genius insight:

     "Ah ha, so that is how I can get this 4yz sites to
      finally give me a 250. I have to retry again at a
      certain time.  Now if I can only figure out how to
      make this primitive software to retry......."

Personally, I never like to provide more information to SMTP transport 
level mail filtering rejections than needs be.  We had the debate here 
on the pros and cons of how much information is provide to assist 
support and/or users, better without "teaching" bad guys why they were 
rejected.

But I think what is different here is that we already naturally 
telling the "bad guy" or any client because GL looks at all anonymous 
senders, how to solve their problem using a standard 4yz response - 
"Try again later"

Just making the point that 4yz inherently presumes clients will retry 
and those that didn't already behave using what is already there to 
solve a problem, I find it hard to believe they will now change to 
begin reading the hint to finally make them to retry.

--
HLS

Date: Wed, 26 Oct 2011 13:33:35 -0400
From: Hector <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Pete Resnick <[email-masked]>, [email-masked], Dave CROCKER <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

Keith Moore wrote:
> One 'interesting' aspect of an interop test for greylisting is that it would only affect servers.  Greylisting is supposed to be backward compatible with standards-conforming clients.   What you would want to establish via interoperability testing is that a server that implements greylisting per the RFC doesn't degrade interoperability with existing, properly-implemented, clients.
> 
> I don't have a strong opinion of BCP vs. standards-track for this.  I think that if it's desirable to publish an RFC on this topic (which I doubt), either BCP or PS would suffice.   But I do think that if it's going to be published as PS, the document needs to define testing criteria for advancement.
> 

The only factor I see that can come close to degrade testing idea:

   - a compliant MTA does retry per SMTP,  but
   - before the Greylisting blocking time.

A key issue, if you need a reminder is the blind MTA world of varying 
degrees in blocking times and this is why we need a mechanical 
client/server automaton as oppose to a BCP which will end up being 
most likely a subjective "Rough Concensus" of the stronger IETF keys 
and participants making a "Best for all world" compromise offering of 
TIME that will not resolve the central issue:

     BCP TIME < GL Blocking Time => wasted attempts
     BCP TIME > GL Blocking Time => Delayed deliveries.

This has long been measured and of recent measure with internal 
testing of the proposal showing the proof of concept has been a 100% 
success in achieving a minimum of 2 attempts at precisely the delay 
time the server suggested.

--

Date: Wed, 26 Oct 2011 13:05:39 -0400
From: Hector <[email-masked]>
To: Tim Kehres <[email-masked]>
CC: [email-masked]
Subject: Re: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

While in the ideal sense, I fully agree a perfect generalized solution 
is better, in reality, I believe there are far too many issues here.

First, the primary focus of SMTPGREY is two folds:

    - Formalize the Greylisting Mail Filtering method in how it applies
      in SMTP and also impacted normal standard SMTP operations, and

    - Offer an advanced Greylisting feature to help lower the impact
      it does have on standard SMTP systems.

This is something people can grasp, the "smoke" is there.   In this 
proposal, there are very minor changes necessary to adopt it and it 
offers a helper "time factor" into a retry/queuing logic that already 
exist.  So the focus is on this one part of SMTP only.  Less than a 
few minutes of code changes and text response change.

On the other hand, when we begin to go general, again, ideal, but now 
the complexity grows, the scenarios grows - subjectively and the code 
changes required will grow as well.  For example, Atkin's proposal, if 
I reading it correctly, it leverages the wait= concept for multiple 
reasons and logics, not just one.  One includes an elegant idea to the 
Connection Sharing/Holding concern where there is a lot of CPU IDLE 
WASTE time being observed by these Connection Sharing/Holding clients, 
by using a WAIT=0 to tell the client to hang up.  He also introduces 
what I believe is something radical of using a wait= time for 
permanent rejection codes (55x), including positive responses (250). 
This is are very subjective ideas and in my view, beyond the scope of 
the central wasteful attempts/mail delays issues.  Its definitely more 
than what I seek.

Further, SMTP Client/server drives each other. The client sends 
commands to move the server into its states, the server drives the 
client with its reply codes.

Greylisting and the SMTPGREY does not deviate from this basic concept. 
  With Atkins, a server can issue a WAIT=0 to tell the client to hang 
up, but will large volumes and/or connection sharing agents do so?  In 
other words, harder to control and you may end up doing what we did; 
shortening the Idle time after the first transaction has taken place. 
  This has provided a huge savings in CPU IDLE waste time.

Lastly, with a full blown more generalize doc, changes are good it 
will not be a broad enough to cover all sizes of the market and would 
cater to the "Rough Consensus" of the larger systems.  It can end up 
where an implementator nitpicks parts of the generalize spec.  I am 
talking both as a server and client implementation, where you  taken a 
chance of being accused of non-compliancy if you use one part for a 
client need but not another part for a server need, etc.   Again, 
there is no guarantee one will listen to a WAIT=0 to make them hang up 
  - one way to guarantee that. Drop them, hence this part has lesser 
enforcement value.  With a specific focus with Greylisting and 4yz, 
there is more focus and control.

So yes, if it can be done in general with open mindedness to address 
all parties, great, true open house, etc, sure, but unfortunately, I 
have my doubts it can be done with losing the rest of my hair.

--
HLS

Tim Kehres wrote:
> 
> I've mentioned this to Hector directly, but my feeling is that while an 
> SMTP extension can be useful, I think tying it to Greylisting is a bad 
> idea.  If what we're looking for is a way within a SMTP response to 
> indicate a retry time for the sending MTA, this can and SHOULD be a 
> generic specification and not one tied to a single application.   
> Something along the lines of what is in the Atkins proposal seems more 
> reasonable to me 
> (http://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/).
> 
> Best Regards,
> 
> -- Tim
> 
> 
> ----- Original Message ----- From: "Hector" <[email-masked]>
> To: <[email-masked]>
> Sent: Wednesday, October 26, 2011 10:04 PM
> Subject: new draft: draft-santos-smtpgrey-01
> 
> 
>>
>> A New Internet-Draft is available from the on-line Internet-Drafts 
>> directories.
>>
>> Title           : SMTP Service Extension for Greylisting Operations
>> Author(s)       : Hector Santos
>>                           Evan Harris
>> Filename        : draft-santos-smtpgrey-01.txt
>> Pages           : 18
>> Date            : 2011-10-26
>>
>>    GREYLIST is a SMTP extension to formalize the widely supported
>>    Greylisting mail filtering method and to help support SMTP rejected
>>    transports by following a new formal structured 4yz server temporary
>>    rejection response by including a "retry=time-delay" tag string which
>>    SMTP clients can use to optimize the rescheduling of the mail
>>    delivery attempts.  With adoption, network overhead reduction in
>>    wasteful mail delivery attempts will be realized.
>>
>> A URL for this Internet-Draft is:
>>
>> text: http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-01.txt
>> html: http://tools.ietf.org/html/draft-santos-smtpgrey-01
>>
>> I would like to extend my grateful appreciation to the interested 
>> parties, onlist, offlist and lurking members of the Greylist-user 
>> support group, who provided input and assisted with the new draft. 
>> Thanks.
>>
>> ==
>> HLS
>>
>>
> 
> 
> 

Subject: Re: draft-kucherawy-greylisting-bcp
From: Keith Moore <[email-masked]>
Date: Wed, 26 Oct 2011 12:51:18 -0400
Cc: <[email-masked]>, Dave CROCKER <[email-masked]>, "[email-masked]" <[email-masked]>
To: Pete Resnick <[email-masked]>
Sender: [email-masked]

One 'interesting' aspect of an interop test for greylisting is that it would only affect servers.  Greylisting is supposed to be backward compatible with standards-conforming clients.   What you would want to establish via interoperability testing is that a server that implements greylisting per the RFC doesn't degrade interoperability with existing, properly-implemented, clients.

I don't have a strong opinion of BCP vs. standards-track for this.  I think that if it's desirable to publish an RFC on this topic (which I doubt), either BCP or PS would suffice.   But I do think that if it's going to be published as PS, the document needs to define testing criteria for advancement.

Keith

Date: Wed, 26 Oct 2011 11:16:24 -0500
From: Pete Resnick <[email-masked]>
To: <[email-masked]>
CC: Dave CROCKER <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

On 10/26/11 10:42 AM, Dave CROCKER wrote:
> On 10/26/2011 5:33 PM, Pete Resnick wrote:
>> On 10/26/11 10:18 AM, Dave CROCKER wrote:
>>> Greylisting is not a protocol, according to any definition I know.
>>>
>>> There is no way to test "interoperability".
>>
>> Rubbish. I can assure you that if someone implemented greylisting in 
>> such a way
>> that "legitimate" mail stopped being delivered, because they set the 
>> greylist
>> timeout too long or...
>
> You have just defined an interoperability criterion that includes all 
> sources of software misbehavior, daemon sleep times, and anything else 
> that can cause delays.

I notice you conveniently snipped my example of using a status code that 
caused trouble. But yes, timeouts are commonly put into standards track 
documents, and reasonably so. Implementation parameters do cause 
interoperability problems and many of those are observable from the net. 
I have no idea how you can claim these aren't "interoperability criteria".

Let's go back to my initial claims: If it implementation experience 
would cause changes to the instructions we give in a document to 
implementers, then it deserves to be on the standards track. If it's a 
"protocol, service, procedure, convention, or format" [2026, 3.1], it 
belongs in a technical specification. If it gives implementation 
instructions (values for parameters, how the protocol ought be 
implemented, etc.), then it belongs in an applicability statement. Both 
TSs and ASs go on the standards track.

>> Greylisting is far more obviously a protocol with testable 
>> interoperability than is, say, RFC 5234 or 5322.

[To clarify: I am not committed to greylisting being a protocol. I'm 
just saying it looks more like a protocol than do 5234 or 5322. It is 
the claim that there is no way to test interoperability that I reject..]

> 5322 and other format specs explicitly entail parsing and processing 
> (and in the case of From: and Reply-to:, replying) by the remote 
> engine.  That's essential interoperability.

How does one "test interoperability" in the case of parsing that is 
going on in a local implementation like 5322? How is that different than 
greylisting? And for ANBF...?

> In other words, formats are inherently bilateral.  Greylisting is 
> unilateral.

Formats are bilateral, but only in the sense that both sides have a 
common understanding of the semantics of the parsed format. Greylisting 
is bilateral for those who actually participate: That is, if I receive 
4xx status code, I know to wait a while and resend. Successful delivery 
happens if we both do our parts correctly. It is certainly the case that 
if I am a poorly implemented SMTP client, I will not try again and my 
mail will not be delivered (the point of the greylisting), but surely 
greylisting still "specifies how, and under what circumstances" SMTP is 
"applied to support a particular Internet capability".  [2026, 3.2]

> The interoperability mechanism relevant to greylisting is the 
> /existing/ set of SMTP mechanisms.  Greylisting is an indirect 
> invocation of those mechanisms.

I agree. See 2026, section 3.2.

> Again:  since my view is rubbish, please define an interoperability 
> test for it.

(I don't think I am making any statement about *your* *view*. I 
apologize for this coming across as a personal attack. I used "rubbish" 
only to mean, "I strongly disagree with the statement that there is no 
way to test interoperability". I certainly did not intend any offense.)

Again, please re-read the sections of 2026 regarding ASs. The 
interoperability test is against SMTP, not greylisting itself. 
Greylisting is a way of implementing SMTP with certain outcomes. If 
implementation choices turn out to be wrong (which we can determine by 
testing the interoperability of SMTP implementations that use 
greylisting), we update the AS describing greylisting. That's what the 
standards track is all about.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Wed, 26 Oct 2011 16:50:20 +0100
From: Paul Smith <[email-masked]>
To: John C Klensin <[email-masked]>
CC: Tim Kehres <[email-masked]>, Hector <[email-masked]>, [email-masked]
Subject: Re: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

On 26/10/2011 16:22, John C Klensin wrote:
> Independent of what advantages it offers people who are trying to 
> cooperate in sending legitimate mail, to paraphrase Paul Vixie, giving 
> spammers the information they need to become smarter and the 
> incentives to do so are really not in our long-term best interests.

The reason why I don't think this is an issue, is that spammers already 
have the information they need to get around greylisting. The 
'retry=<time>' data won't give them any information they need to be able 
to get their messages through. However, the fact is that, for now at 
least, many are not doing anything about countering greylisting. When 
they do decide to do something about it, the 'retry=<time>' data won't 
be vital to them, or even make their job significantly easier.

Having said that, I'd prefer the 'retry=<time>' extension to be generic, 
and not linked to greylisting, except by coincidence.

Subject: Re: new draft: draft-santos-smtpgrey-01
From: Keith Moore <[email-masked]>
Date: Wed, 26 Oct 2011 11:47:49 -0400
Cc: [email-masked]
To: [email-masked]
Sender: [email-masked]

On Oct 26, 2011, at 11:35 AM, Dave CROCKER wrote:

>> I really worry about saying to the spammer
>> "this site does greylisting" and, if that happens enough, having
>> the spammer respond (internally) "ah, it is greylisting and not
>> some random temporary server unavailability, we know what to do
>> about that".
> ...
>> giving spammers the information they need to become
>> smarter and the incentives to do so are really not in our
>> long-term best interests.
> 
> 
> Greylisting only works due to spammer laziness.  It is only a barrier for the trivial spamming engines.
> 
> As soon as a spammer is motivated to do interesting work, they can and do work around greylisting easily.
> 
> So this particular context is not one that warrants worrying about "giving" information to spammers.
> 
> The reason that greylisting continues to be useful is that there happens to be a significant portion of the spamming world that does, in fact, use trivial engines.

I consider it at least plausible that the reason more spammers haven't worked around greylisting is that it's currently below their radar.  They're probably aware that it exists, but either they're not aware of how much it affects their delivery rates, or they are aware and it doesn't affect those rates enough for them to worry about.

Publishing an RFC about greylisting could change this in two ways: (1) it would make spammers more aware of greylisting, (2) it would encourage wider adoption of greylisting and thus give spammers more incentive to defeat it for everybody.

So... not a good idea, IMO.

Keith

Date: Wed, 26 Oct 2011 11:44:27 -0400
From: John C Klensin <[email-masked]>
To: Pete Resnick <[email-masked]>, Keith Moore <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

--On Wednesday, October 26, 2011 10:08 -0500 Pete Resnick
<[email-masked]> wrote:

>...
> I think Keith has it pretty spot-on. BCPs are for documents
> where it makes no sense to talk about the concepts of (to
> quote 2026) "protocol, service, procedure, convention, or
> format", "particular methods of using a [technical
> specification]", "specify particular values or ranges, of TS
> parameters or subfunctions of a TS protocol that must be
> implemented", "interoperability", and "implementation and/or
> operational experience". If it can be deployed and you get get
> incremental implementation experience, it should be on the
> Standards Track; it shouldn't be a second class citizen of
> being a BCP. My personal take is that BCPs should be reserved
> to guidelines for operators and administrators, statements of
> architectural principles, and documentation of procedures and
> operations of the IETF itself.

Agree, with one qualification.   I don't see BCPs as "second
class citizens" even though some of the things that have been
published as BCPs push the whole category in that direction.
But, especially since we now have only a two-step standards
track, if the effect of a document is to modify or give
supplemental instructions about a technical specification or
protocol (or any of the other phrases Pete quotes above), then
we probably owe it to ourselves and the community to use that
two-step mechanism to increase the odds that ideas are exposed
to the wider community before we finalize them.  

Dave wrote:

> Greylisting is not a protocol, according to any definition I
> know.

It isn't.  Depending on one's point of view, it is either a
clever and unanticipated way to use a particular feature/
characteristic of a protocol specification or it is a horrible
abuse of a protocol feature that was intended for another
purpose.  Either way, it extends the originally-intended use of
the protocol.  And, not only is interoperability testing (at
scale, not just two implementations in the lab) relevant to
determine the ways in which the technique interacts with
implementations that are not aware of it, its effects on the
network are in need of continuing and ongoing evaluation.

     john

Subject: Re: draft-kucherawy-greylisting-bcp
From: Steve Atkins <[email-masked]>
Date: Wed, 26 Oct 2011 08:43:57 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 26, 2011, at 8:33 AM, Pete Resnick wrote:

> 
> On 10/26/11 10:18 AM, Dave CROCKER wrote:
> 
>> On 10/26/2011 5:08 PM, Pete Resnick wrote:
>> 
>>> If it can be deployed and you
>>> get get incremental implementation experience, it should be on the Standards
>>> Track; it shouldn't be a second class citizen of being a BCP.
>> 
>> Greylisting is not a protocol, according to any definition I know.
>> 
>> There is no way to test "interoperability".
> 
> Rubbish. I can assure you that if someone implemented greylisting in such a way that "legitimate" mail stopped being delivered, because they set the greylist timeout too long or because they used a status code that everyone choked on or some other such thing that was underspecified in the RFC, people would say that the implemenation was not interoperating in the mailing transport/delivery system and that implementation experience dictated changes to the specification in order to improve future implementations (assuming that the spec failed to specify the correct parameters). Greylisting is far more obviously a protocol with testable interoperability than is, say, RFC 5234 or 5322.

ESMTP is the protocol. Greylisting itself has no protocol (it's just a local policy for a mailserver using the ESMTP protocol) so there's no way to test interoperability.

You can test ESMTP interoperability, you can certainly test interoperabilty between ESMTP peers that are implementing greylisting. But you can't test *greylisting* interoperability, as there's no protocol there.

Cheers,
  Steve

Date: Wed, 26 Oct 2011 17:42:42 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Pete Resnick <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

On 10/26/2011 5:33 PM, Pete Resnick wrote:
> On 10/26/11 10:18 AM, Dave CROCKER wrote:
>> Greylisting is not a protocol, according to any definition I know.
>>
>> There is no way to test "interoperability".
>
> Rubbish. I can assure you that if someone implemented greylisting in such a way
> that "legitimate" mail stopped being delivered, because they set the greylist
> timeout too long or...

You have just defined an interoperability criterion that includes all sources of 
software misbehavior, daemon sleep times, and anything else that can cause delays.

>    Greylisting is far more obviously a protocol
> with testable interoperability than is, say, RFC 5234 or 5322.

5322 and other format specs explicitly entail parsing and processing (and in the 
case of From: and Reply-to:, replying) by the remote engine.  That's essential 
interoperability.

In other words, formats are inherently bilateral.  Greylisting is unilateral.

The interoperability mechanism relevant to greylisting is the /existing/ set of 
SMTP mechanisms.  Greylisting is an indirect invocation of those mechanisms.

Again:  since my view is rubbish, please define an interoperability test for it.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: "Tim Kehres" <[email-masked]>
To: <[email-masked]>, "Pete Resnick" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Date: Wed, 26 Oct 2011 23:39:47 +0800
Sender: [email-masked]

> Greylisting is not a protocol, according to any definition I know.
>
> There is no way to test "interoperability".

Another reason IMHO it should not be tied to any SMTP Extension proposal. 
It is more a methodology subject to local site requirements than a protocol. 
The proposed BCP outline looks like a good start in at least documenting the 
issues (greylisting).

Best Regards,

-- Tim 

Date: Wed, 26 Oct 2011 17:35:50 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: Re: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

>  I really worry about saying to the spammer
> "this site does greylisting" and, if that happens enough, having
> the spammer respond (internally) "ah, it is greylisting and not
> some random temporary server unavailability, we know what to do
> about that".
...
>giving spammers the information they need to become
> smarter and the incentives to do so are really not in our
> long-term best interests.

Greylisting only works due to spammer laziness.  It is only a barrier for the 
trivial spamming engines.

As soon as a spammer is motivated to do interesting work, they can and do work 
around greylisting easily.

So this particular context is not one that warrants worrying about "giving" 
information to spammers.

The reason that greylisting continues to be useful is that there happens to be a 
significant portion of the spamming world that does, in fact, use trivial engines.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: "Tim Kehres" <[email-masked]>
To: "John C Klensin" <[email-masked]>, "Hector" <[email-masked]>, <[email-masked]>
Subject: Re: new draft: draft-santos-smtpgrey-01
Date: Wed, 26 Oct 2011 23:35:30 +0800
Sender: [email-masked]

> While it might not help significantly, something more generic
> would at least slightly and temporarily lower the odds of a
> different problem.

A generic solution would also mean that we don't have to design the same 
solution multiple times for different applications.   A few cases were 
brought up earlier such as if a site needs to be unavailable for a 
pre-determined period of time (site maintenance, etc).   I'm sure with some 
thought we'd be able to come up with more, and as time goes on, additional 
applications will probably also find this of use.

> I really worry about saying to the spammer
> "this site does greylisting" and, if that happens enough, having
> the spammer respond (internally) "ah, it is greylisting and not
> some random temporary server unavailability, we know what to do
> about that".

Another good reason NOT to tie it in with greylisting.   In fact, my 
preference is to never use it for greylisting for precisely the reasons you 
cite above.  But admins on other sites may feel differently - so for those 
that choose to reply in such a manner, it would be nice if the format was 
standardized.

> Independent of what advantages it offers people who are trying
> to cooperate in sending legitimate mail, to paraphrase Paul
> Vixie, giving spammers the information they need to become
> smarter and the incentives to do so are really not in our
> long-term best interests.
>
> I have other doubts about both proposals but, if we are going to
> move forward with either, I think we need to understand and
> consider the tradeoffs very carefully.

Absolutely!   As well as any potential abuse scenarios that could be 
utilized.

Best Regards,

-- Tim 

Date: Wed, 26 Oct 2011 10:33:37 -0500
From: Pete Resnick <[email-masked]>
To: <[email-masked]>
CC: Dave CROCKER <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

On 10/26/11 10:18 AM, Dave CROCKER wrote:

> On 10/26/2011 5:08 PM, Pete Resnick wrote:
>
>> If it can be deployed and you
>> get get incremental implementation experience, it should be on the 
>> Standards
>> Track; it shouldn't be a second class citizen of being a BCP.
>
> Greylisting is not a protocol, according to any definition I know.
>
> There is no way to test "interoperability".

Rubbish. I can assure you that if someone implemented greylisting in 
such a way that "legitimate" mail stopped being delivered, because they 
set the greylist timeout too long or because they used a status code 
that everyone choked on or some other such thing that was underspecified 
in the RFC, people would say that the implemenation was not 
interoperating in the mailing transport/delivery system and that 
implementation experience dictated changes to the specification in order 
to improve future implementations (assuming that the spec failed to 
specify the correct parameters). Greylisting is far more obviously a 
protocol with testable interoperability than is, say, RFC 5234 or 5322.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Wed, 26 Oct 2011 11:22:52 -0400
From: John C Klensin <[email-masked]>
To: Tim Kehres <[email-masked]>, Hector <[email-masked]>, [email-masked]
Subject: Re: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

--On Wednesday, October 26, 2011 22:43 +0800 Tim Kehres
<[email-masked]> wrote:

> I've mentioned this to Hector directly, but my feeling is that
> while an SMTP extension can be useful, I think tying it to
> Greylisting is a bad idea.  If what we're looking for is a way
> within a SMTP response to indicate a retry time for the
> sending MTA, this can and SHOULD be a generic specification
> and not one tied to a single application.   Something along
> the lines of what is in the Atkins proposal seems more
> reasonable to me
> (http://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-con
> trol/).

While it might not help significantly, something more generic
would at least slightly and temporarily lower the odds of a
different problem.   I really worry about saying to the spammer
"this site does greylisting" and, if that happens enough, having
the spammer respond (internally) "ah, it is greylisting and not
some random temporary server unavailability, we know what to do
about that".  

Independent of what advantages it offers people who are trying
to cooperate in sending legitimate mail, to paraphrase Paul
Vixie, giving spammers the information they need to become
smarter and the incentives to do so are really not in our
long-term best interests.

I have other doubts about both proposals but, if we are going to
move forward with either, I think we need to understand and
consider the tradeoffs very carefully.

   john

Date: Wed, 26 Oct 2011 17:18:06 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Pete Resnick <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

On 10/26/2011 5:08 PM, Pete Resnick wrote:
 >      If it can be deployed and you
> get get incremental implementation experience, it should be on the Standards
> Track; it shouldn't be a second class citizen of being a BCP.

Greylisting is not a protocol, according to any definition I know.

There is no way to test "interoperability".

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 26 Oct 2011 10:08:29 -0500
From: Pete Resnick <[email-masked]>
To: Keith Moore <[email-masked]>
CC: <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

I think Keith has it pretty spot-on. BCPs are for documents where it 
makes no sense to talk about the concepts of (to quote 2026) "protocol, 
service, procedure, convention, or format", "particular methods of using 
a [technical specification]", "specify particular values or ranges, of 
TS parameters or subfunctions of a TS protocol that must be 
implemented", "interoperability", and "implementation and/or operational 
experience". If it can be deployed and you get get incremental 
implementation experience, it should be on the Standards Track; it 
shouldn't be a second class citizen of being a BCP. My personal take is 
that BCPs should be reserved to guidelines for operators and 
administrators, statements of architectural principles, and 
documentation of procedures and operations of the IETF itself.

pr

On 10/26/11 7:12 AM, Keith Moore wrote:
> IMO, BCPs are a way for IETF to endorse desirable practices for which the 2026 advancement criteria (e.g. interoperability tests) are not applicable.
>
> On Oct 26, 2011, at 4:24 AM, Dave CROCKER wrote:
>
>    
>> Pete,
>>
>> I don't understand, either.  While it lists a /set/ of alternative or complementary best practices, it /is/ about best practices.  Please explain.
>>
>> d/
>>
>> On 10/25/2011 7:12 PM, Pete Resnick wrote:
>>      
>>> (a) This shouldn't be labeled as a BCP. It's an A/S Standards Track draft if
>>> ever there was one.
>>> (b) Given this and the extension, is it worth spinning up a Greylisting WG?
>>>
>>> pr
>>>
>>> On 10/23/11 3:40 PM, Murray S. Kucherawy wrote:
>>>        
>>>> Maybe a new thread Subject: would help here...
>>>>
>>>> Given the interest, I've started putting together a BCP. It's now posted as
>>>> draft-kucherawy-greylisting-bcp; comments welcome (and thanks to those of you
>>>> that already did). It is only an outline right now, and I'll fill it in as
>>>> people make suggestions about topics it should cover and/or start providing
>>>> text to include.
>>>>
>>>> It doesn't touch the SMTP extension people are discussing. Someone else can
>>>> champion that one if so inclined. That shouldn't go in a BCP because it
>>>> doesn't even exist yet. This document will only describe current and historic
>>>> practices.
>>>>
>>>> Maybe we can get it on the docket for Taipei. I'll send out some friendly pings.
>>>>
>>>> -MSK
>>>>          

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

From: "Tim Kehres" <[email-masked]>
To: "Hector" <[email-masked]>, <[email-masked]>
Subject: Re: new draft: draft-santos-smtpgrey-01
Date: Wed, 26 Oct 2011 22:43:16 +0800
Sender: [email-masked]

I've mentioned this to Hector directly, but my feeling is that while an SMTP 
extension can be useful, I think tying it to Greylisting is a bad idea.  If 
what we're looking for is a way within a SMTP response to indicate a retry 
time for the sending MTA, this can and SHOULD be a generic specification and 
not one tied to a single application.   Something along the lines of what is 
in the Atkins proposal seems more reasonable to me 
(http://datatracker.ietf.org/doc/draft-atkins-smtp-traffic-control/).

Best Regards,

-- Tim

----- Original Message ----- 
From: "Hector" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, October 26, 2011 10:04 PM
Subject: new draft: draft-santos-smtpgrey-01

>
> A New Internet-Draft is available from the on-line Internet-Drafts 
> directories.
>
> Title           : SMTP Service Extension for Greylisting Operations
> Author(s)       : Hector Santos
>                           Evan Harris
> Filename        : draft-santos-smtpgrey-01.txt
> Pages           : 18
> Date            : 2011-10-26
>
>    GREYLIST is a SMTP extension to formalize the widely supported
>    Greylisting mail filtering method and to help support SMTP rejected
>    transports by following a new formal structured 4yz server temporary
>    rejection response by including a "retry=time-delay" tag string which
>    SMTP clients can use to optimize the rescheduling of the mail
>    delivery attempts.  With adoption, network overhead reduction in
>    wasteful mail delivery attempts will be realized.
>
> A URL for this Internet-Draft is:
>
> text: http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-01.txt
> html: http://tools.ietf.org/html/draft-santos-smtpgrey-01
>
> I would like to extend my grateful appreciation to the interested parties, 
> onlist, offlist and lurking members of the Greylist-user support group, 
> who provided input and assisted with the new draft. Thanks.
>
> ==
> HLS
>
> 

Date: Wed, 26 Oct 2011 10:04:57 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: new draft: draft-santos-smtpgrey-01
Sender: [email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts 
directories.

	Title           : SMTP Service Extension for Greylisting Operations
	Author(s)       : Hector Santos
                           Evan Harris
	Filename        : draft-santos-smtpgrey-01.txt
	Pages           : 18
	Date            : 2011-10-26

    GREYLIST is a SMTP extension to formalize the widely supported
    Greylisting mail filtering method and to help support SMTP rejected
    transports by following a new formal structured 4yz server temporary
    rejection response by including a "retry=time-delay" tag string which
    SMTP clients can use to optimize the rescheduling of the mail
    delivery attempts.  With adoption, network overhead reduction in
    wasteful mail delivery attempts will be realized.

A URL for this Internet-Draft is:

text: http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-01.txt
html: http://tools.ietf.org/html/draft-santos-smtpgrey-01

I would like to extend my grateful appreciation to the interested 
parties, onlist, offlist and lurking members of the Greylist-user 
support group, who provided input and assisted with the new draft. 
Thanks.

==
HLS

Subject: Re: draft-kucherawy-greylisting-bcp
From: Keith Moore <[email-masked]>
Date: Wed, 26 Oct 2011 08:12:37 -0400
Cc: Pete Resnick <[email-masked]>, "[email-masked]" <[email-masked]>
To: [email-masked]
Sender: [email-masked]

IMO, BCPs are a way for IETF to endorse desirable practices for which the 2026 advancement criteria (e.g. interoperability tests) are not applicable.

On Oct 26, 2011, at 4:24 AM, Dave CROCKER wrote:

> 
> Pete,
> 
> I don't understand, either.  While it lists a /set/ of alternative or complementary best practices, it /is/ about best practices.  Please explain.
> 
> d/
> 
> On 10/25/2011 7:12 PM, Pete Resnick wrote:
>> 
>> (a) This shouldn't be labeled as a BCP. It's an A/S Standards Track draft if
>> ever there was one.
>> (b) Given this and the extension, is it worth spinning up a Greylisting WG?
>> 
>> pr
>> 
>> On 10/23/11 3:40 PM, Murray S. Kucherawy wrote:
>>> Maybe a new thread Subject: would help here...
>>> 
>>> Given the interest, I've started putting together a BCP. It's now posted as
>>> draft-kucherawy-greylisting-bcp; comments welcome (and thanks to those of you
>>> that already did). It is only an outline right now, and I'll fill it in as
>>> people make suggestions about topics it should cover and/or start providing
>>> text to include.
>>> 
>>> It doesn't touch the SMTP extension people are discussing. Someone else can
>>> champion that one if so inclined. That shouldn't go in a BCP because it
>>> doesn't even exist yet. This document will only describe current and historic
>>> practices.
>>> 
>>> Maybe we can get it on the docket for Taipei. I'll send out some friendly pings.
>>> 
>>> -MSK
>>> 
>>> 
>> 
> 
> -- 
> 
>  Dave Crocker
>  Brandenburg InternetWorking
>  bbiw.net
> 

Date: Wed, 26 Oct 2011 10:24:15 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Pete Resnick <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

Pete,

I don't understand, either.  While it lists a /set/ of alternative or 
complementary best practices, it /is/ about best practices.  Please explain.

d/

On 10/25/2011 7:12 PM, Pete Resnick wrote:
>
> (a) This shouldn't be labeled as a BCP. It's an A/S Standards Track draft if
> ever there was one.
> (b) Given this and the extension, is it worth spinning up a Greylisting WG?
>
> pr
>
> On 10/23/11 3:40 PM, Murray S. Kucherawy wrote:
>> Maybe a new thread Subject: would help here...
>>
>> Given the interest, I've started putting together a BCP. It's now posted as
>> draft-kucherawy-greylisting-bcp; comments welcome (and thanks to those of you
>> that already did). It is only an outline right now, and I'll fill it in as
>> people make suggestions about topics it should cover and/or start providing
>> text to include.
>>
>> It doesn't touch the SMTP extension people are discussing. Someone else can
>> champion that one if so inclined. That shouldn't go in a BCP because it
>> doesn't even exist yet. This document will only describe current and historic
>> practices.
>>
>> Maybe we can get it on the docket for Taipei. I'll send out some friendly pings.
>>
>> -MSK
>>
>>
>

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: "Murray S. Kucherawy" <[email-masked]>
To: Pete Resnick <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Date: Wed, 26 Oct 2011 00:28:23 -0700
Subject: RE: draft-kucherawy-greylisting-bcp
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Pete Resnick [mailto:[email-masked]]
> Sent: Tuesday, October 25, 2011 10:13 AM
> To: Murray S. Kucherawy
> Cc: [email-masked]
> Subject: Re: draft-kucherawy-greylisting-bcp
> 
> (a) This shouldn't be labeled as a BCP. It's an A/S Standards Track
> draft if ever there was one.

I guess I'm still unclear on the difference.  I'm fine with either status, especially if someone can point me at a good definition so I understand why one's better than the other.

> (b) Given this and the extension, is it worth spinning up a Greylisting WG?

IMHO, there's not enough support for the extension as a Standards Track item.  I imagine if there's a group that wants to pursue an Experimental one, that could happen through the ISE.

Given that, I (respectfully!) suggest this could be handled by APPSAWG.  I've already requested a slot for it in Taipei.

Date: Tue, 25 Oct 2011 19:48:51 -0400
From: Hector <[email-masked]>
To: Frank Ellermann <[email-masked]>
CC: [email-masked]
Subject: retry=min/max syntax
Sender: [email-masked]

Frank Ellermann wrote:
> On 25 October 2011 19:28, Hector wrote:
> 
>> I am not expert with ABNF language. Can you tell
>> me what would be is correct here?
> 
> Better, I can tell you how I'd check it:  Visit the
> page <URL:http://www.fenron.com/~fenner/abnf.cgi>.
> In the form enter...

Oh COOL!

> retrytime  = "retry=" min-secs ["-" max-secs]
> 
>> min-secs   = 1*digits   ; minimum blocking time
>> max-secs   = 1*digits   ; optional retry expiration
> 
> Makes sense if the server has a plan *when* it intends
> to purge pending triplets.  And <max-secs> certainly
> could need more than 6 digits; and my suggestion that
> 6 digits fit into an ushort works only in a parallel
> universe with "ternary bits" where they'll use Ipv4
> forever squeezing three digits 0, 1, and "" (void) in
> a single bit...
> 
> -Frank (needs more coffee)

Me too. :)

We could make it hex! This seem to be accepted by the ABNF tester. 
Open Suggestion only of supporting a 2nd retry=min/max format.

Reply-Line  = ( Reply-Code [ SP textstring [SP retrytime] ] CRLF ) /
                  ( Reply-Code "-" [ SP textstring ] CRLF
                    *( Reply-Code "-" [ SP textstring ] CRLF )
                    Reply-Code [ SP textstring [SP retrytime ] ] CRLF )

textstring  = 1*(%x09 / %x20-7E) ; HT, SP, Printable US-ASCII
retrytime   = ( "retry=" [days "-"] hours ":" minutes ":" seconds ) /
               ( "retry="  blocktime ["/" expiretime ] )
days        = 2DIGIT   ; 00-99
hours       = 2DIGIT   ; 00-23
minutes     = 2DIGIT   ; 00-59
seconds     = 2DIGIT   ; 00-59
blocktime   = 8*HEXDGT ; blocking time in secs
expiretime  = 8*HEXDGT ; optional expire time in secs
Reply-Code  = "421" / "450" / "451"

--
HLS

Date: Tue, 25 Oct 2011 15:48:14 -0400
From: Hector <[email-masked]>
To: Frank Ellermann <[email-masked]>
CC: [email-masked]
Subject: retry=min-max [was Re: I-D SMTP Extension for Greylisting]
Sender: [email-masked]

Frank Ellermann wrote:
> On 25 October 2011 20:31, Hector wrote:
> 
>> retrytime  = "retry=" min-secs ["-" max-secs]
> 
>> min-secs   = 1*digits   ; minimum blocking time
>> max-secs   = 1*digits   ; optional retry expiration
> 
> Makes sense if the server has a plan *when* it intends
> to purge pending triplets.

I will venture most Greylisting Servers had followed Harris's original 
whitepaper which included a record expiration offering a time window 
for the MTA to retry.

    http://projects.puremagic.com/greylisting/whitepaper.html

    Implementation Specification

    In order to implement the Greylisting method, we will use some
    form of database to hold a few pieces of information about a
    specific mail relationship that is keyed off of the triplet 
described above:

    The time that the triplet was first seen (record create time)
    The time that the blocking of this triplet will expire
    The time that the record itself will expire (for aging old records)
    The number of delivery attempts that have been blocked
    The number of emails we have successfully passed

Our particular GL server stock setting covers the top three items:

   Expire.block    55         # secs to block greylisted senders
   Expire.record   2880       # mins to expire recording
   Expire.pack     96         # hours to delete records and pack DB

The initial block is 55 seconds, 2 days of monitoring it and 4 days 
when the backend automatically packs the database which can grow 
pretty fast when 60% of the senders are "single shot" senders.  What 
is fascination is that 60% is pretty consistent with most of the 
customers providing their logs for review.

I guess, technically, given a proposed SMTP Extension for Greylisting, 
the tracking time limit (expire.record) of a triplet SHOULD match 
RFC5321 recommendation of allowing a MTA a 4-5 days time window to 
delivery the mail.

--
HLS

From: Frank Ellermann <[email-masked]>
Date: Tue, 25 Oct 2011 20:43:29 +0200
Subject: Re: I-D SMTP Extension for Greylisting
To: Hector <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 25 October 2011 20:31, Hector wrote:

> retrytime �= "retry=" min-secs ["-" max-secs]

> min-secs � = 1*digits � ; minimum blocking time
> max-secs � = 1*digits � ; optional retry expiration

Makes sense if the server has a plan *when* it intends
to purge pending triplets.  And <max-secs> certainly
could need more than 6 digits; and my suggestion that
6 digits fit into an ushort works only in a parallel
universe with "ternary bits" where they'll use Ipv4
forever squeezing three digits 0, 1, and "" (void) in
a single bit...

-Frank (needs more coffee)

Date: Tue, 25 Oct 2011 14:31:46 -0400
From: Hector <[email-masked]>
To: Frank Ellermann <[email-masked]>
CC: [email-masked]
Subject: Re: I-D SMTP Extension for Greylisting
Sender: [email-masked]

Frank Ellermann wrote:
> On 24 October 2011 16:36, Hector wrote:
> 
>> � seconds � � = 2DIGIT � ; 00-59
>> � totalsecs � = 1*(%d48-57) ; total seconds to delay
> 
>> If this suggestion for a 2nd syntax format is
>> acceptable, is the ABNF correct?
> 
> Ignoring the premise, you can use 1*DIGIT instead of
> those decimal constructs ;-)  If you want to specify
> <totalsecs> details you could suppress leading zeros
> and add an upper limit -- the very odd 99-23:59:59
> maximum yields 1*7DIGIT if I got this right, please
> check it if you want to use it.
> 
> And trim this to 1*6DIGIT if they let you, IMO 9 days
> is already far outside of anything remotely related
> to SMTP.

I am personally liking more and more Claus Assmann's suggestion of 
using a MIN/MAX concept and to keep it simple a syntax of:

    retry=min-max

with the ABNF:

    retrytime  = "retry=" min-secs ["-" max-secs]

    min-secs   = 1*digits   ; minimum blocking time
    max-secs   = 1*digits   ; optional retry expiration time

The parsing is simple, no technical limits (only policy based) and it 
can help adoption by operators who can just simply change the response 
language resource files/configuration and don't need to wait for 
developers to alter their greylisting code to calculate a 
[DD-]HH:MM:SS format.

I think I will add it and see how it flies.

--
HLS

From: Frank Ellermann <[email-masked]>
Date: Tue, 25 Oct 2011 20:26:57 +0200
Subject: Re: I-D SMTP Extension for Greylisting
To: Hector <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 25 October 2011 19:28, Hector wrote:

> I am not expert with ABNF language. Can you tell
> me what would be is correct here?

Better, I can tell you how I'd check it:  Visit the
page <URL:http://www.fenron.com/~fenner/abnf.cgi>.
In the form enter...

retry = 1*6DIGIT

...and more lines (your complete collected ABNF).
Click "Senden" (my Chrome configured for en-GB
before en before en-US before de is at it again,
hopefully you'll get "send" as it should be... :-)

If the result is "green" claiming that DIGIT is
undefined and that retry is unused it's good ABNF.
You can "import" a (pseudo-) definition for DIGIT:

DIGIT = <defined in STD 68>
retry = 1*6DIGIT

It's just a way to specify 1..6 digits, no or more than 6 digits would
be a syntax error.  Small bonus:

Implementors would know that say an unsigned short
would be good enough to store this value (and about
two decades ago I'd know that I have to check the
compiler option for sizeof short to be sure... ;-)

> With retry=seconds, there would be no technical
> upper limit

If you want no upper limit simply kill the 6 in 1*6.
Seconds should be a good choice for applications,
even if minutes would better match what you want.

-Frank

Date: Tue, 25 Oct 2011 14:19:11 -0400
From: Hector <[email-masked]>
To: Patrik Faltstrom <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Hector wrote:

>> I.e. why does it help that someone give the hint "gray listed in 5 
>> minutes" but the gray listing is 7, instead of today when no hints are 
>> done at all (in a parseable way)?
> 
> I personally do not think this is a reasonable objection for the reasons 
> stated above. An intentional lie doesn't do anything more than to hurt 
> yourself and probably more so than the MTA.
> 
> Nonetheless, it probably should be noted very strongly in the proposed 
> spec that a MTA be watchful of liars  who are wasting everyone's times 
> with false retry=time hints.

For the next draft, section 3.1.2 SMTP Client Implementation section 
now has SHOULD|MUST|MAY [NOT] language.  I just added the last 
paragraph regarding erroneous retry=time hints by servers.

    3.1.2.  SMTP Client Implementation

    ...

    NEEDS REVIEW: If a SMTP server offers a retry=hint which results in a
    wasted 2nd attempt and requires additional attempts, the SMTP client
    MAY begin to ignore the server's retry=hints after the 2nd wasted
    retry. The SMTP client implementation can decide what limits to 
place
    on honoring retry=time hints and wasted attempts it provides.

Does this satisfy your concern?

The goal of the SMTP greylist extension is to address the current 
wasted attempts and delayed delivery times by offering a method to 
move the waste to the lowest impact boundary which is expected to be 
two attempts at most by leveraging server rety=time hint information. 
   If the 2nd attempt fails, that could mean other reasons beside a 
"Intentional Liar."  I believe the MTA will need to watchful of the 
rescheduling adjustments using retry=hint do not actually cause 
additional attempts before two. While its possible that a liar can 
exist, I don't see any value why it would exist unless it was 100% 
malicious in nature, i.e. A Greylist Honeypot per se or any server for 
that matter that will might also reject no matter way.

Regardless of the server reasons, the MTA using the retry=hint can 
simply limit it's usage to the 2nd attempt only and if that doesn't 
help, than its back to square one with that particular server that is 
more than likely a stupid server anyway you should stay away from.

--
HLS

Date: Tue, 25 Oct 2011 13:28:09 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: I-D SMTP Extension for Greylisting
Sender: [email-masked]

Frank Ellermann wrote:
> On 24 October 2011 16:36, Hector wrote:
> 
>>   seconds     = 2DIGIT   ; 00-59
>>   totalsecs   = 1*(%d48-57) ; total seconds to delay
> 
>> If this suggestion for a 2nd syntax format is
>> acceptable, is the ABNF correct?
> 
> Ignoring the premise, you can use 1*DIGIT instead of
> those decimal constructs ;-)  If you want to specify
> <totalsecs> details you could suppress leading zeros
> and add an upper limit -- the very odd 99-23:59:59
> maximum yields 1*7DIGIT if I got this right, please
> check it if you want to use it.

I am not expert with ABNF language. Can you tell me what would be is 
correct here?

> And trim this to 1*6DIGIT if they let you, IMO 9 days
> is already far outside of anything remotely related
> to SMTP.

I guess we can say the odds are good, it would not be that long or 
beyond 4-5 days RFC5321 already recommends and if so, the MTA might 
consider to implement an automatic bounce idea after seeing a retry= 
hint that goes beyond its local policy maximum days.

So do you think we should?

   a) Keep with the proposed retry=[DD-]HH:MM:SS syntax?
   b) Provide a 2nd syntax retry=seconds? or
   c) Just go with a single, simpler retry=seconds syntax?

With retry=seconds, there would be no technical upper limit, only 
policy based for the MTA to determine.

--
HLS

Date: Tue, 25 Oct 2011 12:12:58 -0500
From: Pete Resnick <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: draft-kucherawy-greylisting-bcp
Sender: [email-masked]

(a) This shouldn't be labeled as a BCP. It's an A/S Standards Track 
draft if ever there was one.
(b) Given this and the extension, is it worth spinning up a Greylisting WG?

pr

On 10/23/11 3:40 PM, Murray S. Kucherawy wrote:
> Maybe a new thread Subject: would help here...
>
> Given the interest, I've started putting together a BCP.  It's now posted as draft-kucherawy-greylisting-bcp; comments welcome (and thanks to those of you that already did).  It is only an outline right now, and I'll fill it in as people make suggestions about topics it should cover and/or start providing text to include.
>
> It doesn't touch the SMTP extension people are discussing.  Someone else can champion that one if so inclined.  That shouldn't go in a BCP because it doesn't even exist yet.  This document will only describe current and historic practices.
>
> Maybe we can get it on the docket for Taipei.  I'll send out some friendly pings.
>
> -MSK
>
>
>    

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

Date: Tue, 25 Oct 2011 17:47:19 +0100
From: Paul Smith <[email-masked]>
To: Hector <[email-masked]>
CC: Patrik Faltstrom <[email-masked]>, [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 25/10/2011 17:08, Hector wrote:
> Patrik wrote:
>
>> On 25 okt 2011, at 15:29, Hector wrote:
>>
>>>> If this became an RFC, which I object to, I immediately would hack 
>>>> my smtp
>>>> server to lie, because only real smtp clients would probably retry 
>>>> enough to get the mail through.
>>> Lie in which way?
>>
>> By saying "you are gray listed in 5 minutes", but in reality the gray 
>> list is 7 minutes. 
What's the point in that?

If a spam sender was going to retry at all, currently it could retry in, 
say, 15 minutes, and have a strong chance of getting through. 
Greylisting won't stop that. The reason grey listing works is because 
many spam senders do not retry at all.

Simply telling a spam sender 'retry in 5 minutes and you'll probably get 
through' won't magically make them start retrying. They'll either retry 
anyway, and get through greylisting regardless, or not retry, and not 
get through greylisting regardless.

All a retry hint does is help senders which retry. Believe it or not, it 
is better to help a retrying spammer as well. If a retrying spammer 
currently tries every 10 seconds, then if your hint tells it not to 
bother retrying for 5 minutes, you get the same amount of spam, but less 
load on your server - you don't get more spam. (Yes, the spammer gets 
less load on their sender as well, but they probably don't care about that).

Senders which don't retry (which is the 'target' for greylisting) still 
won't retry, so their spam won't get through.

Date: Tue, 25 Oct 2011 12:09:24 -0400
From: John C Klensin <[email-masked]>
To: [email-masked], "Rolf E. Sonneveld" <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

--On Tuesday, October 25, 2011 14:00 +0200 Dave CROCKER
<[email-masked]> wrote:

>>> For example, sometimes a header field is added for the
>>> mailing list module's processing. The proposal is for
>>> yet-another Received field if the MLM decides to hold for
>>> moderation (or whatever.)
>> 
>> Another example is greylisting: should a sending SMTP client
>> mark events that
> 
> Greylisting generates an SMTP reply code, whether distinctive
> or not.  Strictly speaking, the message does not enter a new
> queue.  It merely stays in the transmission queue it was
> already in.
> 
> But certainly there would be nothing wrong with the added
> Received logging, modulo the looping count check that's
> already been mentioned.

+1

Back in the days of gateways that needed to do lots of
processing, the informal guideline (i.e., oral tradition) was
that it was desirable for an MTA to add a trace field each time
it did something with/to the message, whether that was an
authority handoff, a transformation of either message or
addressing, or some action that caused or documented a delay.
We still see much the same thing with mailing lists, where we
have one trace field for receiving the message and another for
handoff to the list exploder, even if it is on the same host. As
long as those fields are consistent with the syntax in 5321 and
a little good sense, they are certainly permitted.  For a
variety of reasons including the loop count issue to which Dave
refers, I think requiring them would be a bad idea.

Starting to add more types of trace fields has a lot of appeal
but we haven't done well with ideas like that in the past.  They
raise important design issues about ordering (there is a bit of
history of filtering arrangements treating messages with
material inserted among the Received fields as bogus) and
require additional work.

   john

   john

Date: Tue, 25 Oct 2011 12:08:08 -0400
From: Hector <[email-masked]>
To: Patrik Faltstrom <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Patrik wrote:

> On 25 okt 2011, at 15:29, Hector wrote:
> 
>>> If this became an RFC, which I object to, I immediately would hack my smtp
>>> server to lie, because only real smtp clients would probably retry enough to 
>>> get the mail through.
>> Lie in which way?
> 
> By saying "you are gray listed in 5 minutes", but in reality the gray list is 7 minutes.

But that is exactly whats going on naturally. The MTA is blind and has 
its own retries that is either too low or too high than the actual 
blocking time.

If the SMTP server is going to lie about it, well, we might as well 
give up on everything we do with the notion that its possible anyone 
can lie about anything.

In my view, we go into these things with a positive expectation 
keeping insight on real negatives that can occur.  We are talking 
about a compliant, well meaning server.

I also believe if you lied about it, you are not only hurting 
legitimate MTAs (compliant to the Extension or not), but also hurting 
yourself more with more wasted retries connections to your sites.   At 
what point are you not going to lie or just accept the MTA retries? 
When you get tired of intentionally providing false information?  "Ok, 
I had my fun, let him in!" :)

>>> There is even a risk people will require support for this which would limit 
>>> the ability for spam fighters to on server side guess whether smtp flow is 
>>> legitimate or not.
>>
>> How so?
> 
> Because, as in the example above, the one that gray list in 7 minutes while saying 5 
> will not conform to "the RFCs that describes SMTP" and because of that will be 
> rejected in RFPs that require mandatory implementation of it

My response still applies, if you lie, you are hurting yourself more 
with unnecessary retries.

>> The fact is, Greylisting can't address any SMTP sender that retries. It 
> only address the abusive random "single shot" senders that will never try the 
> same transaction again.
> 
> Why?
> 
> Depends completely what algorithm is used to finally approve and accept the incoming mail.

The classic Greylist concept is as you originally stated:

      "the whole idea with gray listing is that the assumption is that 
real
       smtp clients are more stable than spam senders."

in short, they will retry with the same transaction.  A secondary 
Greylist feature was the "Blocking Delay" which was to address those 
that do retry immediately or with very short frequencies.

>>> I.e. the draft is not explaining what problem there is to solve, the contrary. 
>> Didn't the background section go into details for the various issues and impact due to Greylisting?
> 
> No.
> 
> It does not explain well enough (obviously, otherwise I would not write this ;-) ) 
> what problem this I-D would solve.

I think the impact statements and solution proposed is very clear. 
None of the reviewers brought this up, but if you have a problem with 
it, it probably needs to be restated anyway.

> I.e. why does it help that someone give the hint "gray listed in 5 minutes" but 
> the gray listing is 7, instead of today when no hints are done at all (in 
> a parseable way)?

I personally do not think this is a reasonable objection for the 
reasons stated above. An intentional lie doesn't do anything more than 
to hurt yourself and probably more so than the MTA.

Nonetheless, it probably should be noted very strongly in the proposed 
spec that a MTA be watchful of liars  who are wasting everyone's times 
with false retry=time hints.

Thanks for your comments

--
HLS

Date: Tue, 25 Oct 2011 11:29:28 -0400
From: Hector <[email-masked]>
To: "Patrik Faltstrom (pfaltstr)" <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Patrik Faltstrom (pfaltstr) wrote:

> Further, the whole idea with gray listing is that the assumption is that real 
> smtp clients are more stable than spam senders.

+1

> If we make the life easier for spam senders, what do you think real clients 
> and servers will do? Of course explore some other corner of the protocol.

This was somewhat addressed in the Security section.  Plus there is no 
evidence of exploitation with at least seven current implementations 
already offering informal time hints. Plus, there is no negative 
evidence that moving a sender into a managed time window is actually 
bad, but desirable. Finally, extension or no extension, all SMTP 
systems MUST be able to deal with DoS issues and loading issues 
regardless any SMTP extension.  This extension does not alter this 
prudent requirement and in IMO, actually improves server availability.

> If this became an RFC, which I object to, I immediately would hack my smtp
> server to lie, because only real smtp clients would probably retry enough to 
> get the mail through.

Lie in which way?

> I.e. a paper on this only help the spam generators.

I don't see the cons, but if so, IMV, the pros outweigh the cons.

> There is even a risk people will require support for this which would limit 
> the ability for spam fighters to on server side guess whether smtp flow is 
> legitimate or not.

How so?

There is no change in operations other than to give MTA clients a 
retry time "hint" factor, the MTA can optionally use or not.  It 
doesn't change its behavior that necessitate a retry. If a spammer 
adapts to this, then I ask, what took him so long when it could of 
just adapted the easy way - keep retrying?

I don't see why this give incentives to a spammer to finally do 
something that it could of done anyway long ago.  And I'm sure many 
over the years that could adapt, did.

The fact is, Greylisting can't address any SMTP sender that retries. 
It only address the abusive random "single shot" senders that will 
never try the same transaction again.

> I.e. the draft is not explaining what problem there is to solve, the contrary. 

Didn't the background section go into details for the various issues 
and impact due to Greylisting?   Does it need to be redone, rewritten?

--
HLS

Date: Tue, 25 Oct 2011 10:57:18 -0400
From: Hector <[email-masked]>
To: [email-masked]
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

Dave CROCKER wrote:
> 
> Folks,
> 
> Occasionally, the Other Crocker ventures into email space and he chose 
> to be more thoughtful about a brief thread that was just on the IETF 
> listthat noted a message delay due to being held for moderation.
> 
> Currently, nothing in the message (or smtp responses, of course) signals 
> this reason for delay.  (For those who haven't seen the thread, the 
> person initiating the the thread thought the delay was due to DKIM 
> processing...
> 
> It would be quite simple to add a Received header field, when a message 
> is placed into a moderation queue, noting that fact.
> 
> Given the history of trying to get mailing list software to support 
> common mechanisms (standards?) this might be challenging to pursue, but 
> I thought it worth raising to the group.
> 
> Any thoughts?

I think you need to double check exactly whats going on because I 
asked specifically the list administrators in IETF-SMTP and IETF 
lists, plus specific chairs why there were delays and I was 
uncategorically told by every one of them that the mail was not being 
moderated.

In one case, it was indicated the software was customized, old and 
they had no clue what the queuing algorithm was but to indicate there 
was unexplained backlog. The mail was pushed, finally posted after 
days. In this case, I was also told that he would be traveling and any 
technical issue to be address would be 2-3 weeks later.

After that contact, new postings were immediately posted.  However, 
soon enough it was delayed once again. So this point:

    - Immediate posting
    - Eventually more delays

tells me this is more than just moderations issues.  Honestly, 
sometimes a posting from a specific address will be almost immediate, 
at least within 1-2 hours.  But then it can days.

This was 100% solved by using a different account and in my view, if 
an email domain is used that is popular with others, I believe the 
queuing software is grouping them by domain and sending out by domain 
amount factor, leaving the single domain emails at the bottom of the 
queue.

It could be a matter of how random the "moderator" gets to the 
separated queue, and just pushed a button to forward it into the list 
queue.   Well, is so, the delays in the group are too long and it 
would not be an issue to "highlight" if it was not so ridiculous to 
hold mail over 1 day if that.

In any case, for our mail system, we add a Received: line by most of 
the mail bots that move the mail from point to point:

    SMTP Receiver
    SMTP Router
    MAIL Importer
    LIST Processor

and once upon a time, for the POP3 users, a pickup Received stamp. 
This was done to explore of various MUA can allow the display of one 
or two distinct timestamps in its GUI display:

    Authorship date
    Sender Date

and for sorting reasons vs world wide time synchronization it can make 
a difference.  But if I  recall, the MUA was using the top receive 
line for something and the sorting was not "perfect."

For the idea of a Received time stamp for moderated mail, I think it 
is important to first cover how they is separated by the various 
systems.  For some, the queue is separate and don't need a special 
header to detect the moderated ones.

Personally, while it may be ok to add a normal trace for internal mail 
processing path tracking, in my strong opinion, this is not public 
material to expose if it has any sort of individual labeling or tort 
factor, especially when its done on an isolated basis and not a total 
moderated group basis.

==
HLS

Date: Tue, 25 Oct 2011 10:29:09 -0400
From: Hector <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

Murray S. Kucherawy wrote:
>> Any thoughts?
> 
> Off the top of my head, I'd prefer a new field with a new name 
> ("Quarantined: <timestamp> <host> <reason>" or suchlike), as a new trace field.  
> It would appear between other trace fields to explain the delay that took place 
> at a certain point in the handling chain.

+1 for precision traceability
-1 for difficult implementation (multiple sources of reader/writer)
-2 for privacy invasion (it would have to be removed before public 
posting)

-3 overall.

==
HLS

Subject: Re: Received header field for special processing, such as "held for moderation".
From: Keith Moore <[email-masked]>
Date: Tue, 25 Oct 2011 09:29:41 -0400
Cc: SMTP Interest Group <[email-masked]>
To: [email-masked]
Sender: [email-masked]

On Oct 25, 2011, at 2:58 AM, Dave CROCKER wrote:

> 
> Folks,
> 
> Occasionally, the Other Crocker ventures into email space and he chose to be more thoughtful about a brief thread that was just on the IETF listthat noted a message delay due to being held for moderation.
> 
> Currently, nothing in the message (or smtp responses, of course) signals this reason for delay.  (For those who haven't seen the thread, the person initiating the the thread thought the delay was due to DKIM processing...
> 
> It would be quite simple to add a Received header field, when a message is placed into a moderation queue, noting that fact.
> 
> Given the history of trying to get mailing list software to support common mechanisms (standards?) this might be challenging to pursue, but I thought it worth raising to the group.
> 
> Any thoughts?

I like this idea.

Also, it seems more appropriate to extend Received for this purpose than to create a new field, since the set of Received fields should basically describe what happened to a message as it was transmitted from sender to recipient.

Keith

Date: Tue, 25 Oct 2011 14:00:13 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Rolf E. Sonneveld" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

On 10/25/2011 1:11 PM, Rolf E. Sonneveld wrote:
> On 10/25/11 9:15 AM, Dave CROCKER wrote:
>> My formulation of the generic issue is to have the Received audit trail mark
>> processing events other than simple smtp exchanges.
>
> Does this mean you want to restrict it to only non-SMTP exchanges, or do you
> mean it should also apply to non-simple SMTP exchanges?

simple as in simply.  that is, more than merely smtp activity.

>> For example, sometimes a header field is added for the mailing list module's
>> processing. The proposal is for yet-another Received field if the MLM decides
>> to hold for moderation (or whatever.)
>
> Another example is greylisting: should a sending SMTP client mark events that

Greylisting generates an SMTP reply code, whether distinctive or not.  Strictly 
speaking, the message does not enter a new queue.  It merely stays in the 
transmission queue it was already in.

But certainly there would be nothing wrong with the added Received logging, 
modulo the looping count check that's already been mentioned.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Tue, 25 Oct 2011 13:11:05 +0200
From: "Rolf E. Sonneveld" <[email-masked]>
To: [email-masked]
Cc: Dave CROCKER <[email-masked]>, "Carl S. Gutekunst" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

On 10/25/11 9:15 AM, Dave CROCKER wrote:
>
>
>
> On 10/25/2011 9:10 AM, Carl S. Gutekunst wrote:
>> Dave CROCKER wrote:
>>> It would be quite simple to add a Received header field, when a 
>>> message is
>>> placed into a moderation queue, noting that fact.
>>
>> What problem you're trying to address? A mechanism to report any 
>> arbitrary
>> delay, or are you specifically thinking about moderator delays on a 
>> list?
>
>
> My formulation of the generic issue is to have the Received audit 
> trail mark processing events other than simple smtp exchanges.

Does this mean you want to restrict it to only non-SMTP exchanges, or do 
you mean it should also apply to non-simple SMTP exchanges?

> For example, sometimes a header field is added for the mailing list 
> module's processing.  The proposal is for yet-another Received field 
> if the MLM decides to hold for moderation (or whatever.)

Another example is greylisting: should a sending SMTP client mark events 
that are caused by greylisting? I think it could help to trace down the 
delay in the delivery of the mail, but I don't see why an SMTP client 
should mark this event in a Received header. It'd be rather marked in a 
'Tried-To-Send' header, or a 'Delayed-By' header.

>
> My thought is about the generic "internal processing" construct, if it 
> causes meaningful delay, and is not meant to be restricted to the 
> particular case of moderation.

The same is true for greylisting. An important question here is: what 
problem do we want to solve. And who is the intended 'consumer' of these 
Received lines?

An important caveat of using the Received line for this purpose: there 
are quite a number of MTA's which check for the number of Received lines 
in the header of a message; they may hold a message or sideline it when 
the number of Received lines is exceeding a threshold. Adding Received 
lines at various locations for various reasons may interfere with the 
current installed base and configurations in use. Furthermore, I would 
not be suprised if anti-spam software is also checking the number of 
Received lines, in standard or homegrown rules. For example rules in 
SpamAssassin etc.

Another issue might be, that AS software like SpamAssassin parses 
Received lines in order to perform DNSBL and similar lookups. Not sure 
whether adding Received lines may interfere with this common practice.

Using a new header might introduce overhead, but it can prevent 
confusion and it can prevent impact on the installed base of systems 
that use Received lines in some way or another.

/rolf

From: Frank Ellermann <[email-masked]>
Date: Tue, 25 Oct 2011 12:38:11 +0200
Subject: Re: I-D SMTP Extension for Greylisting
To: Hector <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 24 October 2011 16:36, Hector wrote:

> � seconds � � = 2DIGIT � ; 00-59
> � totalsecs � = 1*(%d48-57) ; total seconds to delay

> If this suggestion for a 2nd syntax format is
> acceptable, is the ABNF correct?

Ignoring the premise, you can use 1*DIGIT instead of
those decimal constructs ;-)  If you want to specify
<totalsecs> details you could suppress leading zeros
and add an upper limit -- the very odd 99-23:59:59
maximum yields 1*7DIGIT if I got this right, please
check it if you want to use it.

And trim this to 1*6DIGIT if they let you, IMO 9 days
is already far outside of anything remotely related
to SMTP.

-Frank

Date: Tue, 25 Oct 2011 03:08:15 -0700
To: [email-masked]
From: SM <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Cc: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

Hi Dave,
At 23:58 24-10-2011, Dave CROCKER wrote:
>Currently, nothing in the message (or smtp responses, of course) 
>signals this reason for delay.  (For those who haven't seen the 
>thread, the person initiating the the thread thought the delay was 
>due to DKIM processing...

A message was sent to a mailing list.  Most of the mailing lists 
hosted by that organization send a message to the sender to notify 
the person that the message is being held for moderation.  The 
mailing list doesn't do that.  There was an attempt to debug trace 
headers generally internally on the receiver's side to determine the problem.

>It would be quite simple to add a Received header field, when a 
>message is placed into a moderation queue, noting that fact.

That would help people debug trace headers generated 
internally.  That works if you have an idea of how the infrastructure 
is set up.  The mailing list software used does not inject a Received 
line.  If it did, it could have added a comment about the message 
being "held".  BTW, that does not really help the sender as the 
person still has to wait for the message to get through to the list 
to know that the ended up in the moderation queue.

Regards,
-sm 

From: Frank Ellermann <[email-masked]>
Date: Tue, 25 Oct 2011 11:50:06 +0200
Subject: Re: Received header field for special processing, such as "held for moderation".
To: [email-masked]
Cc: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On 25 October 2011 08:58, Dave CROCKER wrote:

> Any thoughts?

Good idea.  I vaguely recall that I was able to figure
it out with the complete header fields some years ago,
there used to be two Received: trace fields for IETF
servers with a big delay for moderation OR technical
issues.  And I didn't believe that DKIM could cause a
delay of more than a few seconds, certainly not hours,
but the posted evidence was incomplete.

-Frank

Date: Tue, 25 Oct 2011 10:10:52 +0100
From: Paul Smith <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 25/10/2011 09:33, Murray S. Kucherawy wrote:
> At best, this doesn't mandate standards track work. You might consider 
> "Experimental", the bar for which is far lower. If it catches on, you 
> can look at moving its status upwards. 

I would agree that this should be 'Experimental'.

I'm moderately in favour of such an extension, because in our case it 
shouldn't be that hard to implement, and may help, but I'm not convinced 
how important it is. As it isn't (or shouldn't be) that complicated, I'd 
like to see something, but am not sure whether it warrants standards 
track yet.

From: Frank Ellermann <[email-masked]>
Date: Tue, 25 Oct 2011 11:38:08 +0200
Subject: Re: retry=
To: Hector <[email-masked]>
Cc: [email-masked]
Sender: [email-masked]

On 25 October 2011 03:49, Hector wrote:

> Minutes granularity is not sufficient. �There are
> many servers that use less than 1 minute blocking
> time. e.g.; we use 55 seconds for our GL
> implementation as the default stock value out of
> the box.

Rounding up is okay, it's actually a good idea.  I'd
understand why "not now, and not before X" (minutes
or hours) can be helpful.  But seconds are IMO only
good for the well-known hh:mm:ss format; no chance
to confuse hh:mm with mm:ss.

Using only mmm (minutes, rounded up) instead of the
hh:mm:ss or DD-hh:mm:ss formats could make it clearer
that this is a time _span_ and not a time _stamp_.

Or only ssss (seconds, generously rounded up to avoid
silly states in the direction of "not now, come back
in one second".)

-Frank

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Tue, 25 Oct 2011 02:33:49 -0700
Subject: RE: Received header field for special processing, such as "held for moderation".
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave CROCKER
> Sent: Tuesday, October 25, 2011 2:20 AM
> To: Paul Smith
> Cc: SMTP Interest Group
> Subject: Re: Received header field for special processing, such as "held for moderation".
> 
> Think of a Received header field as providing a record of each message
> transition to a new queue, rather than to a new host.  While the next queue
> /might/ be on a new host, it doesn't have to be.  So mark when it goes into the
> moderation queue and mark when it finally moves back to the
> transmission queue.

Yeah, so that should work.  I re-read RFC5321 4.x about Trace Information and it looks like one could just register an additional trace field clause to contain the details (quarantined or held for moderation, and maybe the reason in a comment, etc.).

I'm glad this conversation led me to that discovery.  It means the storage of the client IP in a comment, as RFC5321 says to do, might be supplanted by something a little more sensible.

Of course, in both cases, adoption might be a bit of a long shot.

-MSK

Date: Tue, 25 Oct 2011 11:20:28 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Paul Smith <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

On 10/25/2011 10:33 AM, Paul Smith wrote:
> On 25/10/2011 09:11, Murray S. Kucherawy wrote:
>> Off the top of my head, I'd prefer a new field with a new name ("Quarantined:
>> <timestamp> <host> <reason>" or suchlike), as a new trace field.
...
> That would be nice - also a 'released' trace field (Received: might not be
> appropriate if the message was quarantined/released WITHIN the mail server
> rather than by a new one).

Think of a Received header field as providing a record of each message 
transition to a new queue, rather than to a new host.  While the next queue 
/might/ be on a new host, it doesn't have to be.  So mark when it goes into the 
moderation queue and mark when it finally moves back to the transmission queue.

The overhead of defining an implementing a new header field and making it useful 
is quite a bit higher than adding more use of an existing one.

In addition, this current need requires context surrounding it, to make sure 
that its application is sufficiently understood.  A stray header field will 
require extra help to deal with this.  A Received header field won't.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Tue, 25 Oct 2011 09:33:55 +0100
From: Paul Smith <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

On 25/10/2011 09:11, Murray S. Kucherawy wrote:
> Off the top of my head, I'd prefer a new field with a new name 
> ("Quarantined: <timestamp> <host> <reason>" or suchlike), as a new 
> trace field. It would appear between other trace fields to explain the 
> delay that took place at a certain point in the handling chain.
That would be nice - also a 'released' trace field (Received: might not 
be appropriate if the message was quarantined/released WITHIN the mail 
server rather than by a new one). We add custom quarantined/released 
headers to messages that go through our system, but it would be good to 
have a standard form.

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 25 Oct 2011 01:33:15 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Hector
> Sent: Monday, October 24, 2011 4:35 PM
> To: [email-masked]
> Subject: Re: SMTP traffic control
> 
> > Further, optimizations are notorious for having unexpected side effects,
> > and I think people have already raised some of those here.
> 
> Yet another generalization cliche.

...with quite a bit of standards and implementation expertise behind it.

> I just don't wish to see a proactive effort to improve/fix/optimize
> the current environment be thrown own because of limited perception
> that smoke isn't good enough to look at some proposals - that we need
> FIRE before anything is done, and not only fire, but everyone's house
> is burning.

I don't think there's consensus that there's even smoke here.  So far, although there have been plenty of creative suggestions about how to go about producing a workable extension to support this optimization, there's exactly one party (you) asserting that this is a problem which requires immediate protocol work to fix.

Protocol and standardization work takes a lot of people resources, usually for years at a time.  For it to be worthwhile, the problem has to be big.  There doesn’t seem to be much consensus that this is.

In one respect, you're saying this will improve delivery speed.  That may be true, but there's not much agreement with your premise that that's a key property of email which greylisting breaks.

In another, you're saying that the wasted retry attempts are a pain point.  That too appears to be a singular perspective.

At best, this doesn't mandate standards track work.  You might consider "Experimental", the bar for which is far lower.  If it catches on, you can look at moving its status upwards.

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Tue, 25 Oct 2011 01:11:54 -0700
Subject: RE: Received header field for special processing, such as "held for moderation".
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave CROCKER
> Sent: Monday, October 24, 2011 11:58 PM
> To: SMTP Interest Group
> Subject: Received header field for special processing, such as "held for moderation".
> 
> It would be quite simple to add a Received header field, when a message is
> placed into a moderation queue, noting that fact.
> 
> Given the history of trying to get mailing list software to support common
> mechanisms (standards?) this might be challenging to pursue, but I thought it
> worth raising to the group.
> 
> Any thoughts?

Off the top of my head, I'd prefer a new field with a new name ("Quarantined: <timestamp> <host> <reason>" or suchlike), as a new trace field.  It would appear between other trace fields to explain the delay that took place at a certain point in the handling chain.

When an MLM holds a message for moderation though, it's held locally; the SMTP reply is 250, so any reply text is probably discarded.  So this might be better addressed in the non-SMTP context (and ietf-822).

-MSK

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 25 Oct 2011 01:06:56 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

--_000_F5833273385BB34F99288B3648C4F06F19C6C14C6DEXCHC2corpclo_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

The point of a BCP isn't to solve problems with greylisting, but merely to =
document the practice.  It's been around long enough that doing so is proba=
bly useful to future implementers.

I'm with you on the extension thing, though.

From: [email-masked] [mailto:[email-masked]] On=
 Behalf Of Keith Moore
Sent: Monday, October 24, 2011 3:44 PM
To: Hector
Cc: [email-masked]
Subject: Re: We need an IETF BCP for GREY LISTING

On Oct 24, 2011, at 5:59 PM, Hector wrote:

There is no suggestion to change the framework.  There is an growing issue =
with increasing deployment of Greylisting.

exactly what is the issue with the increasing deployment of greylisting?  w=
hat problems does it cause that are addressed by a BCP or SMTP extension?

--_000_F5833273385BB34F99288B3648C4F06F19C6C14C6DEXCHC2corpclo_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV=3D"Content-Type" CONTENT=
=3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros=
oft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:Helvetica;
	panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.apple-style-span
	{mso-style-name:apple-style-span;}
span.apple-converted-space
	{mso-style-name:apple-converted-space;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue vli=
nk=3Dpurple style=3D'word-wrap: break-word;-webkit-nbsp-mode: space;-webkit=
-line-break: after-white-space'><div class=3DWordSection1><p class=3DMsoNor=
mal><span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";colo=
r:#1F497D'>The point of a BCP isn&#8217;t to solve problems with greylistin=
g, but merely to document the practice.&nbsp; It&#8217;s been around long e=
nough that doing so is probably useful to future implementers.<o:p></o:p></=
span></p><p class=3DMsoNormal><span style=3D'font-size:11.0pt;font-family:"=
Calibri","sans-serif";color:#1F497D'><o:p>&nbsp;</o:p></span></p><p class=
=3DMsoNormal><span style=3D'font-size:11.0pt;font-family:"Calibri","sans-se=
rif";color:#1F497D'>I&#8217;m with you on the extension thing, though.<o:p>=
</o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:11.0pt;font-=
family:"Calibri","sans-serif";color:#1F497D'><o:p>&nbsp;</o:p></span></p><d=
iv style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.=
0pt'><div><div style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:=
3.0pt 0in 0in 0in'><p class=3DMsoNormal><b><span style=3D'font-size:10.0pt;=
font-family:"Tahoma","sans-serif"'>From:</span></b><span style=3D'font-size=
:10.0pt;font-family:"Tahoma","sans-serif"'> [email-masked] [m=
ailto:[email-masked]] <b>On Behalf Of </b>Keith Moore<br><b>S=
ent:</b> Monday, October 24, 2011 3:44 PM<br><b>To:</b> Hector<br><b>Cc:</b=
> [email-masked]<br><b>Subject:</b> Re: We need an IETF BCP for GREY LIS=
TING<o:p></o:p></span></p></div></div><p class=3DMsoNormal><o:p>&nbsp;</o:p=
></p><div><div><p class=3DMsoNormal>On Oct 24, 2011, at 5:59 PM, Hector wro=
te:<o:p></o:p></p></div><p class=3DMsoNormal><br><br><o:p></o:p></p><p clas=
s=3DMsoNormal><span class=3Dapple-style-span><span style=3D'font-size:13.5p=
t;font-family:"Helvetica","sans-serif"'>There is no suggestion to change th=
e framework. &nbsp;There is an growing issue with increasing deployment of =
Greylisting.</span></span><span class=3Dapple-converted-space><span style=
=3D'font-size:13.5pt;font-family:"Helvetica","sans-serif"'>&nbsp;</span></s=
pan><o:p></o:p></p></div><p class=3DMsoNormal><o:p>&nbsp;</o:p></p><div><p =
class=3DMsoNormal>exactly what is the issue with the increasing deployment =
of greylisting? &nbsp;what problems does it cause that are addressed by a B=
CP or SMTP extension?<o:p></o:p></p></div><div><p class=3DMsoNormal><o:p>&n=
bsp;</o:p></p></div></div></div></body></html>=

--_000_F5833273385BB34F99288B3648C4F06F19C6C14C6DEXCHC2corpclo_--

Date: Tue, 25 Oct 2011 09:15:04 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Carl S. Gutekunst" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

On 10/25/2011 9:10 AM, Carl S. Gutekunst wrote:
> Dave CROCKER wrote:
>> It would be quite simple to add a Received header field, when a message is
>> placed into a moderation queue, noting that fact.
>
> What problem you're trying to address? A mechanism to report any arbitrary
> delay, or are you specifically thinking about moderator delays on a list?

My formulation of the generic issue is to have the Received audit trail mark 
processing events other than simple smtp exchanges.

For example, sometimes a header field is added for the mailing list module's 
processing.  The proposal is for yet-another Received field if the MLM decides 
to hold for moderation (or whatever.)

My thought is about the generic "internal processing" construct, if it causes 
meaningful delay, and is not meant to be restricted to the particular case of 
moderation.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Tue, 25 Oct 2011 00:10:13 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: [email-masked]
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

Dave CROCKER wrote:
> Occasionally, the Other Crocker ventures into email space and he chose 
> to be more thoughtful about a brief thread that was just on the IETF 
> listthat noted a message delay due to being held for moderation.
>
> Currently, nothing in the message (or smtp responses, of course) 
> signals this reason for delay.  (For those who haven't seen the 
> thread, the person initiating the the thread thought the delay was due 
> to DKIM processing...
>
> It would be quite simple to add a Received header field, when a 
> message is placed into a moderation queue, noting that fact. 

What problem you're trying to address? A mechanism to report any 
arbitrary delay, or are you specifically thinking about moderator delays 
on a list?

Mail filtering products have all kinds of reasons they may choose to 
deliberately hold a message, e.g., a message that appears to contain 
restricted content might be approved for sending outside the corporate 
walled garden; a virus filter may have flagged a message as "suspicious" 
and taken it "offline" for a more time-consuming examination, like 
sand-boxing; a delay caused by a referral (4xx response).

<csg>

Date: Tue, 25 Oct 2011 08:58:03 +0200
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: SMTP Interest Group <[email-masked]>
Subject: Received header field for special processing, such as "held for moderation".
Sender: [email-masked]

Folks,

Occasionally, the Other Crocker ventures into email space and he chose to be 
more thoughtful about a brief thread that was just on the IETF listthat noted a 
message delay due to being held for moderation.

Currently, nothing in the message (or smtp responses, of course) signals this 
reason for delay.  (For those who haven't seen the thread, the person initiating 
the the thread thought the delay was due to DKIM processing...

It would be quite simple to add a Received header field, when a message is 
placed into a moderation queue, noting that fact.

Given the history of trying to get mailing list software to support common 
mechanisms (standards?) this might be challenging to pursue, but I thought it 
worth raising to the group.

Any thoughts?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 24 Oct 2011 22:44:19 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: retry=
Sender: [email-masked]

Claus,

This suggestion of exposing the expire time might not be such a bad 
idea as I can see how it can help the MTA make a better decision on 
retrying which  does not need to be precise to

        current attempt time + GL server retry= blocking time.

but delayed to a time that may fit its Queuing/Grouping.

Related, one thing we did explore in our MTA was an option:

       RetryMinuteGranularity  ON/OFF

which help to group rejected emails that came in different times to 
the same minute for the next attempt.  This helped to gather 
Multi-Transaction sessions since we don't use a connection holding 
concept.

So I can see some value where the max value will give logic to the MTA 
a time limit to retry and can possible help in a MTA queuing 
implementation.

In general, in my view, it has value.

The only concern I have would be the "complexity" and the advocates 
who want to keep this as simple as possible.

Personally, I believe most MTA will behave in such a way to minimize 
delays so the odds are very high from a practical standpoint the wait 
time is all is needed and the GL servers would be expected to consider 
the MTA client would have a SMTP recommended total time (i.e. 4-5 
days) to retry before given up.

That reminds me of an off list discussion how a retry= idea can create 
new ideas for MUAs or a MSA to add a new header:

    Delivery-Expire: date-time-value

The idea was this:

The email value, for whatever reasons, is deemed to be important and 
most be delivery as fast as possible or within a certain date/time. 
this is not an email expiration concept, but a delivery time before 
forcing a bounce.

So suppose the header was added (by either the MUA/MSA)

    Delivery-Expire: 6 hours

If the remote server issues a 4yz rejection with time hint

    451 Greylisted. retry=08:00:00

and/or the remote host is down for over 6 hours, then the MTA would 
have immediate information to stop retrying and create a bounce/DSN 
reporting the delivery time can not be met within the time frame the 
email indicated.

==

Hector wrote:
> 
> Claus Assmann wrote:
>> On Mon, Oct 24, 2011, Hector wrote:
>>
>>>    http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-00.txt
>>
>> retry=delay is not entirely sufficient, it would be helpful to also
>> specify an upper bound (after which the entry will be removed from
>> the cache and another attempt will not counted as retry, but as a
>> new one).
>>
>> Hence my suggestion is to use
>>
>> retry=min-max
>>
>> e.g.,
>>
>> retry=5-3600
>>
>> where the time unit is minutes...
> 
> Interesting suggestion.
> 
> Minutes granularity is not sufficient.  There are many servers that use 
> less than 1 minute blocking time. e.g.; we use 55 seconds for our GL 
> implementation as the default stock value out of the box.  I think its 
> needs to be available that seconds is not excluded from the possibility 
> GL servers prefer to minimize their GL impact on legitimate SMTP 
> compliant mail senders using very short delays.
> 
> Regarding the min-max suggestion, I had to think about this to see how 
> it can apply with existing Grey Listing implementations.  Don't quite 
> see the reasoning stated above for the min-max, but maybe this?
> 
> While it varies, the related common GL parameters I am aware of are:
> 
>   blocking time      - time delay required for accepting
>   record expire time - life time of recorded triplet before expire
> 
> So I guess maybe, it can be
> 
>    MIN - blocking time
>    MAX - record expire time
> 
> Is that what you were thinking?
> 
> If so, for us, we use (stock defaults)
> 
> Expire.block    55     # secs to block greylisted senders
> Expire.record   2880   # mins to expire recording
> 
> So in this case, you would be suggesting it would be (keeping it in 
> seconds)?
> 
>    retry=55-172800
> 
> Hmmm, if anything, maybe a 2nd optional expire=tag
> 
>    retry=[DD-]HH:MM:SS [expire=[DD-]HH:MM:SS]
> 
> or
>    retry=[DD-]HH:MM:SS[-[DD-]HH:MM:SS]
> 
> Either way, it would do the same goal you seek (right?) but not force 
> the GL server to expose the expiration if it doesn't wish to.
> 
> Comment?
> 
> 
> 

Date: Mon, 24 Oct 2011 21:49:54 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: retry=
Sender: [email-masked]

Claus Assmann wrote:
> On Mon, Oct 24, 2011, Hector wrote:
> 
>>    http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-00.txt
> 
> retry=delay is not entirely sufficient, it would be helpful to also
> specify an upper bound (after which the entry will be removed from
> the cache and another attempt will not counted as retry, but as a
> new one).
> 
> Hence my suggestion is to use
> 
> retry=min-max
> 
> e.g.,
> 
> retry=5-3600
> 
> where the time unit is minutes...

Interesting suggestion.

Minutes granularity is not sufficient.  There are many servers that 
use less than 1 minute blocking time. e.g.; we use 55 seconds for our 
GL implementation as the default stock value out of the box.  I think 
its needs to be available that seconds is not excluded from the 
possibility GL servers prefer to minimize their GL impact on 
legitimate SMTP compliant mail senders using very short delays.

Regarding the min-max suggestion, I had to think about this to see how 
it can apply with existing Grey Listing implementations.  Don't quite 
see the reasoning stated above for the min-max, but maybe this?

While it varies, the related common GL parameters I am aware of are:

   blocking time      - time delay required for accepting
   record expire time - life time of recorded triplet before expire

So I guess maybe, it can be

    MIN - blocking time
    MAX - record expire time

Is that what you were thinking?

If so, for us, we use (stock defaults)

Expire.block    55     # secs to block greylisted senders
Expire.record   2880   # mins to expire recording

So in this case, you would be suggesting it would be (keeping it in 
seconds)?

    retry=55-172800

Hmmm, if anything, maybe a 2nd optional expire=tag

    retry=[DD-]HH:MM:SS [expire=[DD-]HH:MM:SS]

or
    retry=[DD-]HH:MM:SS[-[DD-]HH:MM:SS]

Either way, it would do the same goal you seek (right?) but not force 
the GL server to expose the expiration if it doesn't wish to.

Comment?

Date: Mon, 24 Oct 2011 21:15:57 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: retry=delay
Sender: [email-masked]

Claus Assmann wrote:
> On Mon, Oct 24, 2011, Hector wrote:
> 
>> - A consideration to simplify Servers implementation of the
>> retry=time tag where the server blocking time is already known in
>> seconds and would be an easier text response change is replace the
>> text word to "retry=seconds" compared to calculated the time in a
>> HH:MM:SS format.
> 
> - Do we need that kind of granularity?
> - What's the purpose of providing all those alternatives:
>   select one format and be done with it, everything else
>   just adds sources of potential errors without any real gain.
> 
> Use just seconds or just minutes.  IMHO minutes are good enough for
> this purpose.

I agree sticking to 1 syntax is probably best with less concerns and 
offers better endorsement value.

I am opening up the idea that for maybe better adoption by existing GL 
servers who already have time hints in their responses and using a I/O 
print format such as:

   Send("451 Greylisted, please try again in %d seconds\r\n",
        GreylistSetup.iBlockTime);

would only need to change the text:

   Send("451 Greylisted, please try again. retry=%d\r\n",
        GreylistSetup.iBlockTime);

which more than likely, the text is pulled from a separate language 
resource file. So the proposed change here is very minimal and would 
not require code image changes.

But I agree, along with others who voiced the same concern, keep it 
simple and less prune to parsing errors will help endorsement.

I'm just pointing out how in supporting other syntax can minimize the 
change needs to text response resource file that is separate from the 
code.  This is something operators can do and do need to wait for SMTP 
developers and/or Greylisting ACL, SHIMS or hooks developers to catch up.

Whatever people think is best for endorsement. I'm open. My specific 
implementation and parser will support the existing formats as well to 
increase the effectiveness with existing GL servers - this has already 
been shown to work very well.

Thanks

Date: Mon, 24 Oct 2011 17:32:01 -0700
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: SMTP traffic control
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Sun, Oct 23, 2011, Murray S. Kucherawy wrote:

> > From: Claus Assmann
> > Sent: Thursday, October 13, 2011 10:22 AM

> > It would be nice to have a standard for a server to tell a client
> > which restrictions it triggered, what the limits are, and what it
> > should do to avoid this from happening again. For example:

> Knowing which MTAs you represent, I'm curious: Are these napkin
> scratchings of things that might be neat or useful to have, or are
> you responding to consumer pain points?

I like to implement things that I consider useful, I don't need to
have some marketing research group to tell me what to do.

Currently we have some document that lists restrictions imposed by
several ISPs and I have to create default setups based on those.
Instead of having a static document with limits that apply to
everyone, it is obviously much more efficient to have (dynamic, per
client) limits that are announced when the service is actually used.

Date: Mon, 24 Oct 2011 17:23:34 -0700
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: retry= (was: I-D SMTP Extension  for Greylisting)
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Mon, Oct 24, 2011, Hector wrote:

>    http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-00.txt

retry=delay is not entirely sufficient, it would be helpful to also
specify an upper bound (after which the entry will be removed from
the cache and another attempt will not counted as retry, but as a
new one).

Hence my suggestion is to use

retry=min-max

e.g.,

retry=5-3600

where the time unit is minutes (see different reply about
the format of the delay specification).

Date: Mon, 24 Oct 2011 17:18:12 -0700
From: Claus Assmann <[email-masked]>
To: [email-masked]
Subject: Re: retry=delay (was: I-D SMTP Extension  for Greylisting)
Reply-To: [email-masked]
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Mon, Oct 24, 2011, Hector wrote:

> - A consideration to simplify Servers implementation of the
> retry=time tag where the server blocking time is already known in
> seconds and would be an easier text response change is replace the
> text word to "retry=seconds" compared to calculated the time in a
> HH:MM:SS format.

- Do we need that kind of granularity?
- What's the purpose of providing all those alternatives:
  select one format and be done with it, everything else
  just adds sources of potential errors without any real gain.

Use just seconds or just minutes.  IMHO minutes are good enough for
this purpose.

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Mon, 24 Oct 2011 19:59:48 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

On Oct 24, 2011, at 7:49 PM, Hector wrote:

> Why did (at least) seven different systems add time hints to their responses?

feeping creaturism?  Aeriously, I've lost count of the number of useless header fields I've seen email MUAs generate.   There are a lot of other silly practices in email software that are commonplace, e.g. rewriting an address that's missing a preceding phrase, to add a phrase that's redundant with the address, which just clutters up the display of the headers.

Implementors often get the idea that adding more "features" improves their product, whether or not those features do anything useful.  (Generally, it's a good idea to avoid those implementations.)

So no, the fact that a few systems have added time hints to their SMTP responses is not by itself an indicator of any kind of need to do so.

Keith

Date: Mon, 24 Oct 2011 19:49:48 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Keith Moore wrote:
> On Oct 24, 2011, at 5:59 PM, Hector wrote:
> 
>> There is no suggestion to change the framework.  There is an 
>> growing issue with increasing deployment of Greylisting. 
> 
> exactly what is the issue with the increasing deployment of 
> greylisting?  what problems does it cause that are addressed by a 
> BCP or SMTP extension?
>

Keith, this has already been stated many times in the past two weeks, 
and in my view, adequate opinion exist that BCP/EXT is desirable to 
address many different the issues and specifically, its now documented 
in the SMTP extension draft:

    http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-00.txt
    http://tools.ietf.org/html/draft-santos-smtpgrey-00

The background section is all you need to read.

Further, you have Steve Atkins's related I-D

    http://www.ietf.org/id/draft-atkins-smtp-traffic-control-00.txt

> can you actually quantify how bad the problem is?

well, first, I am not the one saying its a major problem, but 
recognizing a growing issues that IMV needs to be address at some 
point. Its not a major issue yet, but the "smoke" is there.

Yes, I can 100% measure it and I think anyone can as well that have a 
Greylisting Server employed and/or is monitoring its how volumes of 
mail is delivered.

     For this, one has to believe that delays are not good. If they
     believe a compromise is ok, then they may now see an problem.

I can also measure the benefits of supporting 4yz time hints with 
actual real results supporting MTA recognition of existing Servers 
with time hints in their 4yz responses.  This is more than an 
optimization, but an solution that lowers the impact of Greylisting 
servers had on standard SMTP systems before the Greylisting concept 
existed.

Maybe questions to ask are:

Why did (at least) seven different systems add time hints to their 
responses?

Are their existing MTA that support it, or they did write their own 
MTA to support it and/or prepared for the future?

Are they see and negative adaption to it?

Are they see any benefits?

Are any of those authors here?

Thanks

--

Subject: Re: SMTP traffic control
From: Keith Moore <[email-masked]>
Date: Mon, 24 Oct 2011 19:46:50 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

--Apple-Mail-24--257088912
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Oct 24, 2011, at 7:35 PM, Hector wrote:

> Much of this is about perception and IMV, there is a real issue out =
there, whether you feel it or not.

then name the issue, and explain why that issue justifies adding =
complexity to the mail system.=20

--Apple-Mail-24--257088912
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><div>On Oct 24, 2011, at 7:35 PM, Hector wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
font-family: Helvetica; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; ">Much of this =
is about perception and IMV, there is a real issue out there, whether =
you feel it or not.<br></span></blockquote></div><br><div>then name the =
issue, and explain why that issue justifies adding complexity to the =
mail =
system.&nbsp;</div><div><br></div><div><br></div><div><br></div></body></h=
tml>=

--Apple-Mail-24--257088912--

Date: Mon, 24 Oct 2011 19:35:03 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: SMTP traffic control
Sender: [email-masked]

Carl S. Gutekunst wrote:
> 
> Hector wrote:
>>> Personally I think this is a solution looking for a problem. 
>>
>> The cliche is nice, but there is a recognized problem and a 
>> reproducible proof of concept solution possible to help minimize the 
>> impact.
> 
> Hector, I've seen nothing in the past two of weeks of discussion that 
> identify a "problem." What you've identified is, at best, a narrow 
> potential optimization. A potential optimization is not a problem.

I personally do not wish to get into the engineering philosophical 
ideology and semantics of what constitutes an optimization and/or 
"problem" and whether an optimization is or is not a solution to a 
problem.

There are many views to this, one can be that the system has degraded 
and that in itself can be viewed or defined a problem.  The solution 
is be viewed as a "Fix", "adjustment" and/or optimization.

Me? Technically, an optimization is something that is done to a system 
that has not changed, and you are improving it.  However, when there 
is an recognized issue that has moved it to a less-quality state, an 
action taken  is not necessarily an optimization.  Nonetheless, it can 
all falls under the same concepts where to me worth to waste time on.

> Further, optimizations are notorious for having unexpected side effects, 
> and I think people have already raised some of those here.

Yet another generalization cliche.

Much of this is about perception and IMV, there is a real issue out 
there, whether you feel it or not.

I just don't wish to see a proactive effort to improve/fix/optimize 
the current environment be thrown own because of limited perception 
that smoke isn't good enough to look at some proposals - that we need 
FIRE before anything is done, and not only fire, but everyone's house 
is burning.

--

Subject: Re: SMTP traffic control
From: Keith Moore <[email-masked]>
Date: Mon, 24 Oct 2011 18:44:58 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

--Apple-Mail-23--260801097
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii

On Oct 24, 2011, at 6:10 PM, Hector wrote:

> The cliche is nice, but there is a recognized problem 

what problem is that?  can you actually quantify how bad the problem is?

--Apple-Mail-23--260801097
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><div>On Oct 24, 2011, at 6:10 PM, Hector wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
font-family: Helvetica; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; ">The cliche is =
nice, but there is a recognized problem<span =
class=3D"Apple-converted-space">&nbsp;</span></span></blockquote></div><br=
><div>what problem is that? &nbsp;can you actually quantify how bad the =
problem is?</div><div><br></div></body></html>=

--Apple-Mail-23--260801097--

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Mon, 24 Oct 2011 18:44:21 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

--Apple-Mail-22--260838294
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Oct 24, 2011, at 5:59 PM, Hector wrote:

> There is no suggestion to change the framework.  There is an growing =
issue with increasing deployment of Greylisting.=20

exactly what is the issue with the increasing deployment of greylisting? =
 what problems does it cause that are addressed by a BCP or SMTP =
extension?

--Apple-Mail-22--260838294
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><div>On Oct 24, 2011, at 5:59 PM, Hector wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
font-family: Helvetica; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; ">There is no =
suggestion to change the framework. &nbsp;There is an growing issue with =
increasing deployment of Greylisting.<span =
class=3D"Apple-converted-space">&nbsp;</span></span></blockquote></div><br=
><div>exactly what is the issue with the increasing deployment of =
greylisting? &nbsp;what problems does it cause that are addressed by a =
BCP or SMTP extension?</div><div><br></div></body></html>=

--Apple-Mail-22--260838294--

Date: Mon, 24 Oct 2011 15:38:16 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector <[email-masked]>
CC: [email-masked]
Subject: Re: SMTP traffic control
Sender: [email-masked]

Hector wrote:
>> Personally I think this is a solution looking for a problem. 
>
> The cliche is nice, but there is a recognized problem and a 
> reproducible proof of concept solution possible to help minimize the 
> impact.

Hector, I've seen nothing in the past two of weeks of discussion that 
identify a "problem." What you've identified is, at best, a narrow 
potential optimization. A potential optimization is not a problem.

Further, optimizations are notorious for having unexpected side effects, 
and I think people have already raised some of those here.

<csg>

Date: Mon, 24 Oct 2011 18:10:34 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: SMTP traffic control
Sender: [email-masked]

The cliche is nice, but there is a recognized problem and a 
reproducible proof of concept solution possible to help minimize the 
impact.  Why does it have to be everyone's problem before a solution 
is sought?  I guess some see smoke and others needs to see fire before 
acting on a solution.

MH Michael Hammer (5304) wrote:
> Personally I think this is a solution looking for a problem. 
> 
>> -----Original Message-----
>> From: [email-masked] [mailto:owner-ietf-
>> [email-masked]] On Behalf Of Murray S. Kucherawy
>> Sent: Monday, October 24, 2011 9:50 AM
>> To: [email-masked]
>> Subject: RE: SMTP traffic control
>>
>>
>>> -----Original Message-----
>>> From: Rosenwald, Jordan [mailto:[email-masked]]
>>> Sent: Monday, October 24, 2011 6:07 AM
>>> To: Murray S. Kucherawy
>>> Cc: [email-masked]
>>> Subject: RE: SMTP traffic control
>>>
>>> Murray,
>>> I don't think I'm squashing such an idea (standardization is usually
>>> beneficial). I'm more asking, IF we standardize what benefits do we
>>> expect given that many large ISPs do provide much of the feedback
>> (just
>>> not in a standardized format)?
>> [So that's what that button does.]
>>
>> Right, and I believe the goal is an optimization that reduces retries
>> that are doomed to fail.  The question you could probably answer from
>> your perspective is whether or not those retries are a problem for
> you,
>> either as a sender that provides queueing services for customers, or
> as
>> a receiver that periodically temp-fails for resource reasons.
>> Standardizing the reply increases the likelihood that clients will
>> adhere.  But if the current setup isn't a pain point for you or
> others,
>> then that suggests there's not really a problem here worth solving.
>>
> 
> 
> 
> 

Date: Mon, 24 Oct 2011 17:59:51 -0400
From: Hector <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Richard,

There is no suggestion to change the framework.  There is an growing 
issue with increasing deployment of Greylisting.  Sure, mileage will 
vary. Is it universal issue? No. Nevertheless, it is a problem waiting 
for a solution. There is an recognition regarding wasted overhead and 
increasing delays.  Fortunately, SMTP is robust enough to tamper the 
"pain" per se down because for the most part, the mail will make it - 
abeit with more attempts and longer delays.

That is something I wish to address.  That's all. Its that simple.

==

Richard Kulawiec wrote:
> On Sun, Oct 16, 2011 at 06:37:47PM -0400, Hector Santos wrote:
>> My long 30 years of mail engineering experience has given me the
>> position and view that non-timely delivery (i.e. unreasonable
>> hindrance) is the exception and not the rule. 
> 
> My equally-long years of mail engineering experience suggest that
> non-timely delivery is only an issue when there is an outage,
> a misconfiguration, an attack, or a major overload due to external events.
> Then again, I don't see mail as a real-time service, and I don't see any
> need for it to try to be.  (Other than, perhaps, on a localized basis --
> where it already pretty much is, within the limits imposed by the
> throughput of the system(s) handling it.)
> 
> I'm not arguing, in case it isn't clear, that we shouldn't try to
> make mail systems better, faster, more reliable, etc.  Of course
> we should.  But even if we could design, develop, and deploy the
> perfect greylisting solution Internet-wide tomorrow...we would
> achieve very little.  There are other much more serious problems
> that remain, that have far more serious negative effects.  So if the
> goal is overall improvement, then I think our attention is far better
> directed at the elephants rather than the gnats.
> 
> Moreover: our hypothetical improved greylisting system will be
> subject to scrutiny and attack from the same adversaries who've
> already demonstrated considerable ingenuity and remarkable adaption
> skills.  Merely engineering it so that it works nicely between
> friendly cooperating parties is insufficient; we would need to engineer
> it so that it can't be turned into a weapon. (See below for
> an example of a very poorly-thought-out idea that has long since
> been weaponized in just this fashion.)
> 
> Perhaps that's possible.  But -- so far -- I don't think so.
> It looks to me, from here, like optimizing-for-throughput is
> functionally equivalent to optimizing-for-vulnerability.  I would
> like to be convinced otherwise, but I don't see a cogent argument
> to that effect being made, yet.
> 
>> Its the same thing with CBV (Callback Verification). CBV works
>> wonderfully, but its not something I would recommend to be used
>> widely. It would not scale.  Its not something everyone can do.
> 
> CBV is inherently abusive, as has previously been discussed at length
> on forums such as the old spam-l.  Anyone using it is providing a
> spam support/enabling service and (possibly, IANAL) in violation of the
> applicable statutes which (in some jurisdictions) make it illegal to
> bypass a security mechanism.  Thus it should never be used. [1]
> 
>> Just imagine if GMAIL.COM began to do Greylisting?  One can stand
>> maybe YAHOO with its GL derivative, but with more of the bigger ESPs
>> doing the same thing, and it will get out of hand.
> 
> (a) We could imagine anything, but that doesn't mean it will happen.
> 
> (b) Greylisting has been around long enough that if there was going
> to be a mass adoption of it...it would have already happened.
> 
> (c) But if it does, and if a sufficient number of sufficiently large
> sites begin to do greylisting, then spammers will respond by making it
> ineffective.  (There is some evidence that this is already happening,
> albeit in a scattered way.)
> 
> (d) If (c) comes to pass, then it's only a matter of time until greylisting
> becomes as useless as "greet pause" techniques -- which had their day,
> and still have some marginal utility, but really don't seem terribly
> useful now that most spammers have fixed their SMTP engines.  I therefore
> question whether it's worth putting all this effort into something
> that spammers can defeat on a whim, and that some already have.
> 
> ---rsk
> 
> [1] The exception being purely-internal operations.  Although in
> that case, there is always a better way.  Even static files rsync'd
> by cron every few minutes are a vastly superior solution.
> 
> 
> 

Date: Mon, 24 Oct 2011 17:12:28 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On Sun, Oct 16, 2011 at 06:37:47PM -0400, Hector Santos wrote:
> My long 30 years of mail engineering experience has given me the
> position and view that non-timely delivery (i.e. unreasonable
> hindrance) is the exception and not the rule. 

My equally-long years of mail engineering experience suggest that
non-timely delivery is only an issue when there is an outage,
a misconfiguration, an attack, or a major overload due to external events.
Then again, I don't see mail as a real-time service, and I don't see any
need for it to try to be.  (Other than, perhaps, on a localized basis --
where it already pretty much is, within the limits imposed by the
throughput of the system(s) handling it.)

I'm not arguing, in case it isn't clear, that we shouldn't try to
make mail systems better, faster, more reliable, etc.  Of course
we should.  But even if we could design, develop, and deploy the
perfect greylisting solution Internet-wide tomorrow...we would
achieve very little.  There are other much more serious problems
that remain, that have far more serious negative effects.  So if the
goal is overall improvement, then I think our attention is far better
directed at the elephants rather than the gnats.

Moreover: our hypothetical improved greylisting system will be
subject to scrutiny and attack from the same adversaries who've
already demonstrated considerable ingenuity and remarkable adaption
skills.  Merely engineering it so that it works nicely between
friendly cooperating parties is insufficient; we would need to engineer
it so that it can't be turned into a weapon. (See below for
an example of a very poorly-thought-out idea that has long since
been weaponized in just this fashion.)

Perhaps that's possible.  But -- so far -- I don't think so.
It looks to me, from here, like optimizing-for-throughput is
functionally equivalent to optimizing-for-vulnerability.  I would
like to be convinced otherwise, but I don't see a cogent argument
to that effect being made, yet.

> Its the same thing with CBV (Callback Verification). CBV works
> wonderfully, but its not something I would recommend to be used
> widely. It would not scale.  Its not something everyone can do.

CBV is inherently abusive, as has previously been discussed at length
on forums such as the old spam-l.  Anyone using it is providing a
spam support/enabling service and (possibly, IANAL) in violation of the
applicable statutes which (in some jurisdictions) make it illegal to
bypass a security mechanism.  Thus it should never be used. [1]

> Just imagine if GMAIL.COM began to do Greylisting?  One can stand
> maybe YAHOO with its GL derivative, but with more of the bigger ESPs
> doing the same thing, and it will get out of hand.

(a) We could imagine anything, but that doesn't mean it will happen.

(b) Greylisting has been around long enough that if there was going
to be a mass adoption of it...it would have already happened.

(c) But if it does, and if a sufficient number of sufficiently large
sites begin to do greylisting, then spammers will respond by making it
ineffective.  (There is some evidence that this is already happening,
albeit in a scattered way.)

(d) If (c) comes to pass, then it's only a matter of time until greylisting
becomes as useless as "greet pause" techniques -- which had their day,
and still have some marginal utility, but really don't seem terribly
useful now that most spammers have fixed their SMTP engines.  I therefore
question whether it's worth putting all this effort into something
that spammers can defeat on a whim, and that some already have.

---rsk

[1] The exception being purely-internal operations.  Although in
that case, there is always a better way.  Even static files rsync'd
by cron every few minutes are a vastly superior solution.

Date: Mon, 24 Oct 2011 17:05:58 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On Mon, Oct 17, 2011 at 06:03:28PM -0400, Keith Moore wrote:
> That and it seems that we have a lot of people jumping on a bandwagon to
> define something new without any reason to believe that it will provide
> a significant benefit.

Yes, and:

We have a lot of people jumping on a bandwagon to define something new
without asking WHO it will benefit.  Possible inadvertent beneficiaries
include spammers and other abusers/attackers, so I think it's incumbent
on those advancing such proposals to not only conduct a full analysis
of how such technology will be attacked, but also to show that even in
the presence of such inevitable attacks that the approach is still a win
over the current state of affairs.

---rsk

Date: Mon, 24 Oct 2011 11:58:51 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

I wrote:
> Heck, take the Wikipedia page and turn it into a BCP. That's how I 
> found the link to John's paper.

Just to prevent folks from jumping all over me: that last comment was 
intended as sarcasm.

<csg>

Date: Mon, 24 Oct 2011 11:41:46 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Keith Moore wrote:
> On Oct 24, 2011, at 1:23 PM, [email-masked] wrote:
>   
>>> Personally I think this is a solution looking for a problem.
>>>       
>> +1
>> 				Ned
>>     
>
> +1  
>
> Keith
>   

+1

The one useful and actionable suggestion I've seen so far was to take 
John Levine's 2005 CEAS paper and turn it into a BCP.

Heck, take the Wikipedia page and turn it into a BCP. That's how I found 
the link to John's paper.

<csg>

Subject: Re: SMTP traffic control
From: Keith Moore <[email-masked]>
Date: Mon, 24 Oct 2011 13:37:41 -0400
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 24, 2011, at 1:23 PM, [email-masked] wrote:

> 
> 
>> Personally I think this is a solution looking for a problem.
> 
> +1
> 
> 				Ned

+1  

Keith

From: [email-masked]
Cc: "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Date: Mon, 24 Oct 2011 10:23:05 -0700 (PDT)
Subject: RE: SMTP traffic control
To: "MH Michael Hammer (5304)" <[email-masked]>
Sender: [email-masked]

> Personally I think this is a solution looking for a problem.

+1

				Ned

> > -----Original Message-----
> > From: [email-masked] [mailto:owner-ietf-
> > [email-masked]] On Behalf Of Murray S. Kucherawy
> > Sent: Monday, October 24, 2011 9:50 AM
> > To: [email-masked]
> > Subject: RE: SMTP traffic control
> >
> >
> > > -----Original Message-----
> > > From: Rosenwald, Jordan [mailto:[email-masked]]
> > > Sent: Monday, October 24, 2011 6:07 AM
> > > To: Murray S. Kucherawy
> > > Cc: [email-masked]
> > > Subject: RE: SMTP traffic control
> > >
> > > Murray,
> > > I don't think I'm squashing such an idea (standardization is usually
> > > beneficial). I'm more asking, IF we standardize what benefits do we
> > > expect given that many large ISPs do provide much of the feedback
> > (just
> > > not in a standardized format)?
> >
> > [So that's what that button does.]
> >
> > Right, and I believe the goal is an optimization that reduces retries
> > that are doomed to fail.  The question you could probably answer from
> > your perspective is whether or not those retries are a problem for
> you,
> > either as a sender that provides queueing services for customers, or
> as
> > a receiver that periodically temp-fails for resource reasons.
> > Standardizing the reply increases the likelihood that clients will
> > adhere.  But if the current setup isn't a pain point for you or
> others,
> > then that suggests there's not really a problem here worth solving.
> >

Date: Mon, 24 Oct 2011 17:02:03 +0100
From: Paul Smith <[email-masked]>
To: Hector <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 24/10/2011 16:22, Hector wrote:
>
> This is too much. The simple solution is to not to try to figure out 
> the server's reasons but to propose a simple data point it can expose 
> the idea:
>
>       "ok, you rejected me, when do you want me to try again?"
>
> I really don't care what the server reasons are and I don't wish to 
> add tons of dollars and time to research a super advanced queuing 
> concept that most likely isn't to work across the board anyway by 
> using a Guessing Game on retries.

Hector, I think that you're misunderstanding what we were talking about. 
We were discussing another possible extension to have the server publish 
information on number of concurrent connections, messages per session 
etc (I think)

The 'when to retry' idea is a simple one which I think has the potential 
to really help. The 'number of messages per session' one would be 
'nice', but probably not that important in the grand scheme of things.

Date: Mon, 24 Oct 2011 11:22:05 -0400
From: Hector <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Hi Paul,

I disagree with the "pain point" criteria being used a barrier to 
address a recognized issue.  Pain is a subjective concept and as in 
humans, even mail systems have different tolerance for "pain." I 
really hope we don't measure the need to move forward to improve the 
"ills" of the system by waiting until XYZ system gets a belly aches 
and it becomes a pain in the neck for them.

When I see my MTA wasting attempts, delaying the mail delivery - that 
is "pain" to me and my customers because it isn't working for them as 
well as they wish it will.  I am tired of having to answer customer 
questions such as

        "Why it taking so long to send this message and fast to
         send that another message?"

or

        "I thought by using DKIM, these rejections were going to go away."

Go figure.  So that guys "pains" if we to put it in those terms, is 
felt by us as well.

In ideal terms, some AI is required.  MTAs need to learn on a PER 
remote site basis how they are accepting/rejecting transactions. For 
example, it might learn that XYZ domain required 10 mins delays. It 
may learn this after it initial first time transaction of 4 attempts 
and 20 minutes, and maybe use a binary 50% cut the next time, etc.

This is too much. The simple solution is to not to try to figure out 
the server's reasons but to propose a simple data point it can expose 
the idea:

       "ok, you rejected me, when do you want me to try again?"

I really don't care what the server reasons are and I don't wish to 
add tons of dollars and time to research a super advanced queuing 
concept that most likely isn't to work across the board anyway by 
using a Guessing Game on retries.

Paul Smith wrote:
> 
> On 24/10/2011 14:49, Murray S. Kucherawy wrote:
>>> -----Original Message-----
>>> From: Rosenwald, Jordan [mailto:[email-masked]]
>>> Sent: Monday, October 24, 2011 6:07 AM
>>> To: Murray S. Kucherawy
>>> Cc: [email-masked]
>>> Subject: RE: SMTP traffic control
>>>
>>> Murray,
>>> I don't think I'm squashing such an idea (standardization is usually
>>> beneficial). I'm more asking, IF we standardize what benefits do we
>>> expect given that many large ISPs do provide much of the feedback (just
>>> not in a standardized format)?
>> [So that's what that button does.]
> :)
>> Right, and I believe the goal is an optimization that reduces retries 
>> that are doomed to fail.  The question you could probably answer from 
>> your perspective is whether or not those retries are a problem for 
>> you, either as a sender that provides queueing services for customers, 
>> or as a receiver that periodically temp-fails for resource reasons.  
>> Standardizing the reply increases the likelihood that clients will 
>> adhere.  But if the current setup isn't a pain point for you or 
>> others, then that suggests there's not really a problem here worth 
>> solving.
>>
> We get 'pain' from these limits, but mainly on customer facing 
> submission servers ('smarthosts'), rather than MX servers. It would be 
> good if we could easily tell that a '4yz' error we got was due to trying 
> to send 6 messages in a single session (when we could immediately start 
> a new session) rather than it being due to the server's disk being full 
> (in which case immediately starting a new session would be pointless)
> 
> I haven't come across it much in MX servers, but that could be because 
> most of our customers are small businesses, so sending large numbers of 
> messages to a particular MX server is unlikely, and the automatic 
> retries probably mask the few cases where it happens.
> 
> (Of course we have bigger 'pain points' - the '10 errors and we'll drop 
> you' is the worst:
> rcpt to:<[email-masked]>
> 4yz invalid recipient
> rcpt to:<[email-masked]>
> 4yz invalid recipient
> ... 8 more times
> 4yz too many syntax errors, connection dropped
> 
> (now a message in the outbound queue which will never go)
> but that's off topic)
> 
> 
> 
> 
> 

Content-class: urn:content-classes:message
Subject: RE: SMTP traffic control
Date: Mon, 24 Oct 2011 10:53:13 -0400
From: "MH Michael Hammer (5304)" <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Personally I think this is a solution looking for a problem. 

> -----Original Message-----
> From: [email-masked] [mailto:owner-ietf-
> [email-masked]] On Behalf Of Murray S. Kucherawy
> Sent: Monday, October 24, 2011 9:50 AM
> To: [email-masked]
> Subject: RE: SMTP traffic control
> 
> 
> > -----Original Message-----
> > From: Rosenwald, Jordan [mailto:[email-masked]]
> > Sent: Monday, October 24, 2011 6:07 AM
> > To: Murray S. Kucherawy
> > Cc: [email-masked]
> > Subject: RE: SMTP traffic control
> >
> > Murray,
> > I don't think I'm squashing such an idea (standardization is usually
> > beneficial). I'm more asking, IF we standardize what benefits do we
> > expect given that many large ISPs do provide much of the feedback
> (just
> > not in a standardized format)?
> 
> [So that's what that button does.]
> 
> Right, and I believe the goal is an optimization that reduces retries
> that are doomed to fail.  The question you could probably answer from
> your perspective is whether or not those retries are a problem for
you,
> either as a sender that provides queueing services for customers, or
as
> a receiver that periodically temp-fails for resource reasons.
> Standardizing the reply increases the likelihood that clients will
> adhere.  But if the current setup isn't a pain point for you or
others,
> then that suggests there's not really a problem here worth solving.
> 

Date: Mon, 24 Oct 2011 10:48:07 -0400
From: Hector <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Rosenwald, Jordan wrote:
> Murray,
> I don't think I'm squashing such an idea (standardization is usually beneficial). I'm more asking, IF we standardize what benefits do we expect given that many large ISPs do provide much of the feedback (just not in a standardized format)?
> 
> The question you asked at the end of your email is substantively different from my interpretation of the thread previously, so let me try to answer it: Most ISPs I'm familiar with are conveying the information through an over-use (perhaps "abuse") of 4xy errors, where several return codes may require reading the plain-text content to discern the variations in the responses.

Jordon, the proposed GREYLIST SMTP extension addresses in section:

    http://tools.ietf.org/html/draft-santos-smtpgrey-00#section-2.3.1

where the goal is to distinguish between loading limiting rejections 
and greylist rejections using the proposed "retry=time" hint.  This 
will help the MTA to better decide between a next hop MX/IP immediate 
attempt or a reschedule delay.   Because of the impact Greylisting, we 
had to change our MTA logic to make all 4yz responses force a 
reschedule retry.

Its been suggested the extension should consider additional tags for 
421 greeting responses such as:

     "trymx=IP/HOST"

But moves the extension in a more expanded direction and in addition, 
the MX already has the information in its DNS queries.  IMV, any 
server hint for a mx should not be outside the group already collected.

--
HLS

Date: Mon, 24 Oct 2011 10:36:04 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: I-D SMTP Extension  for Greylisting
Sender: [email-masked]

I have started on -01 draft.

- XML corrections so that the XML2RFC TXT conversion (some figures, 
list, line spacing) is laid out correctly.

- Added text to background section paragraph highlighting the fact the 
   proposed SMTP extension does not change RFC5321 due to the fact 
Greylisting and MTA specific retry strategies are already out of scope.

- Added to section 3.1 SMTP Client/Server Implementation 
SHOULD|MAY|MUST [NOT] semantics.  This will need review to get right.

- A consideration to simplify Servers implementation of the retry=time 
tag where the server blocking time is already known in seconds and 
would be an easier text response change is replace the text word to 
"retry=seconds" compared to calculated the time in a HH:MM:SS format.

If considered, I think the ABNF change would be:

    retrytime   = ("retry=" [days "-"] hours ":" minutes ":" seconds) /
                  ("retry=" totalsecs)

    days        = 2DIGIT   ; 00-99
    hours       = 2DIGIT   ; 00-23
    minutes     = 2DIGIT   ; 00-59
    seconds     = 2DIGIT   ; 00-59
    totalsecs   = 1*(%d48-57) ; total seconds to delay

If this suggestion for a 2nd syntax format is acceptable, is the ABNF 
correct?

Hector wrote:
> 
> Folks,
> 
> An SMTP Extension I-D was posted
> 
>     Title           : SMTP Service Extension for Greylisting Operations
>     Author(s)       : Hector Santos
>                           Evan Harris
>     Filename        : draft-santos-smtpgrey-00.txt
>     Pages           : 17
>     Date            : 2011-10-23
> 
>    GREYLIST is a SMTP extension to formalize the widely supported
>    Greylisting mail filtering method and to help support SMTP rejected
>    transports by following a new formal structured 4yz server temporary
>    rejection response by including a retry=time tag string which SMTP
>    clients can use to optimize the rescheduling of the mail delivery
>    attempts.  With adoption, network overhead reduction in wasteful mail
>    delivery attempts will be realized.
> 
> URLs:
> 
>    http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-00.txt
>    http://tools.ietf.org/html/draft-santos-smtpgrey-00
> 
> I would like to thank all the people who provided guidance in producing 
> this initial draft. Much obliged.   It is far from done so all input and 
> considerations from large, small to mom and pop people would be greatly 
> welcome.
> 
> Our mail servers have already implemented the SMTP extension so if you 
> need a test site, you can send mail to:
> 
>         [email-masked]
> 
> If I have written to you in the past privately, you are probably already 
> auto white listed. So use an different sender account if you are 
> interested in testing the server with its 45x greylist responses that 
> now include the proposed retry=time hint information.
> 
> In addition, if you are exploring the proposal for your own Greylist 
> server implementation, provide the address info so we can run test as well.
> 
> Thanks
> 
> -- 
> HLS
> 
> 
> 
> 
> 
> 
> 
> 

Date: Mon, 24 Oct 2011 15:24:49 +0100
From: Paul Smith <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 24/10/2011 14:49, Murray S. Kucherawy wrote:
>> -----Original Message-----
>> From: Rosenwald, Jordan [mailto:[email-masked]]
>> Sent: Monday, October 24, 2011 6:07 AM
>> To: Murray S. Kucherawy
>> Cc: [email-masked]
>> Subject: RE: SMTP traffic control
>>
>> Murray,
>> I don't think I'm squashing such an idea (standardization is usually
>> beneficial). I'm more asking, IF we standardize what benefits do we
>> expect given that many large ISPs do provide much of the feedback (just
>> not in a standardized format)?
> [So that's what that button does.]
:)
> Right, and I believe the goal is an optimization that reduces retries that are doomed to fail.  The question you could probably answer from your perspective is whether or not those retries are a problem for you, either as a sender that provides queueing services for customers, or as a receiver that periodically temp-fails for resource reasons.  Standardizing the reply increases the likelihood that clients will adhere.  But if the current setup isn't a pain point for you or others, then that suggests there's not really a problem here worth solving.
>
We get 'pain' from these limits, but mainly on customer facing 
submission servers ('smarthosts'), rather than MX servers. It would be 
good if we could easily tell that a '4yz' error we got was due to trying 
to send 6 messages in a single session (when we could immediately start 
a new session) rather than it being due to the server's disk being full 
(in which case immediately starting a new session would be pointless)

I haven't come across it much in MX servers, but that could be because 
most of our customers are small businesses, so sending large numbers of 
messages to a particular MX server is unlikely, and the automatic 
retries probably mask the few cases where it happens.

(Of course we have bigger 'pain points' - the '10 errors and we'll drop 
you' is the worst:
rcpt to:<[email-masked]>
4yz invalid recipient
rcpt to:<[email-masked]>
4yz invalid recipient
... 8 more times
4yz too many syntax errors, connection dropped

(now a message in the outbound queue which will never go)
but that's off topic)

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Mon, 24 Oct 2011 06:49:51 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Rosenwald, Jordan [mailto:[email-masked]]
> Sent: Monday, October 24, 2011 6:07 AM
> To: Murray S. Kucherawy
> Cc: [email-masked]
> Subject: RE: SMTP traffic control
> 
> Murray,
> I don't think I'm squashing such an idea (standardization is usually
> beneficial). I'm more asking, IF we standardize what benefits do we
> expect given that many large ISPs do provide much of the feedback (just
> not in a standardized format)?

[So that's what that button does.]

Right, and I believe the goal is an optimization that reduces retries that are doomed to fail.  The question you could probably answer from your perspective is whether or not those retries are a problem for you, either as a sender that provides queueing services for customers, or as a receiver that periodically temp-fails for resource reasons.  Standardizing the reply increases the likelihood that clients will adhere.  But if the current setup isn't a pain point for you or others, then that suggests there's not really a problem here worth solving.

From: "Murray S. Kucherawy" <[email-masked]>
To: "Rosenwald, Jordan" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Date: Mon, 24 Oct 2011 06:47:16 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Rosenwald, Jordan [mailto:[email-masked]]
> Sent: Monday, October 24, 2011 6:07 AM
> To: Murray S. Kucherawy
> Cc: [email-masked]
> Subject: RE: SMTP traffic control
> 
> Murray,
> I don't think I'm squashing such an idea (standardization is usually
> beneficial). I'm more asking, IF we standardize what benefits do we
> expect given that many large ISPs do provide much of the feedback (just
> not in a standardized format)?

Right, and the goal 

From: "Rosenwald, Jordan" <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: RE: SMTP traffic control
Date: Mon, 24 Oct 2011 13:07:08 +0000
Accept-Language: en-US
Sender: [email-masked]

Murray,
I don't think I'm squashing such an idea (standardization is usually beneficial). I'm more asking, IF we standardize what benefits do we expect given that many large ISPs do provide much of the feedback (just not in a standardized format)?

The question you asked at the end of your email is substantively different from my interpretation of the thread previously, so let me try to answer it: Most ISPs I'm familiar with are conveying the information through an over-use (perhaps "abuse") of 4xy errors, where several return codes may require reading the plain-text content to discern the variations in the responses.

----- Original Message -----
From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 10/24/2011 2:04 PM
Subject: RE: SMTP traffic control

> -----Original Message-----
> From: Rosenwald, Jordan [mailto:[email-masked]]
> Sent: Monday, October 24, 2011 4:16 AM
> To: Murray S. Kucherawy; [email-masked]
> Subject: RE: SMTP traffic control
>
> I know most major ISPs already provide that kind of data.  It's not
> difficult with existing response codes.  Granted they're not in a
> standard format (across ISPs). Given that any sender could mimic the
> types of response codes the ISPs are using, but haven't, what level of
> success can we really expect after a standardization effort?

Hi Jordan,

I'm confused: Senders are clients, not servers, so what response codes are they providing?

I think the proposal is to try to standardize what various servers are doing in the area of expressing retry or rate policy so that clients can comply universally rather than having to understand different syntaxes for different servers.  That seems like it can only be a benefit, as long as some critical mass of servers intend to adopt it.  And that, to me, is the real question.

How are the ISPs you know about conveying this information to clients at large?

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Mon, 24 Oct 2011 04:59:54 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Rosenwald, Jordan [mailto:[email-masked]]
> Sent: Monday, October 24, 2011 4:16 AM
> To: Murray S. Kucherawy; [email-masked]
> Subject: RE: SMTP traffic control
> 
> I know most major ISPs already provide that kind of data.  It's not
> difficult with existing response codes.  Granted they're not in a
> standard format (across ISPs). Given that any sender could mimic the
> types of response codes the ISPs are using, but haven't, what level of
> success can we really expect after a standardization effort?

Hi Jordan,

I'm confused: Senders are clients, not servers, so what response codes are they providing?

I think the proposal is to try to standardize what various servers are doing in the area of expressing retry or rate policy so that clients can comply universally rather than having to understand different syntaxes for different servers.  That seems like it can only be a benefit, as long as some critical mass of servers intend to adopt it.  And that, to me, is the real question.

How are the ISPs you know about conveying this information to clients at large?

From: "Rosenwald, Jordan" <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: SMTP traffic control
Date: Mon, 24 Oct 2011 11:16:15 +0000
Accept-Language: en-US
Sender: [email-masked]

I know most major ISPs already provide that kind of data.  It's not difficult with existing response codes.  Granted they're not in a standard format (across ISPs). Given that any sender could mimic the types of response codes the ISPs are using, but haven't, what level of success can we really expect after a standardization effort?

"Murray S. Kucherawy" <[email-masked]> wrote:

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Claus Assmann
> Sent: Thursday, October 13, 2011 10:22 AM
> To: SMTP Interest Group
> Subject: Re: SMTP traffic control
>
> It would be nice to have a standard for a server to tell a client
> which restrictions it triggered, what the limits are, and what it
> should do to avoid this from happening again. For example:
>
> - number of sessions per time unit
> - concurrent number of sessions
> - number of transactions per session
> ...
>
> It would also be helpful to have an enhanced status code that tells
> a client: "don't even try one of the other MXs, it will reject you
> too." That can be rather helpful for greylisting and for systems
> that share traffic control information.
>
> Anyone interested in specifying this?

Knowing which MTAs you represent, I'm curious: Are these napkin scratchings of things that might be neat or useful to have, or are you responding to consumer pain points?

If the latter, knowing the background of these ideas might go a long way to justifying the work and, of course, having faith that there will be a ton of interoperability shakedown, which makes for a solid specification.

-MSK

Date: Mon, 24 Oct 2011 02:17:50 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: I-D SMTP Extension  for Greylisting
Sender: [email-masked]

Folks,

An SMTP Extension I-D was posted

	Title           : SMTP Service Extension for Greylisting Operations
	Author(s)       : Hector Santos
                           Evan Harris
	Filename        : draft-santos-smtpgrey-00.txt
	Pages           : 17
	Date            : 2011-10-23

    GREYLIST is a SMTP extension to formalize the widely supported
    Greylisting mail filtering method and to help support SMTP rejected
    transports by following a new formal structured 4yz server temporary
    rejection response by including a retry=time tag string which SMTP
    clients can use to optimize the rescheduling of the mail delivery
    attempts.  With adoption, network overhead reduction in wasteful mail
    delivery attempts will be realized.

URLs:

    http://www.ietf.org/internet-drafts/draft-santos-smtpgrey-00.txt
    http://tools.ietf.org/html/draft-santos-smtpgrey-00

I would like to thank all the people who provided guidance in 
producing this initial draft. Much obliged.   It is far from done so 
all input and considerations from large, small to mom and pop people 
would be greatly welcome.

Our mail servers have already implemented the SMTP extension so if you 
need a test site, you can send mail to:

         [email-masked]

If I have written to you in the past privately, you are probably 
already auto white listed. So use an different sender account if you 
are interested in testing the server with its 45x greylist responses 
that now include the proposed retry=time hint information.

In addition, if you are exploring the proposal for your own Greylist 
server implementation, provide the address info so we can run test as 
well.

Thanks

--
HLS

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Sun, 23 Oct 2011 13:40:52 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Claus Assmann
> Sent: Thursday, October 13, 2011 10:22 AM
> To: SMTP Interest Group
> Subject: Re: SMTP traffic control
> 
> It would be nice to have a standard for a server to tell a client
> which restrictions it triggered, what the limits are, and what it
> should do to avoid this from happening again. For example:
> 
> - number of sessions per time unit
> - concurrent number of sessions
> - number of transactions per session
> ...
> 
> It would also be helpful to have an enhanced status code that tells
> a client: "don't even try one of the other MXs, it will reject you
> too." That can be rather helpful for greylisting and for systems
> that share traffic control information.
> 
> Anyone interested in specifying this?

Knowing which MTAs you represent, I'm curious: Are these napkin scratchings of things that might be neat or useful to have, or are you responding to consumer pain points?

If the latter, knowing the background of these ideas might go a long way to justifying the work and, of course, having faith that there will be a ton of interoperability shakedown, which makes for a solid specification.

-MSK

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Sun, 23 Oct 2011 13:40:40 -0700
Subject: draft-kucherawy-greylisting-bcp
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

Maybe a new thread Subject: would help here...

Given the interest, I've started putting together a BCP.  It's now posted as draft-kucherawy-greylisting-bcp; comments welcome (and thanks to those of you that already did).  It is only an outline right now, and I'll fill it in as people make suggestions about topics it should cover and/or start providing text to include.

It doesn't touch the SMTP extension people are discussing.  Someone else can champion that one if so inclined.  That shouldn't go in a BCP because it doesn't even exist yet.  This document will only describe current and historic practices.

Maybe we can get it on the docket for Taipei.  I'll send out some friendly pings.

-MSK

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Trusted agency
Organization: The Eyrie
Date: Thu, 20 Oct 2011 18:46:57 -0700
Sender: [email-masked]

Douglas Otis <[email-masked]> writes:

> When a user is able to limit those with which they exchange messages (a
> buddy list), there isn't any spam problem.  Even when bad actors return
> under a different name, they still don't make the list.

This statement is only true in a world with perfect host security.  In
practice, compromising a system and then spamming everyone in the person's
address book using that person's configured mail server, mail settings,
and authentication credentials is common.  The more emphasis is placed on
buddy-list authentication, the more prevelant this practice will become as
spam adapts to the email environment it's attempting to exploit.

Most users even use webmail systems and many of those users will happily
send their passwords to anyone who asks nicely, making this even easier.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Wed, 19 Oct 2011 15:22:58 -0700
To: Douglas Otis <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Trusted agency
Cc: [email-masked]
Sender: [email-masked]

Hi Doug,
At 13:49 19-10-2011, Douglas Otis wrote:
>Are you saying that it is not in the IETF's interest to establish 
>methods that allow receivers a means to identify those generating 
>noise from those making music?

The IETF's interest is determined by its participants.  Let's not get 
carried away by analogies.

>The current technology is heavily biased.  It gives an advantage to 
>those considered "too big to block" where current identification 
>methods do not allow the defense of one's

Yes.

>  reputation.  Such a situation keeps everyone stuck with the noise 
> or turning to a better system. :^(

It's unlikely that it can be changed by restating every now and then 
that method X is bad.    There are people who are do not care to 
delve into the technical details.  They will use the method without 
looking into the caveats.

>By most definitions, it is not spam when you wish to receive a 
>buddy's message.  I guess you are saying that you are not happy with 
>all of their messages.  There is no protocol that can determine what 
>makes you happy. :^)

No, I was saying that accounts get compromised.  I thought that RFC 
5514 would be able to determine what makes me happy. :-)

Regards,
-sm 

Date: Wed, 19 Oct 2011 17:27:41 -0400
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Rosenwald, Jordan" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/19/2011 10:11 AM, Rosenwald, Jordan wrote:
> Got it.  Conversation done.

That would be nice.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 19 Oct 2011 13:49:18 -0700
From: Douglas Otis <[email-masked]>
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: Trusted agency
Sender: [email-masked]

On 10/18/11 10:30 PM, SM wrote:
>
>  Hi Doug, At 21:59 18-10-2011, Douglas Otis wrote:
> > You are confusing consumerism with how an oligarchy works, right?
>
>  No. :-)
>
> > Currently, the Internet represents an open market where receivers
> > should be allowed to ask who is piping sewage into their home.
>
>  It's more like an open space where cacophony fills the void.

Are you saying that it is not in the IETF's interest to establish 
methods that allow receivers a means to identify those generating noise 
from those making music?

> > SPF doesn't care who should be held accountable, nor does DKIM care
> > whether a message is being abusively replayed and by whom or
> > whether it has been modified to offer deceptive
>
>  I gather that the people on this mailing list have enough clue to
>  read the specifications and assess the technology.

The current technology is heavily biased.  It gives an advantage to 
those considered "too big to block" where current identification methods 
do not allow the defense of one's reputation.  Such a situation keeps 
everyone stuck with the noise or turning to a better system. :^(

> > When a user is able to limit those with which they exchange
> > messages (a buddy list), there isn't any spam problem. Even when
> > bad actors return under a different name, they still don't make the
> > list.
>
>  I actually got some spam from people who I could consider as being on
>  the buddy list.

By most definitions, it is not spam when you wish to receive a buddy's 
message.  I guess you are saying that you are not happy with all of 
their messages.  There is no protocol that can determine what makes you 
happy. :^)

> > A web of trust is not possible without actually authenticating the
> > _accountable_ entity
>
>  Yes.

On this, I am very happy we agree.  I also assume you know what 
authentication means.

-Doug

Date: Wed, 19 Oct 2011 13:25:16 -0700
From: Douglas Otis <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Frank Ellermann <[email-masked]>, [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/19/11 6:53 AM, Keith Moore wrote:
> On Oct 19, 2011, at 8:52 AM, Frank Ellermann wrote:
>
>> That RFC 5782 isn't on standards track is actually a disgrace on the
>> IETF side of "non-research".
>
> That RFC 5782 was published at all is a disgrace.  Blacklists based on 
> IP addresses are an abomination, and DNS is a lousy mechanism for 
> distributing them.
Also this strategy will fail and will cause disruption when extended to 
IPv6 addresses and those translated to IPv4.

With respect to accountability, clients hold services accountable for 
satisfying their needs.  Without authentication of outbound MTAs, 
blocking could be due to malefactor spoofing.  Such errors are not the 
fault of the blocking service when senders offer nothing else.

The practice of abusive blocking based upon the IP address is largely 
due to there being NO practical outbound MTA authentication method.  
Accurate accountability requires scalable authentication.

IP address _authorization_ does not authenticate those transmitting the 
message.  Nor can actions be determined by a signature scheme that omits 
both those transmitting the message and their intended recipients.  
Accountability must be based upon verified actions.

While example.com may have signed content, DKIM signatures do not 
indicate whether example.com acted to send the message to specific 
recipients.  With many domains sharing the same IP address for their 
outbound MTA, authorization should never be construed as verifying that 
example.com acted to send the message.

These two schemes ONLY satisfy those domains afforded "too big to block" 
status.  Everyone else might as well outsource these services to them.  
It seems the prevailing attitude is to advance blocking services for 
IPv6 by standardizing on rating and reporting methods, while still 
neglecting a fair method to authenticate.

-Doug

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Wed, 19 Oct 2011 10:35:39 -0400
Cc: SMTP Interest Group <[email-masked]>
To: "Derek J. Balling" <[email-masked]>
Sender: [email-masked]

On Oct 19, 2011, at 10:24 AM, Derek J. Balling wrote:

>> , based on data that is often irrelevant,  without any accountability?
> 
> You're right. In fact, to be honest, we should definitely just do without DNS entirely. Because DNS data itself is often irrelevant, out of date, with no real accountability towards getting it corrected.

DNS is often out of sync with reality, but generally there is some accountability.   Presumably the owner of domain foo.com has an interest in making sure that foo.com's RRs are correct and up-to-date, and will hire someone to maintain the foo.com zone who can see to that.  

But as you point out, DNS often works poorly even for zones that have a direct interest in being correct and up-to-date.   DNS RBLs are at least two steps worse than that: (1) they make assertions about other parties without any accountability to those parties, and (2) even assuming that DNS is perfectly up-to-date, an IP address is a poor indicator of whether a message sourced from that address is likely to be spam (and getting worse all the time with the introduction of LSNs). 

Subject: Re: We need an IETF BCP  for GREY LISTING
From: "Derek J. Balling" <[email-masked]>
Date: Wed, 19 Oct 2011 10:24:51 -0400
Cc: SMTP Interest Group <[email-masked]>
To: Keith Moore <[email-masked]>
Sender: [email-masked]

On Oct 19, 2011, at 10:09 AM, Keith Moore wrote:

> mechanism for what?  for dropping legitimate mail

That makes an awful big assumption. The message could just be rejected (perfectly legitimate behavior).

> , based on data that is often irrelevant,  without any accountability?

You're right. In fact, to be honest, we should definitely just do without DNS entirely. Because DNS data itself is often irrelevant, out of date, with no real accountability towards getting it corrected.

D

From: "Rosenwald, Jordan" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Frank Ellermann <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: We need an IETF BCP  for GREY LISTING
Date: Wed, 19 Oct 2011 14:11:00 +0000
Accept-Language: en-US
Sender: [email-masked]

> -----Original Message-----
> From: Keith Moore [mailto:[email-masked]]
> Sent: Wednesday, October 19, 2011 10:10 AM
> To: Rosenwald, Jordan
> Cc: Frank Ellermann; [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> On Oct 19, 2011, at 10:05 AM, Rosenwald, Jordan wrote:
> 
> >>> That RFC 5782 isn't on standards track is actually a disgrace on the
> >>> IETF side of "non-research".
> >
> >> That RFC 5782 was published at all is a disgrace.  Blacklists based on IP
> addresses are an abomination, and DNS is a lousy mechanism for distributing
> them.
> >
> >> Keith
> >
> >
> > Respectfully, but completely disagree.  Show me a better, less expensive,
> more effective mechanism.
> 
> mechanism for what?  for dropping legitimate mail, based on data that is
> often irrelevant,  without any accountability?  I don't know that we need a
> better mechanism for that.
> 
> Keith
> 
> 
> 

Got it.  Conversation done.

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Wed, 19 Oct 2011 10:09:39 -0400
Cc: Frank Ellermann <[email-masked]>, "[email-masked]" <[email-masked]>
To: "Rosenwald, Jordan" <[email-masked]>
Sender: [email-masked]

On Oct 19, 2011, at 10:05 AM, Rosenwald, Jordan wrote:

>>> That RFC 5782 isn't on standards track is actually a disgrace on the
>>> IETF side of "non-research". 
> 
>> That RFC 5782 was published at all is a disgrace.  Blacklists based on IP addresses are an abomination, and DNS is a lousy mechanism for distributing them.
> 
>> Keith
> 
> 
> Respectfully, but completely disagree.  Show me a better, less expensive, more effective mechanism.

mechanism for what?  for dropping legitimate mail, based on data that is often irrelevant,  without any accountability?  I don't know that we need a better mechanism for that.

Keith

From: "Rosenwald, Jordan" <[email-masked]>
To: Keith Moore <[email-masked]>, Frank Ellermann <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: RE: We need an IETF BCP  for GREY LISTING
Date: Wed, 19 Oct 2011 14:05:28 +0000
Accept-Language: en-US
Sender: [email-masked]

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Keith Moore
> Sent: Wednesday, October 19, 2011 9:53 AM
> To: Frank Ellermann
> Cc: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING

> On Oct 19, 2011, at 8:52 AM, Frank Ellermann wrote:

>>  That RFC 5782 isn't on standards track is actually a disgrace on the
>>  IETF side of "non-research".�

> That RFC 5782 was published at all is a disgrace. �Blacklists based on IP addresses are an abomination, and DNS is a lousy mechanism for distributing them.

> Keith

Respectfully, but completely disagree.  Show me a better, less expensive, more effective mechanism.

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Wed, 19 Oct 2011 09:53:19 -0400
Cc: [email-masked]
To: Frank Ellermann <[email-masked]>
Sender: [email-masked]

--Apple-Mail-54--724700290
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Oct 19, 2011, at 8:52 AM, Frank Ellermann wrote:

> That RFC 5782 isn't on standards track is actually a disgrace on the
> IETF side of "non-research".=20

That RFC 5782 was published at all is a disgrace.  Blacklists based on =
IP addresses are an abomination, and DNS is a lousy mechanism for =
distributing them.

Keith

--Apple-Mail-54--724700290
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><div>On Oct 19, 2011, at 8:52 AM, Frank Ellermann wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
font-family: Helvetica; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; ">That RFC 5782 =
isn't on standards track is actually a disgrace on the<br>IETF side of =
"non-research".<span =
class=3D"Apple-converted-space">&nbsp;</span></span></blockquote></div><br=
><div>That RFC 5782 was published at all is a disgrace. &nbsp;Blacklists =
based on IP addresses are an abomination, and DNS is a lousy mechanism =
for distributing =
them.</div><div><br></div><div>Keith</div><div><br></div></body></html>=

--Apple-Mail-54--724700290--

Date: Wed, 19 Oct 2011 14:52:40 +0200
From: Frank Ellermann <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 2011-10-18 13:08, Keith Moore wrote:

>> and let's now move the BCP part of this discussion to another venue,
 >> like ASRG. GL just uses ordinary SMTP technology and with respect to
 >> GL there's nothing to be fixed in SMTP, so I fail to see why this BCP
 >> work has to be discussed on ietf-smtp.

> I fail to see how ASRG can expect to get a BCP if the people on ietf-smtp
 > don't like the proposal.

That RFC 5782 isn't on standards track is actually a disgrace on the
IETF side of "non-research".  But why are folks talking about a BCP
at all?  The simple problem reported by Hector were multiple GL 4xx
codes with various "human readable" retry delay hints.  To fix that,
if it needs to be "fixed" at all, there has to be one GL code with
one "machine readable" way to indicate when senders MAY try again.

IMO that doesn't sound like "BCP", any RFC status that's good enough
to define or modify a 4xx will do.  No SMTP extension needed; it is
no problem if a sender doesn't understand the "machine readable" 4xx
GL info, they can continue to follow their own retry strategies.

-Frank

Date: Tue, 18 Oct 2011 22:30:44 -0700
To: Douglas Otis <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Trusted agency
Cc: [email-masked]
Sender: [email-masked]

Hi Doug,
At 21:59 18-10-2011, Douglas Otis wrote:
>You are confusing consumerism with how an oligarchy works, right?

No. :-)

>Currently, the Internet represents an open market where receivers 
>should be allowed to ask who is piping sewage into their home.

It's more like an open space where cacophony fills the void.

>SPF doesn't care who should be held accountable, nor does DKIM care 
>whether a message is being abusively replayed and by whom or whether 
>it has been modified to offer deceptive

I gather that the people on this mailing list have enough clue to 
read the specifications and assess the technology.

>When a user is able to limit those with which they exchange messages 
>(a buddy list), there isn't any spam problem.  Even when bad actors 
>return under a different name, they still don't make the list.

I actually got some spam from people who I could consider as being on 
the buddy list.

>A web of trust is not possible without actually authenticating the 
>_accountable_ entity

Yes.

Regards,
-sm 

Date: Tue, 18 Oct 2011 21:59:14 -0700
From: Douglas Otis <[email-masked]>
To: SM <[email-masked]>
CC: [email-masked]
Subject: Re: Trusted agency
Sender: [email-masked]

On 10/18/11 12:20 AM, SM wrote:
>
>  At 19:56 17-10-2011, Douglas Otis wrote:
> > Agreed as well. What is needed is a light weight method to avoid
> > abusive sources with a glimmer of hope it might actually work. We
> > defend our services dynamically from
>
>  Taking people off the web removes a source of abuse. Keep in mind
>  that without these people, you won't be paid. :-)

Allowing endless abuse of SMTP seems destine to cause its destruction, 
as happy eyeballs look elsewhere.  Happy eyeballs pay our salaries not 
those want to abuse the inbox.  Adoption of 65K IPv4 Internet (prefix) 
equivalents, rapidly growing another 6 fold fairly soon, will decimate 
all reactive blocking strategies.  Sanctuary will not be found by only 
accepting IPv4 connections either.

> > Neither SPF nor DKIM properly defend domains. IP address
> > authorization and signatures omitting senders and intended
> > recipients against actually authenticating the accountable domain
> > is what is lacking. These two schemes largely support
> > white-listing domains considered "too big to block" and glaringly
> > lack a practical means to defend smaller domains or at inhibiting
> > spam. Email is in the ditch.
>
>  SPF and DKIM, like any other scheme, is not some holy grail that will
>  solve all the email problems. All schemes largely support "too big
>  to block". That is how consumerism works.

You are confusing consumerism with how an oligarchy works, right?  
Currently, the Internet represents an open market where receivers should 
be allowed to ask who is piping sewage into their home.

SPF doesn't care who should be held accountable, nor does DKIM care 
whether a message is being abusively replayed and by whom or whether it 
has been modified to offer deceptive presentations despite its promise 
in the introduction it could be safely deployed without changing 
clients.  Neither SPF nor DKIM offer the means to authenticate an entity 
that can be held accountable for abuse.  SPF will soon horribly break, 
so that will go away on its own.

> > In the face of IPv6, address authorizations schemes become
> > increasingly problematic and disruptive. Email needs to learn
> > from social networks.
>
>  Social networks is where a lot of spam come from but we cannot call
>  it that.

When a user is able to limit those with which they exchange messages (a 
buddy list), there isn't any spam problem.  Even when bad actors return 
under a different name, they still don't make the list.

Admittedly, security with all the dancing fruit within these proprietary 
services is a nightmare, where cleaning up SMTP so it can play its 
proper role seems far more appropriate, productive and safe.

> > Have each develop their own authenticated "buddy" list as an
> > overlay to what individual users might adopt. Perhaps Apples
> > example explained in RFC6281, might provide a method
>
>  See web of trust.

A web of trust is not possible without actually authenticating the 
_accountable_ entity or domain.  There are some options that could 
reduce the related overhead without inventing new protocols.

> > compete with social networks, a light weight method to authenticate
> > outbound MTAs is needed, or eventually email will be supplanted by
> > various proprietary services. Many
>
>  From draft-ietf-marid-csv-csa-02:
>
>  "Internet operation has typically required no public mechanism for
>  announcing restriction or permission of particular hosts to operate
>  clients or servers for particular services on behalf of particular
>  domains. What is missing is an open, interoperable means by which a
>  trusted agency can announce authorization for a host to operate a
>  service."
>
>  Which trusted agency should it be?

John Leslie has already answered this fairly well.  With the way the 
transition to IPv6 is likely to play out, any effort to utilize an IP 
address can only be done on a transient basis.  The underlying entity 
should still be confirmed by way of something like a kerberos ticket.  
In that view, a new type of service could emerge that offers tickets for 
various activities within the framework defined by RFC6281.

-Doug

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 18 Oct 2011 11:26:15 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Hector
> Sent: Tuesday, October 18, 2011 7:31 AM
> To: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> Murray, I'm sure you are aware, this doesn't mean that a) it isn't so,
> and b) they are not "feeling" the Greylisting in their outbound mail
> for which more than likely, its a pure guessing game for them on the
> retries.

I'm simply attempting to determine who would implement the SMTP extension you're espousing.  As far as I can tell there are some vendors willing to support it, but we haven't yet found a community of players that would be willing to activate and use such a feature, simply because it's not a pain point for any of them; certainly the wasted retries could certainly be optimized, but if they're not harmful at the moment then spending the energy to produce a standard and patch all the software out there seems a pretty high cost for little perceived gain.  And since most of them operate at enormous scales compared to you or me, and it's not a problem for them, well, ... you do the math.

> As I stated before, even if a system hasn't implemented it yet, GL is
> widely supported and they are feeling it in greater numbers.

There seems to be no data supporting this, only conjecture.  You've observed that it's happening, and there's no doubt about that.  But that doesn't mean it's creating a widespread problem that needs to be solved with protocol changes.

> Its good to get feed back from your peers if you don't have the direct
> experiences yourself in the actual product coding, field testing, fine
> tunning and seeing how it has evolved over the last 8 years, [...]

It's folly to presume that people participating in this thread don't have the experiences and duties you're listing here, and more.

Greylisting has been common practice for many years, so there's plenty of experience with using it from both client and server perspectives, enough to provide useful input to produce a BCP.  Since you're asking what it would probably say, I imagine the content of that BCP (timeouts, databases, relationship to SMTP, SMTP reply contents, etc.) would be based on consensus of this or some other group.  You're already familiar with the consensus procedure.

-MSK

Date: Tue, 18 Oct 2011 12:02:41 -0400
From: John Leslie <[email-masked]>
To: SM <[email-masked]>
Cc: [email-masked]
Subject: Re: Trusted agency (was: We need an IETF BCP  for GREY LISTING)
Sender: [email-masked]

SM <[email-masked]> wrote:
> At 19:56 17-10-2011, Douglas Otis wrote:
>>
>> What is needed is a light weight method to avoid abusive sources
>> with a glimmer of hope it might actually work...
>>
>> Neither SPF nor DKIM properly defend domains...
> 
> SPF and DKIM, like any other scheme, is not some holy grail that will 
> solve all the email problems.  All schemes largely support "too big 
> to block".  That is how consumerism works.

   2B2B is indeed a feature we can't avoid. I would disagree that "all
schemes" support it at all well.

>> to compete with social networks, a light weight method to 
>> authenticate outbound MTAs is needed, or eventually email will be 
>> supplanted by various proprietary services.
> 
> From draft-ietf-marid-csv-csa-02:

   Dear to my heart!

> "Internet operation has typically required no public mechanism for
>  announcing restriction or permission of particular hosts to operate
>  clients or servers for particular services on behalf of particular
>  domains.  What is missing is an open, interoperable means by which a
>  trusted agency can announce authorization for a host to operate a
>  service."
> 
> Which trusted agency should it be?

   "Any trusted agency" was our intent.

   "Trust" here must be based on a plausible basis for trust. There
cannot be only-one "trusted agency".

   The "csv" drafts considered that receivers would choose which
"reputation services" to trust, while senders would choose which
"vouching services" to ask to vouch for them, with no limit to the
number of each. Reputation services would each consider which of
the vouching services they consider reliable. Thus the problem of
trust reduces to a 1 x N problem, where N is limited to the number
of vouching services a particular reputation service chooses to
consider.

   The 2B2B problem remains, of course; but it is up to reputation
services to compile their own lists of 2B2B and decide whether to
assign high reliability to one or more of their vouching services
and/or just always report "good-enough reputation".

   This is what Doug and I thought could scale to Internet scale.
YMMV, of course...

--
John Leslie <[email-masked]>

Date: Tue, 18 Oct 2011 10:31:09 -0400
From: Hector <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Murray S. Kucherawy wrote:

> In light of this thread, I polled several large mailbox providers.  None of them say 
> they do greylisting.  One of them (Hotmail) replied that they don't do greylisting 
> because they just don't see enough benefit from it.  None of our large customers do it either.

Murray, I'm sure you are aware, this doesn't mean that a) it isn't so, 
and b) they are not "feeling" the Greylisting in their outbound mail 
for which more than likely, its a pure guessing game for them on the 
retries.

As I stated before, even if a system hasn't implemented it yet, GL is 
widely supported and they are feeling it in greater numbers.  For 
some, they haven't feel it yet. For many, they may never see an issue. 
  For others, like for our MTA and customers, we see a pending issue 
that needs to be addressed before it gets worst.  The more of an 
anonymous sender (Direct Marketers, Trade Newsletters, etc) when you 
may not have pre-arrangements, no whitelisting, no authentication, 
etc, the greater the issue. And as I also stated, when volume to 
different email domains are centralized MX hosted with a GL frontend, 
the delays are applies as a whole to all the different email domains. 
So a MTA Queue-By-Email-Domain strategy is now more complicated.

Its good to get feed back from your peers if you don't have the direct 
experiences yourself in the actual product coding, field testing, fine 
tunning and seeing how it has evolved over the last 8 years, but at 
some point you are going to have some to accept there are issues being 
felt and pending issues that waiting to happen as GL frontends grows.

It may all comes down to two things:

    1) For the non-intelligent GL MTA, it needs to use a "Best Guess" 
retry
       that is a compromise in lowering attempts and timely delivery,

    2) For a new intelligent GL MTA/MDA supporting retry=hints, it 
CAN|SHOULD
       benefit the client, server and the network with reduced 
overhead and
       transfer.

You need to understand that initial delays by an MTA such as 1 minute 
is done for a reason - they can't afford the negative business 
operation with untimely deliveries, especially when a customer is 
trying to contact you.  This is a REAL issue.

Yet, a customer doesn't wish to get delayed in sending its own mail. 
So if blocked, our GL setup has the short delay to help alleviate that 
concern.

The issue is that with the GL growth over the past 8 years, they are 
hitting more servers where the rejects are more common place and 
delays are more wide spread and longer in blocking times.

So sure, we can come up with a BCP with a recommended delay, but what 
is that? 1, 5, 10, 15, 30, 1 hour?   While one can reasonable suggest 
that a statistic average suggest 5 minutes is the most appropriate, I 
technically believe that is not good enough for active MTAs who see 
CLIENT/SERVER intelligence is possible and they don't have to wait 
that long for many GL servers who issues responses like:

    451 Greylisting enabled, try again in 1 minutes
    451 Greylisting enabled, try again in 30 seconds

and if a response that

    451 4.7.1 Greylisting in action, please come back in 00:10:00

then a possible BCP recommendation of 5 minutes retry would be a waste.

--
HLS

Date: Tue, 18 Oct 2011 09:47:28 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Tim Kehres wrote:

> Progressive backoff strategies 
> were put in place precisely for this reason - but some care needs to be 
> taken to choose reasonable timing values.
> 
> Greylisting sites that configure timeout windows that greatly exceed the 
> norm for remote MTA retry times (hours or more for instance) are also 
> improperly configured and will not play well with MTA's with reasonable 
> backoff times.
> 
> A BCP that describes various queuing strategies in terms of retry times, 
> and the relationship between MTA timing and Greylist delay windows might 
> be a good thing for a BCP.   It's really not all that difficult, however 
> sometimes putting these things into writing can help new administrators. 
> The BCP can then formalize what the community considers "reasonable" 
> values for MTA's and GL running servers.
> 
> Anything else I'm really at a loss to understand what we are trying to 
> fix. Greylisting for the most part works and other than the queue timing 
> issues above seems largely to be a non-issue.
> 
> Best Regards,

This group seems to make things more complicated than necessary. 
You're right. It isn't that difficult. I called for a BCP that would 
provide the reasonable guidelines we would expect to be reasonable, 
with a goal to alleviate the growing overhead.  You stated your points 
as client and server poor choices.  A BCP should help, especially as 
you noted with new GL implementators.

Lets note, there is overhead redundancy when it has becomes a 
"guessing" game and the Progressive Backoff strategies don't alway 
apply. The higher volume systems will see this. A large mail system 
doesn't need to support GL but they also feel it in their outbound mail.

It might just become what is a "reasonable initial delay?"

     1 minute?
     5 minutes?
    10 minutes?
    20 minutes?

etc.

There are many GL servers that provide a "retry time hint" in some 
form, hence we should consider if this can help clients lower their 
wasteful attempts and also help server's availability.

Suppose the MTA default attempts frequency is:

    1st immediate
    2nd 1 minutes
    3rd 4 mins
    4th 15 mins
    ...

and the server issues a response on the first attempt:

    451 4.7.1 Greylisting in action, please come back in 00:10:00

then the MTA is not going to make it until the 4th attempt, 20 minutes 
later.

Sure, not unreasonable one may suggest, and one may suggest that the 1 
min 2nd attempt is too short, but it is overhead and in volume, it is 
much greater.

So yesterday, I put into place logic into our MTA to extract the retry 
time hints from the various forms of GL responses with time hints and 
so far, it seems to work.  I get the same remote that provided the 
above response, the MTA extracted the 10 minutes and rescheduled the 
2nd attempt 10 minutes later.  I waited the 10 mins (plus a few 
seconds to start the engine) and poof! The 2nd attempt was accepted.

I'm using this C code (with test and expected output) to parse the 4yz 
response:

#include <stdio.h>
#include <windows.h>

//-----------------------------------------------------------
// int GetRetrySecs(const char *psz)
//
// return # of seconds from formats
//
//  HH:MM:SS
//  # seconds
//  # minutes
//  # hours
//
//-----------------------------------------------------------

int GetRetrySecs(const char *psz)
{
     char *pOut = NULL;
     int secs = strtol(psz,&pOut,10);
     if (*pOut == ':') {
        secs = secs*3600+strtol(pOut+1,&pOut,10)*60;
        if (*pOut == ':') secs += atoi(pOut+1);
        return secs;
     }
     if (secs) {
        if (*pOut == ' ') pOut++;
        if (*pOut == 'm' || *pOut == 'M') secs=secs*60;
        if (*pOut == 'h' || *pOut == 'H') secs=secs*3600;
     }
     return secs;
}

//-----------------------------------------------------------
// int ExtractRetryHint(const char *psz)
//
// For given SMTP 4yz response, return any known retry time
// hint in seconds. Return 0 for no time hint or non 4yz
// response.
//-----------------------------------------------------------

char *stristr(const char *string1, const char *string2)
{
     if (!string1 || !string2) {
         return NULL;
     }
     char *copy1 = _strupr(_strdup(string1));
     char *copy2 = _strupr(_strdup(string2));
     char *result = strstr(copy1, copy2);
     if (result) result = (char *)string1 + (result - copy1);
     free(copy1);
     free(copy2);
     return result;
}

int ExtractRetryHint(const char *psz)
{
     if (!psz || !psz[0] || ((atoi(psz)/100*100) != 400)) {
         return 0;
     }
     const char *tags[] = {
         "retry=",
         "Greylisted for ",
         "try again in ",
         "please come back in ",
         NULL};
     const char **p = tags;
     while (*p) {
       char *pos = stristr(psz,*p);
       if (pos) return GetRetrySecs(pos+strlen(*p));
       p++;
     }
     return 0;
}

void test(const char *sz)
{
     int secs = ExtractRetryHint(sz);
     printf("secs: %5d | %s\n",secs, sz);
}

void main(char argc, char *argv[])
{
     test("421 This server implements greylisting, please try again in 
90 seconds");
     test("450 4.7.1 <RCPT>: Recipient address rejected: Greylisted 
for 5 minutes");
     test("450 4.7.1 <RCPT>: Recipient address rejected: Greylisted 
for 120 seconds");
     test("451 4.7.1 Greylisting in action, please come back in 
00:30:00");
     test("451 4.7.1 Greylisting in action, PLEASE COME BACK in 
01:05:20");
     test("451 Greylisted for 55 seconds");
     test("451 Greylisted, please try again in 167 seconds");
     test("451 Greylisting enabled, try again in 22 minutes");
     test("451 Greylisting. Try again later.");
     test("451 Greylisting. Try again later. retry=5 mins");
     test("471 Greylisting. Try again later. retry=300 secs");
     test("471 Greylisting. Try again later. retry=2 hours");
     test("471 Greylisting. Try again later. retry=2h");
     test("471 Greylisting. Try again later. retry=120");
     test("471 Greylisting. Try again later. retry=01:01:02");
     test("471 retry=01:01:02 Greylisting. Try again later.");
     test("471 Greylist: RETRY=1M Try again later.");
     test("550 Permament Reject");
}

Expected output:

secs:    90 | 421 This server implements greylisting, please try again 
in 90 seconds
secs:   300 | 450 4.7.1 <RCPT>: Recipient address rejected: Greylisted 
for 5 minutes
secs:   120 | 450 4.7.1 <RCPT>: Recipient address rejected: Greylisted 
for 120 seconds
secs:  1800 | 451 4.7.1 Greylisting in action, please come back in 
00:30:00
secs:  3920 | 451 4.7.1 Greylisting in action, PLEASE COME BACK in 
01:05:20
secs:    55 | 451 Greylisted for 55 seconds
secs:   167 | 451 Greylisted, please try again in 167 seconds
secs:  1320 | 451 Greylisting enabled, try again in 22 minutes
secs:     0 | 451 Greylisting. Try again later.
secs:   300 | 451 Greylisting. Try again later. retry=5 mins
secs:   300 | 471 Greylisting. Try again later. retry=300 secs
secs:  7200 | 471 Greylisting. Try again later. retry=2 hours
secs:  7200 | 471 Greylisting. Try again later. retry=2h
secs:   120 | 471 Greylisting. Try again later. retry=120
secs:  3662 | 471 Greylisting. Try again later. retry=01:01:02
secs:  3662 | 471 retry=01:01:02 Greylisting. Try again later.
secs:    60 | 471 Greylist: RETRY=1M Try again later.
secs:     0 | 550 Permament Reject

I plan to do at least 1 week of internal testing before I select some 
of the larger volume field testers that are getting hit harder with GL 
servers.

There are many responses with hints that says:

  S: 421 This server implements greylisting, please try again in 300 
seconds
  S: 451 Greylisted, please try again in 600 seconds
  S: 451 Greylisted, please try again in 511 seconds
  S: 451 Greylisted, please try again in 304 seconds
  S: 451 Greylisted, please try again in 900 seconds
  S: 451 Greylisted, please try again in 700 seconds
  S: 451 Greylisted, please try again in 600 seconds
  S: 451 Greylisted, please try again in 900 seconds
  S: 451 Greylisted, please try again in 491 seconds
  S: 451 Greylisted, please try again in 511 seconds
  S: 451 Greylisted, please try again in 873 seconds
  S: 451 Greylisted, please try again in 298 seconds
  S: 451 Greylisted, please try again in 628 seconds
  S: 451 Greylisting enabled, try again in 1 minutes
  S: 451 Greylisting enabled, try again in 5 minutes
  S: 450 4.7.1 <[email-masked]>: Recipient address rejected: 
Greylisted for 5 minutes
  S: 451 Greylisting enabled, try again in 5 minutes
  S: 451 Greylisting enabled, try again in 2 minutes

Many have hints between 1 to 5 minutes so its one can say that a guess 
of 1 min and 4 mins is not  "reasonable," and these will be best for 
"faster" timely delivery. A smaller system can possibly afford some 
wasted attempts.  I think it becomes more overhead with larger volume.

Overall, I prefer not to guess and waste attempts if the GL server can 
assist with a more "appropriate" blocking time and if I can add the GL 
MTA feature for our customers to enable, then I believe they will benefit.

Off hand, 5 minutes seem to be common, but as you see see, there are 
those with 15 minutes initial delays.

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Tue, 18 Oct 2011 07:08:40 -0400
Cc: [email-masked], Tim Kehres <[email-masked]>, Hector <[email-masked]>
To: "Rolf E. Sonneveld" <[email-masked]>
Sender: [email-masked]

On Oct 18, 2011, at 6:01 AM, Rolf E. Sonneveld wrote:

> and let's now move the BCP part of this discussion to another venue, like ASRG. GL just uses ordinary SMTP technology and with respect to GL there's nothing to be fixed in SMTP, so I fail to see why this BCP work has to be discussed on ietf-smtp.

I fail to see how ASRG can expect to get a BCP if the people on ietf-smtp don't like the proposal.

Date: Tue, 18 Oct 2011 12:01:34 +0200
From: "Rolf E. Sonneveld" <[email-masked]>
To: [email-masked]
Cc: Tim Kehres <[email-masked]>, Keith Moore <[email-masked]>, Hector <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/18/11 11:12 AM, Tim Kehres wrote:
>
>>> Keith Moore wrote:
>>>> On Oct 17, 2011, at 12:25 PM, Hector wrote:
>>>>> It seems there is a majority consensus that a IETF document for 
>>>>> Greylisting would be a worthy effort.
>>>> I'm still unconvinced.  I haven't seen any evidence to support this 
>>>> at all.
>>>
>>> hmmm,  unless I mis-read people's input, many people contributing to 
>>> the thread recognized there are issues and most have indicated a 
>>> note that a IETF document for greylisting would be a good thing.
>>>
>>> I believe you cited an interest in an SMTP extension for a retry hint.
>>
>> I mentioned how it could be done, but also expressed concern that it 
>> might not be worth the trouble... also I said that greylisting wasn't 
>> the reason for such an extension.
>
> I am in complete agreement with Keith's observations.   To further 
> elaborate the issue of formalizing 4xx return codes is NOT an issue 
> with Greylisting, and this is not the context to address such 
> extensions.   If SMTP were to include such extensions, it should be a 
> general feature, and not something perceived to be tied with GL.
>
> Now, with respect to Greylisting - precisely what existing problem are 
> we trying to solve?  4xx return codes are not a greylisting issue.  
> The few problems that were mentioned to me seem to be configuration 
> related, and really not subject to an RFC.  Sites that either run too 
> short MTA retry times (30 seconds for a retry for example) will always 
> run into problems, regardless if they are trying to connect to a 
> remote server that has them on a temporary greylist, or any other site 
> that is inaccessible for some period of time.  Progressive backoff 
> strategies were put in place precisely for this reason - but some care 
> needs to be taken to choose reasonable timing values.
>
> Greylisting sites that configure timeout windows that greatly exceed 
> the norm for remote MTA retry times (hours or more for instance) are 
> also improperly configured and will not play well with MTA's with 
> reasonable backoff times.
>
> A BCP that describes various queuing strategies in terms of retry 
> times, and the relationship between MTA timing and Greylist delay 
> windows might be a good thing for a BCP.   It's really not all that 
> difficult, however sometimes putting these things into writing can 
> help new administrators. The BCP can then formalize what the community 
> considers "reasonable" values for MTA's and GL running servers.
>
> Anything else I'm really at a loss to understand what we are trying to 
> fix. Greylisting for the most part works and other than the queue 
> timing issues above seems largely to be a non-issue.

and let's now move the BCP part of this discussion to another venue, 
like ASRG. GL just uses ordinary SMTP technology and with respect to GL 
there's nothing to be fixed in SMTP, so I fail to see why this BCP work 
has to be discussed on ietf-smtp.

/rolf

From: "Tim Kehres" <[email-masked]>
To: "Keith Moore" <[email-masked]>, "Hector" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Date: Tue, 18 Oct 2011 17:12:32 +0800
Sender: [email-masked]

>> Keith Moore wrote:
>>> On Oct 17, 2011, at 12:25 PM, Hector wrote:
>>>> It seems there is a majority consensus that a IETF document for 
>>>> Greylisting would be a worthy effort.
>>> I'm still unconvinced.  I haven't seen any evidence to support this at 
>>> all.
>>
>> hmmm,  unless I mis-read people's input, many people contributing to the 
>> thread recognized there are issues and most have indicated a note that a 
>> IETF document for greylisting would be a good thing.
>>
>> I believe you cited an interest in an SMTP extension for a retry hint.
>
> I mentioned how it could be done, but also expressed concern that it might 
> not be worth the trouble... also I said that greylisting wasn't the reason 
> for such an extension.

I am in complete agreement with Keith's observations.   To further elaborate 
the issue of formalizing 4xx return codes is NOT an issue with Greylisting, 
and this is not the context to address such extensions.   If SMTP were to 
include such extensions, it should be a general feature, and not something 
perceived to be tied with GL.

Now, with respect to Greylisting - precisely what existing problem are we 
trying to solve?  4xx return codes are not a greylisting issue.  The few 
problems that were mentioned to me seem to be configuration related, and 
really not subject to an RFC.  Sites that either run too short MTA retry 
times (30 seconds for a retry for example) will always run into problems, 
regardless if they are trying to connect to a remote server that has them on 
a temporary greylist, or any other site that is inaccessible for some period 
of time.  Progressive backoff strategies were put in place precisely for 
this reason - but some care needs to be taken to choose reasonable timing 
values.

Greylisting sites that configure timeout windows that greatly exceed the 
norm for remote MTA retry times (hours or more for instance) are also 
improperly configured and will not play well with MTA's with reasonable 
backoff times.

A BCP that describes various queuing strategies in terms of retry times, and 
the relationship between MTA timing and Greylist delay windows might be a 
good thing for a BCP.   It's really not all that difficult, however 
sometimes putting these things into writing can help new administrators. 
The BCP can then formalize what the community considers "reasonable" values 
for MTA's and GL running servers.

Anything else I'm really at a loss to understand what we are trying to fix. 
Greylisting for the most part works and other than the queue timing issues 
above seems largely to be a non-issue.

Best Regards,

-- Tim 

Date: Tue, 18 Oct 2011 00:20:17 -0700
To: [email-masked]
From: SM <[email-masked]>
Subject: Trusted agency (was: We need an IETF BCP  for GREY LISTING)
Sender: [email-masked]

At 19:56 17-10-2011, Douglas Otis wrote:
>Agreed as well.  What is needed is a light weight method to avoid 
>abusive sources with a glimmer of hope it might actually work.  We 
>defend our services dynamically from

Taking people off the web removes a source of abuse.  Keep in mind 
that without these people, you won't be paid. :-)

>Neither SPF nor DKIM properly defend domains.  IP address 
>authorization and signatures omitting senders and intended 
>recipients against actually authenticating the accountable domain is 
>what is lacking.  These two schemes largely support white-listing 
>domains considered "too big to block" and glaringly lack a practical 
>means to defend smaller domains or at inhibiting spam.  Email is in the ditch.

SPF and DKIM, like any other scheme, is not some holy grail that will 
solve all the email problems.  All schemes largely support "too big 
to block".  That is how consumerism works.

>In the face of IPv6, address authorizations schemes become 
>increasingly problematic and disruptive.  Email needs to learn from 
>social networks.

Social networks is where a lot of spam come from but we cannot call it that.

>Have each develop their own authenticated "buddy" list as an overlay 
>to what individual users might adopt.  Perhaps Apples example 
>explained in RFC6281, might provide a method

See web of trust.

>  compete with social networks, a light weight method to 
> authenticate outbound MTAs is needed, or eventually email will be 
> supplanted by various proprietary services.  Many

 From draft-ietf-marid-csv-csa-02:

   "Internet operation has typically required no public mechanism for
    announcing restriction or permission of particular hosts to operate
    clients or servers for particular services on behalf of particular
    domains.  What is missing is an open, interoperable means by which a
    trusted agency can announce authorization for a host to operate a
    service."

Which trusted agency should it be?

Regards,
-sm 

Date: Mon, 17 Oct 2011 19:56:44 -0700
From: Douglas Otis <[email-masked]>
To: John C Klensin <[email-masked]>
CC: "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/17/11 4:51 AM, John C Klensin wrote:
> --On Sunday, October 16, 2011 22:57 -0700 "Murray S. Kucherawy"
> <[email-masked]>  wrote:
>
>>> Maybe a shorter way to put that last paragraph is that if we
>>> agree that reasonably timely delivery of mail is a Good
>>> Thing, then I think we have many other problems to tackle
>>> before we turn our attention to greylisting.
>> Agree here too.
> +1
>
> And even more strongly agree if "delivery of all legitimate mail
> without forcing senders to use a small number of senders with
> bilateral arrangements" is added to the list.
Agreed as well.  What is needed is a light weight method to avoid 
abusive sources with a glimmer of hope it might actually work.  We 
defend our services dynamically from questionable sources as a method to 
prioritize the rest.  Not grey listing in the classic sense, but no less 
of a PITA and an extremely complex system still constantly being gamed.

Neither SPF nor DKIM properly defend domains.  IP address authorization 
and signatures omitting senders and intended recipients against actually 
authenticating the accountable domain is what is lacking.  These two 
schemes largely support white-listing domains considered "too big to 
block" and glaringly lack a practical means to defend smaller domains or 
at inhibiting spam.  Email is in the ditch.

In the face of IPv6, address authorizations schemes become increasingly 
problematic and disruptive.  Email needs to learn from social networks.  
Have each develop their own authenticated "buddy" list as an overlay to 
what individual users might adopt.  Perhaps Apples example explained in 
RFC6281, might provide a method to include separate services that 
perform authentication, and then simply hand out SMTP tickets that are 
good for some period of time.

It is interesting that services like twitter's ad revenue "unique" 
criteria gives them an incentive to cancel abusive accounts.  Two 
seconds later, they may reappear under a different name, but that is 
okay as now they must then beg for attention.  For email to compete with 
social networks, a light weight method to authenticate outbound MTAs is 
needed, or eventually email will be supplanted by various proprietary 
services.  Many domains are not aware of incoming connection failure 
rates due to loads not seen in SMTP logs.  Incoming messages might not 
be blocked, but may never get through the growing volume of noise.

To: [email-masked]
From: [email-masked]

Please add this new service to your "buddy" list.

I understand failure to respond to your complaints in a timely manner 
may cause my domain to be removed from your "buddy" list based on 
protocol X authentication methods.

-Doug

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Mon, 17 Oct 2011 18:03:28 -0400
Cc: Hector <[email-masked]>, [email-masked]
To: Paul Smith <[email-masked]>
Sender: [email-masked]

On Oct 17, 2011, at 5:47 PM, Paul Smith wrote:

> On 17/10/2011 18:46, Keith Moore wrote:
>> 
>>> BTW, Keith, many GL servers already have responses with hints.
>> yes, I know.   but just because someone decided to add a %d to a printf doesn't mean it should be standardized.
> 
> I must be missing something. I'm struggling to see why so many people think that changing a server that gives a hint from saying
> 
>  421 This server implements greylisting, please try again in # seconds
> 
> to saying
> 
>  421 [retry=#s] This server implements greylisting, please try again in # seconds
> 
> could possibly be a bad thing...

Anything that increases complexity of mail clients should be considered suspect.   (i.e. it's not the burden on the server that I'm concerned about...)

That and it seems that we have a lot of people jumping on a bandwagon to define something new without any reason to believe that it will provide a significant benefit.

Every new RFC that needs to be written can, in some sense, be considered a failure.... unless that RFC replaces some other document that was more ambiguous or took up more pages.

Keith

Date: Mon, 17 Oct 2011 22:47:03 +0100
From: Paul Smith <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Hector <[email-masked]>, [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 17/10/2011 18:46, Keith Moore wrote:
>
>> BTW, Keith, many GL servers already have responses with hints.
> yes, I know.   but just because someone decided to add a %d to a printf doesn't mean it should be standardized.

I must be missing something. I'm struggling to see why so many people 
think that changing a server that gives a hint from saying

   421 This server implements greylisting, please try again in # seconds

to saying

   421 [retry=#s] This server implements greylisting, please try again 
in # seconds

could possibly be a bad thing...

If the [retry=#s] gets ignored, then, so what. If it's there then it is 
much easier to parse for someone who decides they want to try to 
cooperate with the other end

IMV, an experimental extension to 5321 saying that a 4xx response can 
optionally have a "[retry=#s]" hint should be non-controversial and 
straight forward. It may not achieve much, but it may help a bit, and it 
would not have a down side as far as I can see.

Maybe there are other things to worry about in SMTP that could have more 
benefit, but things like DKIM are a lot more complex to specify and 
implement and are more likely to cause interoperability issues. What's 
wrong with doing very simple things as well as the complex things?

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Mon, 17 Oct 2011 12:20:19 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

Given the interest, I've started putting together a BCP.  It's now posted as draft-kucherawy-greylisting-bcp; comments welcome.  It is only an outline right now, and I'll fill it in as people make suggestions about topics it should cover and/or start providing text to include.

It doesn't touch the SMTP extension people are discussing.  Someone else can champion that one if so inclined.  It shouldn't go in a BCP because it doesn't even exist yet.  This document will only describe current and historic practices.

Maybe we can get it on the docket for Taipei.  I'll send out some friendly pings.

-MSK

Date: Mon, 17 Oct 2011 14:46:25 -0400
From: Hector <[email-masked]>
To: [email-masked]
CC: Evan Harris <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

For the record, Evan Harris, the author of Greylisting, has responded 
to my email and have indicated interest to participate in an IETF 
effort to address Greylisting.   He is catching up with the IETF-SMTP 
archived thread discussions.

This is a great start to get Evan involved.  He can possibly help 
explain the original reasonings for some of the initial 
recommendations and any new adjustments and recommendations he would 
make today.

Evan, what are your thoughts regarding a formal "retry=hint" (or 
similar) concept in the 4yz responses?

--
HLS

Hector wrote:
> 
> Hector wrote:
> 
>> ...
>> I think someone should take up the effort to begin/draft an GreyList 
>> BCP for systems to follow.
>>
>> Comments?
> 
> 
> Hi,
> 
> It seems there is a majority consensus that a IETF document for 
> Greylisting would be a worthy effort.
> 
> I do not have the time, so I am hoping some other interested party can 
> write it. But if not, I may consider writing this document. I can 
> certainly help co-author it. If someone wishes to assist, please write 
> me off list.
> 
> I sent a note to Evan Harris, AFAIK, the original author of Greylisting 
> asking for his interest in authoring or contributing to the development 
> official IETF GreyListing document.   I seem to recall long ago, a 
> comment he made that he had avoided the IETF knowing there would be 
> major resistance with some an "odd idea" of using an initial rejection 
> as a form to filter a certain form of "single shot" random spammers.  I 
> agreed 100% the IETF resistance would of been there, but that was back 
> then in 2003. Not today. Attitudes have changed about GL and now you see 
> GL implementation widely adopted, even among the early IETF critics.
> 
> Maybe to start such a document, we can gather the following regarding 
> the basic GL framework I think many use or part of the following:
> 
>   - Recorded Information
>   - Class C support
>   - Blocking Time
>   - White Listed Time
>   - Rejection Point
>   - Rejection response
> 
> For example, for our GL implementation:
> 
>   - Recorded Information
> 
>       Connecting IP, MAIL FROM, RCPT TO
> 
>   - Class C support
> 
>     We record the IP A.B.C.0 parts only so that additional
>     attempts from different A.B.C.0 are also blocked.
> 
>   - Blocking Time
> 
>      55 seconds
> 
>     This was learned. First, I wanted the shorted possible to
>     not hurt timely deliveries with the initial rejection, but
>     many spammers do try almost immediately and 55 was a good
>     delay I found to capture most of these abusers.
> 
>   - Sender White Listed Time
> 
>      96 hours
> 
>     For us, if set at 0, then the accepted sender after
>     being greylisted is permanently greylisted.  Otherwise,
>     the sender is just white listed for 4 days.
> 
>   - Rejection Point
> 
>     We reject after the DATA is captured and issue an
>     EOD 451 response when the transaction is greylisted.
> 
>   - Rejection Response
> 
>     Current, we have stock response of:
> 
>       451 Please try again later (TempFail)
> 
> I wish to explore adding a retry=hint concept for the response, so maybe 
> that is something we can work on to come up with a formal format/hint 
> value.
> 
> Additional features or BCP discussions items:
> 
> It will be good to gather GL implementation field experiences out there 
> among the different SMTP developers and operators.  I have a few points 
> here:
> 
> - Auto White Listing
> 
> I would suggest that "auto-whitelist" concepts be part of the BCP 
> recommendations.
> 
> For our system, we use an auto white list concept where if we send mail 
> to an email address and the address is added to an grey-list-autowhite 
> list file that the greylister component will check when a new anonymous 
> sender comes in.  This was a major god-send to not only accept the GL 
> concept, but to deal with customer support issues when we need them to 
> send some logs or data to us.  So during a support call, we will send 
> them an email called a "Permission List" that will auto-whitelist the 
> customer and have them reply back to the email to send the support 
> data.  Otherwise, we have to wait. Although we have a short 55 seconds, 
> we don't know what is the reschedule time for customer's or their 
> company's MTA setup and that can be in the minutes or hour.
> 
> - Rejection point
> 
> We use DATA rejection point only because originally the "confident 
> level" was still low with the then new "odd" concept of initial 
> rejections with an expectation of retrying before accepting the 
> transaction.  I recall operators setting at the greylist monitors where 
> they will see a rejection entry from a legit sender with non-spam email 
> they can view and they sat there wondering when it will retry again.  
> When the sender's MTA has retries that not short, that made some people 
> paranoid so capturing the DATA alleviated the concern that the mail was 
> lost.   They can push the email if they were impatience and of course, 
> they got a dupe eventually.
> 
> But many systems reject at RCPT or MAIL or even at the Connection level 
> with a 421 response. These are of course, lower overhead rejections than 
> a DATA capture and rejection.  So today, I would recommend a RCPT 
> rejection since the GL proof of concept is valid and the lost of legit 
> mail is low or zero. I don't recall a lost of email, delays in retries, 
> but no lost among SMTP compliant MTAs.
> 
> - Recorded Information
> 
> We use IP:FROM:TO but I believe is the common practice among GL 
> systems.  But I have read where some GL systems use the 5322.Message-ID 
> header.
> 
> Of course, if the GL system is rejecting at the connection level, then 
> the only recording it has is the IP and if the GL system is rejected at 
> MAIL FROM, then it is recording IP and MAIL FROM only.
> 
> The only thing I can note here is the blur with using GL at the 
> connection level where a 421 is used.
> 
> Some retry strategies may try the next different MX/IP immediately when 
> they see a 421 assuming it is a Loading restriction.
> 
> However, for us, because of the increasing GL usage at the connection 
> level and this 421 blur, like YAHOO and others, a 421 response is 
> rescheduled rather than try the next different MX/IP, if any, for 
> current outbound attempt.
> 
> -- 
> HLS
> 
> 
> 

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Mon, 17 Oct 2011 11:24:28 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

So far we've heard from exactly one source that claims this is a real pain point.  Actually, that wasn’t' even the claim; it was merely an observed possible optimization.  That's not enough in my book to support any kind of formal standardization effort.  It might sound like a nifty idea and there might be myriad implementations around that do stuff like this, but that doesn't mean there's a large-scale problem with the practice that needs to be solved in the form of protocol changes.

In light of this thread, I polled several large mailbox providers.  None of them say they do greylisting.  One of them (Hotmail) replied that they don't do greylisting because they just don't see enough benefit from it.  None of our large customers do it either.

To quote Dave from another posting just now: Standards are extremely expensive to create and deploy.  So far I haven't seen anything supporting the idea that this is worth the expense.  Maybe what has to happen is that some small team needs to go off and try it, and report back results.

Absent that, I suggest this would at best deserve "Experimental" status.  I'd support that sort of thing, though even those aren't cheap in IETF terms.

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Mon, 17 Oct 2011 11:11:19 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Keith Moore
> Sent: Monday, October 17, 2011 9:37 AM
> To: Hector
> Cc: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> On Oct 17, 2011, at 12:25 PM, Hector wrote:
> > It seems there is a majority consensus that a IETF document for
> > Greylisting would be a worthy effort.
> 
> I'm still unconvinced.  I haven't seen any evidence to support this at
> all.

I disagree.  I think there's enough evidence to support a BCP, but I haven't seen evidence to suggest that a protocol enhancement of any kind would see enough interest to get enough adoption to be worthwhile.

Date: Mon, 17 Oct 2011 14:01:44 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Keith Moore wrote:

>> BTW, Keith, many GL servers already have responses with hints. 
> 
> yes, I know.   but just because someone decided to add a %d to a 
> printf doesn't mean it should be standardized.

Sure, but not just someone.  That was at least 7 different systems 
(based on a different response form) I listed from my quick viewing of 
the logs. In addition, one needs to also consider that an engineering 
thought process was done to add that information.  I can see the need 
for it to alleviate the waste and overhead in rejected delays, the 
redundant retries, network communications and reducing the queuing 
pressures.  I'm sure if a formal standard was available, it would of 
been used and perhaps others will also implement as well.

--
HLS

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Mon, 17 Oct 2011 13:46:26 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

On Oct 17, 2011, at 1:39 PM, Hector wrote:

> Hector wrote:
>> Keith Moore wrote:
>>> Personally I think writing up an RFC on greylisting will only dilute its effectiveness as a spam countermeasure.
>> I don't agree with this concern.
>> ...
>> So I see no negatives at all here.

I'm not a fan of creeping featurism.  New features need justification.  So far, nobody has supplied any.  (or maybe I've missed it). 

> BTW, Keith, many GL servers already have responses with hints. 

yes, I know.   but just because someone decided to add a %d to a printf doesn't mean it should be standardized.

Keith

Date: Mon, 17 Oct 2011 13:39:50 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Hector wrote:
> Keith Moore wrote:
>> Personally I think writing up an RFC on greylisting will only dilute 
>> its effectiveness as a spam countermeasure.
> 
> I don't agree with this concern.
> 
> ...
> 
> So I see no negatives at all here.

BTW, Keith, many GL servers already have responses with hints. From my 
logs, I have collected these varying responses with hints:

   421 This server implements greylisting, please try again in # seconds
   450 4.7.1 <RCPT>: Recipient address rejected: Greylisted for # minutes
   450 4.7.1 <RCPT>: Recipient address rejected: Greylisted for # seconds
   451 4.7.1 Greylisting in action, please come back in HH:MM:SS
   451 Greylisted for # seconds
   451 Greylisted, please try again in # seconds
   451 Greylisting enabled, try again in # minutes

I don't know of a current client MTA that leverages this yet for 
rescheduling, but I plan to explore parsing these responses to apply 
to rescheduling to see how that improves thing (it expect it will).

One goal would be to formalize this response.

--
HLS

Date: Mon, 17 Oct 2011 13:29:24 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Keith Moore wrote:

>> Hector responded to Keith:
>> hmmm,  unless I mis-read people's input, many people contributing to 
>> the thread recognized there are issues and most have indicated a note 
>> that a IETF document for greylisting would be a good thing.
>>
>> I believe you cited an interest in an SMTP extension for a retry hint.
> 
> I mentioned how it could be done, but also expressed concern that it might 
> not be worth the trouble... also I said that greylisting wasn't the 
> reason for such an extension.

GL would be among the reasons as a retry=hint proposal would apply to 
all 4yz reasons a remote receiver may have, which in practical terms 
today are receivers:

    - Implementing GreyListing
    - Applying any Connection Load Restrictions

That is the reality among many servers that can not be denied.  Either 
421 or 45x responses are issued and they are applied in varying ways, 
in other words, some will use 421 with a text response indicating a 
GreyListed action at the connection or as a response to MAIL FROM, 
RCPT or DATA.

The restrictions whether due to loading or greylisting, delays varies 
as well,  some have very short, some have longer delays.

> Personally I think writing up an RFC on greylisting will only 
> dilute its effectiveness as a spam countermeasure.

I don't agree with this concern.

First, if one is to assume adaptation will  takes place, one can not 
just assume only the negative adaption, but both the positive and 
negative.  The positive adaption will provide a tremendous technical 
benefit in reduce traffic on the network, and also improve system 
availability and better throughput.

Second, what is the negative adaptation?

A bad guy learning when (time wise) to hit a server?

GL #1 benefit is addressing the "single shot" random blaster of mail 
when a 4yz means to TRY again with the same information, not different 
information.

The bad guy just needs to try again with the same information if he 
wanted to get around GL restriction.  If he can't do that, a 
retry=hint will not going to get him in any shape of form.

But when about the spammer who does try again?  Well, GL allows it as 
long as its done within after the blocking time.

If the spammer adapts to the retry hint, not only does it help him, it 
also helps the server by reducing the rejection overhead.   For us, 
even with a 55 second blocking time, many spammers will redundantly 
retry 3-4 or more times within 55 seconds before its finally accepted. 
   If they adapted to the retry=hint, that equal a tremendous amount 
of overhead rejection on both ends.

So I see no negatives at all here.

Can you describe how the GL effectiveness is diluted with a retry=hint 
concept?

--
HLS

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Mon, 17 Oct 2011 13:07:28 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

> Keith Moore wrote:
>> On Oct 17, 2011, at 12:25 PM, Hector wrote:
>>> It seems there is a majority consensus that a IETF document for Greylisting would be a worthy effort.
>> I'm still unconvinced.  I haven't seen any evidence to support this at all.
> 
> hmmm,  unless I mis-read people's input, many people contributing to the thread recognized there are issues and most have indicated a note that a IETF document for greylisting would be a good thing.
> 
> I believe you cited an interest in an SMTP extension for a retry hint.

I mentioned how it could be done, but also expressed concern that it might not be worth the trouble... also I said that greylisting wasn't the reason for such an extension.

Personally I think writing up an RFC on greylisting will only dilute its effectiveness as a spam countermeasure.

Keith

Date: Mon, 17 Oct 2011 12:55:47 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Keith Moore wrote:
> On Oct 17, 2011, at 12:25 PM, Hector wrote:
> 
>> It seems there is a majority consensus that a IETF document for 
>> Greylisting would be a worthy effort.
> 
> I'm still unconvinced.  I haven't seen any evidence to support this at all.

hmmm,  unless I mis-read people's input, many people contributing to 
the thread recognized there are issues and most have indicated a note 
that a IETF document for greylisting would be a good thing.

I believe you cited an interest in an SMTP extension for a retry hint.

If I understand the "IETF" document breaking, as Murray describe, a 
BCP for Greylisting could not proposed a change to SMTP so based on 
that comment, they would be two documents:

    BCP - For GreyListing
    SMTP Extension for a Retry=Hint

If people believe the SMTP extension can cover both, that is fine by me.

In addition, as Steve Atkins proposed, if we wish to wrap this under a 
"SMTP Traffic Management" guideline, that also fine by me.

But I do believe that will begin to make it all more complicated if it 
goes beyond the basic 4zy issues mostly related to Greylisting and the 
need for an optional retry hint structured response concept.

--
HLS

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Mon, 17 Oct 2011 12:37:07 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

On Oct 17, 2011, at 12:25 PM, Hector wrote:

> It seems there is a majority consensus that a IETF document for Greylisting would be a worthy effort.

I'm still unconvinced.  I haven't seen any evidence to support this at all.

Keith

Date: Mon, 17 Oct 2011 12:25:18 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Hector wrote:

> ...
> I think someone should take up the effort to begin/draft an GreyList BCP 
> for systems to follow.
> 
> Comments?

Hi,

It seems there is a majority consensus that a IETF document for 
Greylisting would be a worthy effort.

I do not have the time, so I am hoping some other interested party can 
write it. But if not, I may consider writing this document. I can 
certainly help co-author it. If someone wishes to assist, please write 
me off list.

I sent a note to Evan Harris, AFAIK, the original author of 
Greylisting asking for his interest in authoring or contributing to 
the development official IETF GreyListing document.   I seem to recall 
long ago, a comment he made that he had avoided the IETF knowing there 
would be major resistance with some an "odd idea" of using an initial 
rejection as a form to filter a certain form of "single shot" random 
spammers.  I agreed 100% the IETF resistance would of been there, but 
that was back then in 2003. Not today. Attitudes have changed about GL 
and now you see GL implementation widely adopted, even among the early 
IETF critics.

Maybe to start such a document, we can gather the following regarding 
the basic GL framework I think many use or part of the following:

   - Recorded Information
   - Class C support
   - Blocking Time
   - White Listed Time
   - Rejection Point
   - Rejection response

For example, for our GL implementation:

   - Recorded Information

       Connecting IP, MAIL FROM, RCPT TO

   - Class C support

     We record the IP A.B.C.0 parts only so that additional
     attempts from different A.B.C.0 are also blocked.

   - Blocking Time

      55 seconds

     This was learned. First, I wanted the shorted possible to
     not hurt timely deliveries with the initial rejection, but
     many spammers do try almost immediately and 55 was a good
     delay I found to capture most of these abusers.

   - Sender White Listed Time

      96 hours

     For us, if set at 0, then the accepted sender after
     being greylisted is permanently greylisted.  Otherwise,
     the sender is just white listed for 4 days.

   - Rejection Point

     We reject after the DATA is captured and issue an
     EOD 451 response when the transaction is greylisted.

   - Rejection Response

     Current, we have stock response of:

       451 Please try again later (TempFail)

I wish to explore adding a retry=hint concept for the response, so 
maybe that is something we can work on to come up with a formal 
format/hint value.

Additional features or BCP discussions items:

It will be good to gather GL implementation field experiences out 
there among the different SMTP developers and operators.  I have a few 
points here:

- Auto White Listing

I would suggest that "auto-whitelist" concepts be part of the BCP 
recommendations.

For our system, we use an auto white list concept where if we send 
mail to an email address and the address is added to an 
grey-list-autowhite list file that the greylister component will check 
when a new anonymous sender comes in.  This was a major god-send to 
not only accept the GL concept, but to deal with customer support 
issues when we need them to send some logs or data to us.  So during a 
support call, we will send them an email called a "Permission List" 
that will auto-whitelist the customer and have them reply back to the 
email to send the support data.  Otherwise, we have to wait. Although 
we have a short 55 seconds, we don't know what is the reschedule time 
for customer's or their company's MTA setup and that can be in the 
minutes or hour.

- Rejection point

We use DATA rejection point only because originally the "confident 
level" was still low with the then new "odd" concept of initial 
rejections with an expectation of retrying before accepting the 
transaction.  I recall operators setting at the greylist monitors 
where they will see a rejection entry from a legit sender with 
non-spam email they can view and they sat there wondering when it will 
retry again.  When the sender's MTA has retries that not short, that 
made some people paranoid so capturing the DATA alleviated the concern 
that the mail was lost.   They can push the email if they were 
impatience and of course, they got a dupe eventually.

But many systems reject at RCPT or MAIL or even at the Connection 
level with a 421 response. These are of course, lower overhead 
rejections than a DATA capture and rejection.  So today, I would 
recommend a RCPT rejection since the GL proof of concept is valid and 
the lost of legit mail is low or zero. I don't recall a lost of email, 
delays in retries, but no lost among SMTP compliant MTAs.

- Recorded Information

We use IP:FROM:TO but I believe is the common practice among GL 
systems.  But I have read where some GL systems use the 
5322.Message-ID header.

Of course, if the GL system is rejecting at the connection level, then 
the only recording it has is the IP and if the GL system is rejected 
at MAIL FROM, then it is recording IP and MAIL FROM only.

The only thing I can note here is the blur with using GL at the 
connection level where a 421 is used.

Some retry strategies may try the next different MX/IP immediately 
when they see a 421 assuming it is a Loading restriction.

However, for us, because of the increasing GL usage at the connection 
level and this 421 blur, like YAHOO and others, a 421 response is 
rescheduled rather than try the next different MX/IP, if any, for 
current outbound attempt.

--
HLS

Date: Mon, 17 Oct 2011 10:43:48 -0400
From: Hector <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

+1

Dave CROCKER wrote:
> 
> On 10/16/2011 10:57 PM, Murray S. Kucherawy wrote:
>>> From: [email-masked] 
>>> [mailto:[email-masked]] On Behalf Of Richard Kulawiec
>>> Maybe a shorter way to put that last paragraph is that if we agree that
>>> reasonably timely delivery of mail is a Good Thing, then I think we have
>>> many other problems to tackle before we turn our attention to
>>> greylisting.
>>
>> Agree here too.
> 
> Greylisting is a widely used, undocumented set of features.  Even the 
> meaning of the term is ambiguous.  For a widespread 'feature' of a 
> critical infrastructure service, that's a generically Bad Thing.
> 
> It also has significant downsides, including delivery failure of 
> legitimate mail.  (I realize the thread so far has been unsympathetic to 
> this, but it's a real issue.)
> 
> Any activity qualifying with the above needs documentation.
> 
> As for priorities, the wishlist of things to do is infinite.  This issue 
> is here now.  It's gotten enough attention to sustain a thread.  That 
> makes it worth pursuing, if it doesn't take too much effort.
> 
> d/
> 

Date: Mon, 17 Oct 2011 10:40:30 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

John C Klensin wrote:
> 
> --On Sunday, October 16, 2011 22:57 -0700 "Murray S. Kucherawy"
> <[email-masked]> wrote:
> 
>>> Maybe a shorter way to put that last paragraph is that if we
>>> agree that reasonably timely delivery of mail is a Good
>>> Thing, then I think we have many other problems to tackle
>>> before we turn our attention to greylisting.
>> Agree here too.
> 
> +1
> 
> And even more strongly agree if "delivery of all legitimate mail
> without forcing senders to use a small number of senders with
> bilateral arrangements" is added to the list.

Hello John,

I hope we are not losing focus here.

No one is talking (at least I wasn't) about the unrealistic 
expectation that any receiver/server should adhere to the demands of 
the "anonymous" clients to accept mail immediately.  We never worked 
that way, nor can one expect it and SMTP has enough robustness to 
allow delivery within a certain period.

But in my view, the retry concept was meant for operational issues for 
which GL has  leverage successfully.  However, with greater adoption 
over the years, I believe it has changed now to one where its is an 
automatic expectation where GL frontends are par for the course.

That in itself is not too bad, it actually works and the growth of it 
is the proof of how it helps to filter legacy "single shot" spamers.

But when it becomes where its used inconsistently among good systems 
and the retry attempts go beyond the first time and now the RFC5321 
recommendations for an initial short delay with a back off pattern 
begin to become less effective, then in my engineering opinion, this 
begins to add design pressure to consider new queuing strategies.

As we discussed offlist, when you also have a growing amount of 
Anti-Spam service bureaus hosting for many different email domains, a 
Queue-By-Email-Domain is less adequate when a centralized host is 
greylisting many different domains.

So its not about any client expecting that an email should be 
immediately accepted in short time, its more about controlling the 
unreasonable delays when its not necessary.  If a GL system goal is to 
test that you are a not a "Single Shot" random abuser, then rejecting 
multiple times after the client has agreed to delay it for X minutes 
is not reasonable.

If a receiver is doing this because it its a heavy loaded system and 
clients are unfortunately hitting it during high loads, then we should 
be able to add some client/server negotiated smarts to reduce all the 
increasing overhead for a client and also the network.

That said,  I don't think its a good idea, and I hope I am mis-reading 
this, that we begin to alter the mindset of industry that TIMELY 
DELIVERY is not to be expected.

--
HLS

Date: Mon, 17 Oct 2011 06:53:49 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/16/2011 10:57 PM, Murray S. Kucherawy wrote:
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Richard Kulawiec
>> Maybe a shorter way to put that last paragraph is that if we agree that
>> reasonably timely delivery of mail is a Good Thing, then I think we have
>> many other problems to tackle before we turn our attention to
>> greylisting.
>
> Agree here too.

Greylisting is a widely used, undocumented set of features.  Even the meaning of 
the term is ambiguous.  For a widespread 'feature' of a critical infrastructure 
service, that's a generically Bad Thing.

It also has significant downsides, including delivery failure of legitimate 
mail.  (I realize the thread so far has been unsympathetic to this, but it's a 
real issue.)

Any activity qualifying with the above needs documentation.

As for priorities, the wishlist of things to do is infinite.  This issue is here 
now.  It's gotten enough attention to sustain a thread.  That makes it worth 
pursuing, if it doesn't take too much effort.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 17 Oct 2011 06:48:33 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 10/13/2011 5:08 PM, Claus Assmann wrote:
>
> On Thu, Oct 13, 2011, Dave CROCKER wrote:
>
>> What client implementors will add this to their code?  What server
>
> I will.

Individual statements do not justify an international standards effort.  What is 
needed is a basis for assessing likely /industry/ adoption.  A few individuals 
are not a representative sample of an industry.

>> In other words, no matter how nice this all sounds -- and I happen
>> to agree that it does sound nice -- it's /very/ expensive to add to
>> the infrastructure and it needs a good indication that it will be
>> adopted.  Where is the market 'pull' for the benefits this will
>> provide?
>
> Sorry, I'm not in marketing, I simply implement things I like.

That's fine, but it's a very different job, with a very different skillset, for 
determining what industry standards need to be developed.  To a large extent, 
the latter /does/ require the ability to assess a (likely) market.

> "Build it and they will come"?

which usually proves wrong.

standards are extremely expensive to create and deploy.  their success should 
not be a matter of pure gambling.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Mon, 17 Oct 2011 07:51:47 -0400
From: John C Klensin <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>, [email-masked]
Subject: RE: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

--On Sunday, October 16, 2011 22:57 -0700 "Murray S. Kucherawy"
<[email-masked]> wrote:

>> Maybe a shorter way to put that last paragraph is that if we
>> agree that reasonably timely delivery of mail is a Good
>> Thing, then I think we have many other problems to tackle
>> before we turn our attention to greylisting.
> 
> Agree here too.

+1

And even more strongly agree if "delivery of all legitimate mail
without forcing senders to use a small number of senders with
bilateral arrangements" is added to the list.

   john

Date: Mon, 17 Oct 2011 07:49:12 -0400
From: Hector <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Murray S. Kucherawy wrote:

>> On Sun, Oct 16, 2011 at 01:42:18PM -0400, Hector wrote:
>>> Here is the thing, IMO, the prize at the end of the day is:
>>>
>>>       Fastest, less waste in Delivery and High Throughput
>>>
>>> For many, an email today has "No Value" tomorrow, especially in the
>>> business world.  I'm sure that is the normal expectation for most
>>> systems and people expecting timely emails.

>> Richard Kulawiec stated:
>>
>> The problem here, I think, isn't the technology: it's unrealistic and/or
>> misplaced expectations.  If I might quote my own haiku on this subject:
>>
>> 	Mail is not the web
>> 	Mail is not for file transfer
>> 	Mail is not I M
>>
>> It's certainly true that a subset of people expect mail to be very
>> fast; it might even be true that this subset is large enough that
>> it's "normal".  But I don't think that necessarily means we should
>> engineer to meet that expectation, just as I don't think the propensity
>> of some people to attempt transfer of 100M files via email means that
>> we should engineer to make that feasible.
> 
> I agree.  What's more, I think the "no value" premise seems to 
> contradict the massive efforts undertaken by some very large companies 
> to maintain user trust in email.  I have to think given the weight behind 
> those efforts that it's not so easily dismissed as a fool's errand.

Who said anything dismissing efforts?  For many, the longer email is 
delayed, the less value it has. That "premise" is not contradictive of 
anything or any efforts to improve the timely/acceptance delivery 
problem. There are many representations here - the issues apply to 
everyone who is the business of supplying SMTP software or who operate 
mail systems of any size.  Lets not create a chasm once again.

>> Maybe a shorter way to put that last paragraph is that if we agree that
>> reasonably timely delivery of mail is a Good Thing, then I think we have
>> many other problems to tackle before we turn our attention to
>> greylisting.
> 
> Agree here too.

It timely delivery is still important.  But the reality is GL is 
contributing to that problem.  GL is something that we can grasp and 
so far, almost everyone here agrees a IETF document should help in 
some manner, if only to document the BCP.

I don't think you are grasping that for many GL rejections is going 
beyond the first attempt.

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Sun, 16 Oct 2011 22:57:30 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Richard Kulawiec
> Sent: Sunday, October 16, 2011 11:15 AM
> To: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> On Sun, Oct 16, 2011 at 01:42:18PM -0400, Hector wrote:
> > Here is the thing, IMO, the prize at the end of the day is:
> >
> >       Fastest, less waste in Delivery and High Throughput
> >
> > For many, an email today has "No Value" tomorrow, especially in the
> > business world.  I'm sure that is the normal expectation for most
> > systems and people expecting timely emails.
> 
> The problem here, I think, isn't the technology: it's unrealistic and/or
> misplaced expectations.  If I might quote my own haiku on this subject:
> 
> 	Mail is not the web
> 	Mail is not for file transfer
> 	Mail is not I M
> 
> It's certainly true that a subset of people expect mail to be very
> fast; it might even be true that this subset is large enough that
> it's "normal".  But I don't think that necessarily means we should
> engineer to meet that expectation, just as I don't think the propensity
> of some people to attempt transfer of 100M files via email means that
> we should engineer to make that feasible.

I agree.  What's more, I think the "no value" premise seems to contradict the massive efforts undertaken by some very large companies to maintain user trust in email.  I have to think given the weight behind those efforts that it's not so easily dismissed as a fool's errand.

> Maybe a shorter way to put that last paragraph is that if we agree that
> reasonably timely delivery of mail is a Good Thing, then I think we have
> many other problems to tackle before we turn our attention to
> greylisting.

Agree here too.

-MSK

Date: Sun, 16 Oct 2011 18:37:47 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Richard Kulawiec <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

My long 30 years of mail engineering experience has given me the 
position and view that non-timely delivery (i.e. unreasonable 
hindrance) is the exception and not the rule. I don't believe the 
industry would of endured such a poor history of operations and I 
believe, possibly unlike some others, all mail systems were designed 
to deliver mail in a timely matter. I would certainly get out of the 
market if this system became a "game" with laissez-faire attitudes 
that SMTP is a matter of just flipping a coin, heads and tails.  No, I 
still believe there is a strong ethical engineering responsibility by 
the majority of the SMTP systems, in total that make it all work, who 
still believe timely delivery is important. If that was not the case, 
for me, I would of been long out of the market.

That said, I do worry that Greylisting (GL) is growing with a varying 
non-consistency - overhead are increasings. There is no IETF guideline 
and IMTO, its about time it did.  I recall, not too long ago, many key 
people here thought GL was wrong (hence the idea of a IETF I-D was not 
even a consideration) and today, you see these same people are GL 
implementators.  So attitudes have changed to accept GL with an 
effective slightly lower the "SMTP Quality" in the name of SPAM.

Its the same thing with CBV (Callback Verification). CBV works 
wonderfully, but its not something I would recommend to be used 
widely. It would not scale.  Its not something everyone can do.

I am suggesting that we begin to nip this GL bud before it gets out of 
hand, when every server you hit has implemented GL and if left as is, 
in essence, every transaction will now take two or more attempts.

Just imagine if GMAIL.COM began to do Greylisting?  One can stand 
maybe YAHOO with its GL derivative, but with more of the bigger ESPs 
doing the same thing, and it will get out of hand.

I will say that if left as is, the SMTP recommendation of an initial 
short delay, is not effective any more - you might as well leave it to 
1 hr per each attempt and you might be better off for non-wasteful 
deliveries.

--

Richard Kulawiec wrote:
> On Sun, Oct 16, 2011 at 01:42:18PM -0400, Hector wrote:
>> Here is the thing, IMO, the prize at the end of the day is:
>>
>>       Fastest, less waste in Delivery and High Throughput
>>
>> For many, an email today has "No Value" tomorrow, especially in the
>> business world.  I'm sure that is the normal expectation for most
>> systems and people expecting timely emails.
> 
> The problem here, I think, isn't the technology: it's unrealistic and/or
> misplaced expectations.  If I might quote my own haiku on this subject:
> 
> 	Mail is not the web
> 	Mail is not for file transfer
> 	Mail is not I M
> 
> It's certainly true that a subset of people expect mail to be very
> fast; it might even be true that this subset is large enough that
> it's "normal".  But I don't think that necessarily means we should
> engineer to meet that expectation, just as I don't think the propensity
> of some people to attempt transfer of 100M files via email means that
> we should engineer to make that feasible.
> 
> On the other side of that, I do recognize that inordinate mail delays
> do occasionally matter; but I'll argue that (a) these are rare, absent
> outages/misconfigurations that would still persist even if greylisting
> were handled very effectively and (b) these are often caused by
> non-greylisting measures of dubious use.  (I'll cite "quarantines" and
> "quotas" as examples of the latter, but will omit for brevity the lengthy
> arguments I've made elsewhere about why they're a horrible idea.)
> (And let me cite "poor choice of mail service technology or provider"
> as another, especially in light of the past week's events involving
> a well-known operation which is experiencing systemic, prolonged outages,
> please see the NANOG list for extensive discussion.)
> 
> Maybe a shorter way to put that last paragraph is that if we agree that
> reasonably timely delivery of mail is a Good Thing, then I think we have
> many other problems to tackle before we turn our attention to greylisting.
> 
> ---rsk
> 
> 
> 

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Sun, 16 Oct 2011 14:15:06 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On Sun, Oct 16, 2011 at 01:42:18PM -0400, Hector wrote:
> Here is the thing, IMO, the prize at the end of the day is:
> 
>       Fastest, less waste in Delivery and High Throughput
> 
> For many, an email today has "No Value" tomorrow, especially in the
> business world.  I'm sure that is the normal expectation for most
> systems and people expecting timely emails.

The problem here, I think, isn't the technology: it's unrealistic and/or
misplaced expectations.  If I might quote my own haiku on this subject:

	Mail is not the web
	Mail is not for file transfer
	Mail is not I M

It's certainly true that a subset of people expect mail to be very
fast; it might even be true that this subset is large enough that
it's "normal".  But I don't think that necessarily means we should
engineer to meet that expectation, just as I don't think the propensity
of some people to attempt transfer of 100M files via email means that
we should engineer to make that feasible.

On the other side of that, I do recognize that inordinate mail delays
do occasionally matter; but I'll argue that (a) these are rare, absent
outages/misconfigurations that would still persist even if greylisting
were handled very effectively and (b) these are often caused by
non-greylisting measures of dubious use.  (I'll cite "quarantines" and
"quotas" as examples of the latter, but will omit for brevity the lengthy
arguments I've made elsewhere about why they're a horrible idea.)
(And let me cite "poor choice of mail service technology or provider"
as another, especially in light of the past week's events involving
a well-known operation which is experiencing systemic, prolonged outages,
please see the NANOG list for extensive discussion.)

Maybe a shorter way to put that last paragraph is that if we agree that
reasonably timely delivery of mail is a Good Thing, then I think we have
many other problems to tackle before we turn our attention to greylisting.

---rsk

Date: Sun, 16 Oct 2011 13:42:18 -0400
From: Hector <[email-masked]>
To: Richard Kulawiec <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Richard Kulawiec wrote:
> On Thu, Oct 13, 2011 at 10:25:35PM +0100, Paul Smith wrote:
>> Anyway, what are 'the existing retry schedules that MTAs use'? If
>> they were standardised then it would be a great idea to work with
>> them, but they're not, and every MTA could be using a different
>> retry schedule!
> 
> Many MTAs are (apparently) using a different retry schedule, and that's a
> good thing.  (Well...provided those retry schedules are reasonably sane,
> that is, not way too fast.)  I say that it's good because it helps scatter
> retries rather than concentrating them.
> 
> (Consider, as a counter-example, if there were a way to communicate retry
> availability back to client MTAs.  Suppose an MX says "Please try again
> at 10 AM".  Suppose that client MTAs pay attention to this.  ALL of them.
> Then at 10:00:01 AM we might expect that the MX might find itself on
> the receiving end of quite a few connection attempts.  This may turn out
> to be much worse for all concerned than just letting client MTAs retry
> on their own varied schedules.)

Good point Richard, however, speaking for our MTA, we already deal 
with such similar resultant grouping issues when the MTA implements a 
retry granularity (i.e. reschedule by the min or hour) factor. 
Coupled with a Queue-By-Domain strategy, it help gathers future 
attempts as multi-transaction sessions when rejected attempts come in 
at spread out times.  The problem I have observed here is when many 
different domains all post to the same MX/IP host.  Not having a 
granularity can reduce this issue.

Nonetheless, I support a proposed concept for a 4yz retry=hint because 
a supportive client will most likely factor all this as part of its 
overall rescheduling/queuing algorithm, along with other possible 
factors it may already consider.  I do think, there is a different 
between 421 and 450 and maybe a BCP can help here.

Here is the thing, IMO, the prize at the end of the day is:

       Fastest, less waste in Delivery and High Throughput

For many, an email today has "No Value" tomorrow, especially in the 
business world.  I'm sure that is the normal expectation for most 
systems and people expecting timely emails.

While we can't do anything about legacy software, I think active SMTP 
systems and the network can benefit greatly with a sound BCP to better 
coordinate SMTP client/server traffic and to help alleviate the 
increasing delays, wasted retries, DNS overheads and IMO, new, complex 
queuing strategy considerations to address the issues.

-- 
HLS

Date: Sun, 16 Oct 2011 09:52:44 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On Thu, Oct 13, 2011 at 10:25:35PM +0100, Paul Smith wrote:
> Anyway, what are 'the existing retry schedules that MTAs use'? If
> they were standardised then it would be a great idea to work with
> them, but they're not, and every MTA could be using a different
> retry schedule!

Many MTAs are (apparently) using a different retry schedule, and that's a
good thing.  (Well...provided those retry schedules are reasonably sane,
that is, not way too fast.)  I say that it's good because it helps scatter
retries rather than concentrating them.

(Consider, as a counter-example, if there were a way to communicate retry
availability back to client MTAs.  Suppose an MX says "Please try again
at 10 AM".  Suppose that client MTAs pay attention to this.  ALL of them.
Then at 10:00:01 AM we might expect that the MX might find itself on
the receiving end of quite a few connection attempts.  This may turn out
to be much worse for all concerned than just letting client MTAs retry
on their own varied schedules.)

---rsk

Date: Fri, 14 Oct 2011 12:44:10 -0400
From: Hector <[email-masked]>
To: Paul Smith <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Greylisting retry hints
Sender: [email-masked]

Paul Smith wrote:
> 
> It wouldn't even need to be much, just respond
> 
> 421 4.2.x [retry:300s] connection refused for now. Try again later
> 
> it wouldn't interfere with MTAs which don't understand it, and would 
> give a hint to those which do.

FWIW, Here are some forms of apparent GL responses with hints I have 
collected:

421 This server implements greylisting, please try again in # seconds
450 4.7.1 <RCPT>: Recipient address rejected: Greylisted for # minutes
450 4.7.1 <RCPT>: Recipient address rejected: Greylisted for # seconds
451 4.7.1 Greylisting in action, please come back in HH:MM:SS
451 Greylisted for # seconds
451 Greylisted, please try again in # seconds
451 Greylisting enabled, try again in # minutes

--
HLS

Date: Fri, 14 Oct 2011 00:46:07 -0400
To: SMTP Interest Group <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

At 15:34 -0700 on 10/13/2011, SM wrote about Re: SMTP traffic control:

>>It would also be helpful to have an enhanced status code that tells
>>a client: "don't even try one of the other MXs, it will reject you
>>too." That can be rather helpful for greylisting and for systems
>>that share traffic control information.
>
>This one may be doable as currently the SMTP client will retry with 
>the next available host on a temporary error.  As it would be a 
>significant departure from the existing SMTP specifications, it 
>would be better to do it through a SMTP extension.

The support that would allow the MXs to sync with the main MX (so 
when the main MX blows off a sender with a "try again no sooner than 
time" so will the secondary MX) would also help to stop those 
spammers who try to send to a secondary MX as a means of attempting 
to bypass spam controls at the main MX. Any attempt to connect to the 
secondary MX when the primary MX is available and accepting messages 
would cause the secondary to blow off the sender.

Date: Thu, 13 Oct 2011 23:07:18 -0400
From: Hector <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

Claus Assmann wrote:
> It would be nice to have a standard for a server to tell a client
> which restrictions it triggered, what the limits are, and what it
> should do to avoid this from happening again. For example:
> 
> - number of sessions per time unit
> - concurrent number of sessions
> - number of transactions per session
> ...
> 
> It would also be helpful to have an enhanced status code that tells
> a client: "don't even try one of the other MXs, it will reject you
> too." That can be rather helpful for greylisting and for systems
> that share traffic control information.
> 
> 
> Anyone interested in specifying this?

Thinking about it, initially I would have an issue with the exposure 
of this information, but then it may not be a problem since a server 
will be securing its capacity against DoS attacks.

Speaking for our MTA design, we have certain features, although not 
automated yet, operators can limit their outbound volumes:

      concurrent connections per IP
      multiple transactions per session

from what they they see in the Queue Monitor and transaction results.

So every attempt, for example, has a log entry:

   Total Connections: X  Current IP [A.B.C.D] Connections: Y

The operator can now learn when remote systems begin to show 421 
connection results and add a limit fro this site.

Our plan to automate the learning.

So would "Server Capacity" exposure help?

Well, if it was available today, I think currently actively supported 
adaptive software vendors will consider it.

Mind you, its only an issue when dealing with volume because most 
systems don't need it.

What is needed though is a a "RETRY=HINT" concept for GREYLISTING 
because this is what everyone is beginning to see, from small to large.

The RETRY=HINT concept could/would be part of this Server Capability 
exposure too, but IMTO, it could also be the only result needed for a 
client as it is the ultimate outcome to assist in scheduling.

If we wanted to know more about server operations, helpful information 
would be when a SYSTEM HOURS are too as I have seen remote sites where:

   - Accept mail only after hours,
   - Accept in the morning, stop after hours.

and similar behavior.

Finally, its not an odd idea to have "System Hours" exposed as that 
ideat was par for the course in the Fidonet Mailer era.  You were 
either 24x7 or not and if not, the minimum requirement was 1 hour 
called the ZONE MAIL HOUR (ZMH) and this was to allow for HOST ROUTING 
during this period.  The following was part of Fidonet Policy 4 document:

10.2  Timing of Zone Mail Hour

Zone Mail Hour is observed each day, including weekends and holidays.  The
time is based upon Universal Coordinated Time (UTC), also known as 
Greenwich
Mean time (GMT).  In areas which observe Daylight Savings Time during 
part of
the year, the local time of zone mail hour will change because FidoNet 
does
not observe Daylight Savings Time. The exact timing of Zone Mail Hour 
is set
for each zone by the Zone Coordinator.

In FidoNet Zone 1, Zone Mail Hour is observed from 0900 to 1000 UTC. 
In each
of the time zones, this is:

      Eastern Standard Time          4 AM to 5 AM
      Central Standard Time          3 AM to 4 AM
      Mountain Standard Time          2 AM to 3 AM
      Pacific Standard Time          1 AM to 2 AM
      Hawaii Standard Time          11 PM to Midnight

In FidoNet Zone 2, Zone Mail Hour is observed from 0230 to 0330 UTC.

In Fidonet Zone 3, Zone Mail Hour is observed from 1800 to 1900 UTC. 
In each
of the time Zones involved this is:

   GMT +12 Zone                        6:00 AM to 7:00 AM
   (New Zealand)

   GMT +10 Zone                        4:00 AM to 5:00 AM
   (East Australia)
   (Papua New Guinea)
   (Micronesia)

   GMT +9.5 Zone                       3:30 AM to 4:30 AM
   (Central Australia)

   GMT +9 Zone                         3:00 AM to 4:00 AM
   (Japan)
   (Korea)
   (Eastern Indonesia)

   GMT +8 Zone                         2:00 AM to 3:00 AM
   (Hong Kong)
   (Taiwan)
   (Central Indonesia)
   (Philippines)
   (Western Australia)

   GMT +7 Zone                         1:00 AM to 2:00 AM
   (Malaysia)
   (Singapore)
   (Thailand)
   (Western Indonesia)

Could we do something like that for Internet Email public port policies?

I seem to doubt it, but who knows.  I will note that one of the 
beauties of migrating to SMTP was that it did REDUCE these advanced 
scheduling concepts of the early days where channels were more scared 
(dialup was the norm).

But if some SMTP vendor started it for their own MTAs, you might to 
see others follow it.

If a remote site is going to REJECT mail until a certain time period, 
it would help MTAs to reduce the overhead and that helps the general 
network as well in the long run.

--
HLS

Date: Thu, 13 Oct 2011 18:07:02 -0700
From: Douglas Otis <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 10/13/11 3:30 PM, Murray S. Kucherawy wrote:
> On Thursday, October 13, 2011 3:18 PM, Keith Moore wrote:
>> Before we invest any significant amount of energy in this idea, I'd
>> like to hear from a few large mail service operators who think they'd
>> implement this (on client and/or server ends) and benefit from it.
> I've asked.  Waiting to hear back from any of them.
When our systems were heavily over-loaded, it was difficult to know the 
number of failed connections since this was not seen in MTA logs.  By 
adding additional MTAs, only then did such connection failures become 
apparent.  When we introduced a temp error strategy, there were a few 
non-complaint MTAs that would immediately retry.  There were others 
using compliant MTAs that still retried too soon.  When we asked why, 
they said it was necessary to flush their outbound queues.

It seems doubtful any information added to temp error codes will prove 
beneficial.  Even when receivers know when in the future they will be 
willing to accept questionable traffic and wish to accommodate those 
that adhere to their requested retry intervals, this strategy would be 
based on a questionable assumption the MTA carry the state to make the 
determinations.  Spammers will call their bluff.

When inbound systems are overloaded, DKIM will not offer relief either.  
If it did, spammers would mimic any  allowance some profile provided.  
Without an upfront light weight AUTHENTICATION scheme, only white-listed 
domains considered "too big to block" have a chance.  Everyone else will 
need to seek their services.  The current snafu benefits the larger 
providers where there is minimal overall reduction in spam.

-Doug

Date: Thu, 13 Oct 2011 17:08:01 -0700
From: Claus Assmann <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Reply-To: [email-masked]
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Thu, Oct 13, 2011, Dave CROCKER wrote:

> What client implementors will add this to their code?  What server

I will.

> implementers will add this to their code?  What sysadmins will

Same guy.

> configure to use this?

Any of our users (if they upgrade to the next version...)

> In other words, no matter how nice this all sounds -- and I happen
> to agree that it does sound nice -- it's /very/ expensive to add to
> the infrastructure and it needs a good indication that it will be
> adopted.  Where is the market 'pull' for the benefits this will
> provide?

Sorry, I'm not in marketing, I simply implement things I like.
Luckily I can do this in two MTAs (one of which is actually used
by more than 5 people :-), so if nobody else is interested, I'll
just do it for "my" MTAs (and let marketing claim this as a
"competitive advantage" :-)

"Build it and they will come"?

Date: Thu, 13 Oct 2011 15:34:55 -0700
To: SMTP Interest Group <[email-masked]>
From: SM <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

At 10:21 13-10-2011, Claus Assmann wrote:
>It would be nice to have a standard for a server to tell a client
>which restrictions it triggered, what the limits are, and what it
>should do to avoid this from happening again. For example:
>
>- number of sessions per time unit
>- concurrent number of sessions
>- number of transactions per session

Yes that would be nice but it is more of a local policy issue.  The 
above comes under connection control where the SMTP server could 
provide the information to the client on a temporary error 
condition.  Some of the information may already be passed out as text.

>...
>
>It would also be helpful to have an enhanced status code that tells
>a client: "don't even try one of the other MXs, it will reject you
>too." That can be rather helpful for greylisting and for systems
>that share traffic control information.

This one may be doable as currently the SMTP client will retry with 
the next available host on a temporary error.  As it would be a 
significant departure from the existing SMTP specifications, it would 
be better to do it through a SMTP extension.

Regards,
-sm 

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Thu, 13 Oct 2011 15:30:12 -0700
Subject: RE: SMTP traffic control
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Keith Moore
> Sent: Thursday, October 13, 2011 3:18 PM
> To: [email-masked]
> Cc: SMTP Interest Group
> Subject: Re: SMTP traffic control
> 
> Before we invest any significant amount of energy in this idea, I'd
> like to hear from a few large mail service operators who think they'd
> implement this (on client and/or server ends) and benefit from it.

I've asked.  Waiting to hear back from any of them.

Subject: Re: SMTP traffic control
From: Keith Moore <[email-masked]>
Date: Thu, 13 Oct 2011 18:17:57 -0400
Cc: SMTP Interest Group <[email-masked]>
To: [email-masked]
Sender: [email-masked]

On Oct 13, 2011, at 5:41 PM, Dave CROCKER wrote:

> In other words, no matter how nice this all sounds -- and I happen to agree that it does sound nice -- it's /very/ expensive to add to the infrastructure and it needs a good indication that it will be adopted.  Where is the market 'pull' for the benefits this will provide?

Before we invest any significant amount of energy in this idea, I'd like to hear from a few large mail service operators who think they'd implement this (on client and/or server ends) and benefit from it.

(The reason I say "large" is that unless there are some providers that generate or accept a significant volume of traffic involved, I have a difficult time imagining how this would help many users other than in a very marginal way.  That's not to say that it wouldn't be useful to some.  The question is,  how many sites and users would benefit in comparison to the potential for disruption?)

Keith

Date: Thu, 13 Oct 2011 14:41:46 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Sender: [email-masked]

On 10/13/2011 10:21 AM, Claus Assmann wrote:
> It would be nice to have a standard for a server to tell a client
> which restrictions it triggered, what the limits are, and what it
> should do to avoid this from happening again. For example:

What client implementors will add this to their code?  What server implementers 
will add this to their code?  What sysadmins will configure to use this?

Why?

What's your empirical basis for making these claims?

In other words, no matter how nice this all sounds -- and I happen to agree that 
it does sound nice -- it's /very/ expensive to add to the infrastructure and it 
needs a good indication that it will be adopted.  Where is the market 'pull' for 
the benefits this will provide?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Thu, 13 Oct 2011 22:25:35 +0100
From: Paul Smith <[email-masked]>
To: John Levine <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 13/10/2011 15:51, John Levine wrote:
>> So, if the server says 'retry in 1 hour', and the client retries every 5
>> minutes, that's no crisis, just a wasted bit of effort for the client
>> and the server. ....
> Since the number of MTAs vastly exceeds the number of greylisters,
> don't you think it would be better for greylisters to work with the
> existing retry schedules that MTAs use than to try to persuade a
> million MTAs to implement a new feature that is of at best marginal
> use to them?
Well, it's not just greylisters, it could be a general load management 
hint. eg a site which normally has 3 MX servers has one down for 
maintenance for a couple of hours, and the other two are very busy, they 
could say 'try again at the time when we expect the system to be back at 
full strength'. This wouldn't necessarily be able to tie in to 'normal' 
retry schedules.

If a sender retries at 5 minutes, 30 minutes, then 2 hours, being able 
to say 'try again in 1 hour because I'm not going to accept new messages 
until then' means the message could be delivered in 60 minutes rather 
than in 155 minutes if the sender heeds that response. If it doesn't, 
then it will still be delivered in 155 minutes as it would have been 
without the hint.

Anyway, what are 'the existing retry schedules that MTAs use'? If they 
were standardised then it would be a great idea to work with them, but 
they're not, and every MTA could be using a different retry schedule!

I suppose I'm biased because I can see how this would help a little bit, 
and it would take about 10 minutes to implement the client and 
greylisting server parts of it in our server, so even if it only helps 
with 0.0001% of connections, it might be worth doing...

The only reason I can see for not specifying a mechanism to allow a 
retry hint it is that it won't help very much, I can't see any realistic 
interoperability or security risks or excess complexity.

Date: Thu, 13 Oct 2011 10:21:33 -0700
From: Claus Assmann <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control
Reply-To: [email-masked]
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

It would be nice to have a standard for a server to tell a client
which restrictions it triggered, what the limits are, and what it
should do to avoid this from happening again. For example:

- number of sessions per time unit
- concurrent number of sessions
- number of transactions per session
...

It would also be helpful to have an enhanced status code that tells
a client: "don't even try one of the other MXs, it will reject you
too." That can be rather helpful for greylisting and for systems
that share traffic control information.

Anyone interested in specifying this?

Date: Thu, 13 Oct 2011 13:06:29 -0400
From: Hector <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: SMTP traffic control (was Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Steve Atkins wrote:
> 
> On Oct 13, 2011, at 7:51 AM, John Levine wrote:
> 
>>> So, if the server says 'retry in 1 hour', and the client retries every 5 
>>> minutes, that's no crisis, just a wasted bit of effort for the client 
>>> and the server. ....
>> Since the number of MTAs vastly exceeds the number of greylisters,
>> don't you think it would be better for greylisters to work with the
>> existing retry schedules that MTAs use than to try to persuade a
>> million MTAs to implement a new feature that is of at best marginal
>> use to them?
> 
> We're not talking about greylisting, and haven't been for quite a 
> few levels of this thread. We're talking about use of 4xx deferrals
> for traffic management and prioritization.

However, the blur includes both and and I would suggest it be valid in 
a related BCP it will include current considerations for how clients 
have been designed to behave and follow SMTP retry recommendations.

Ultimately, a "retry=hint" suggestion would apply to both concepts.

I think that in most cases, traffic control is done with 421 at the 
connection level, and traditionally GL is done at either RCTP or DATA. 
  But now you do see it at connections too - based on the varied text 
response of what I see out there, like

     421 too many connections
     421 temporary rejection - try again later

Who knows if the latter is just another way of limiting loads, but if 
a GL concept, it only based on IP only which IMO, it not a true GL 
concept. i.e. yahoo.com blocking people at the connection level with 
some secret source formula.

--
HLS

Subject: SMTP traffic control (was Re: We need an IETF BCP  for GREY LISTING
From: Steve Atkins <[email-masked]>
Date: Thu, 13 Oct 2011 08:00:38 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 13, 2011, at 7:51 AM, John Levine wrote:

> 
>> So, if the server says 'retry in 1 hour', and the client retries every 5 
>> minutes, that's no crisis, just a wasted bit of effort for the client 
>> and the server. ....
> 
> Since the number of MTAs vastly exceeds the number of greylisters,
> don't you think it would be better for greylisters to work with the
> existing retry schedules that MTAs use than to try to persuade a
> million MTAs to implement a new feature that is of at best marginal
> use to them?

We're not talking about greylisting, and haven't been for quite a 
few levels of this thread. We're talking about use of 4xx deferrals
for traffic management and prioritization.

Cheers,
  Steve

Date: 13 Oct 2011 14:51:57 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Organization: 
Sender: [email-masked]

>So, if the server says 'retry in 1 hour', and the client retries every 5 
>minutes, that's no crisis, just a wasted bit of effort for the client 
>and the server. ....

Since the number of MTAs vastly exceeds the number of greylisters,
don't you think it would be better for greylisters to work with the
existing retry schedules that MTAs use than to try to persuade a
million MTAs to implement a new feature that is of at best marginal
use to them?

R's,
John

Date: Thu, 13 Oct 2011 09:30:05 +0100
From: Paul Smith <[email-masked]>
To: Keith Moore <[email-masked]>
CC: [email-masked], John Levine <[email-masked]>, [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 13/10/2011 05:13, Keith Moore wrote:
> My guess is that most MTAs aren't set up to be able to (re)attempt delivery
> within a particular time window.
>> Which is exactly my point. This is a tricky thing to add to an existing
>> queuing strategy.
> yes, I agree.  parsing the dates isn't a problem, but changing a queueing strategy in an existing product could be a huge problem.
>
But no one is saying they would have to. The server would just give a 
'hint' about the delay needed for the next retry. The client can ignore 
that and just use its existing retry mechanism, and there would be no 
change to existing behaviour, or the client could choose to use the hint 
to avoid having unnecessary failed retries in the meantime.

So, if the server says 'retry in 1 hour', and the client retries every 5 
minutes, that's no crisis, just a wasted bit of effort for the client 
and the server. If the client can heed the advice, it just saves a few 
retries which were doomed to failure. If the server says 'retry in 24 
hours' the client is quite at liberty to say 'I don't believe you, I'll 
try sooner than that'.

As I see it, it would be a bit like the "[in-use]" flag in POP3 login 
error responses. If the client ignores it, it's no problem at all, but 
the extra information is there in case the client wants to use it. I see 
it as a minimal impact extension.

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Thu, 13 Oct 2011 00:13:47 -0400
Cc: John Levine <[email-masked]>, [email-masked]
To: [email-masked]
Sender: [email-masked]

On Oct 12, 2011, at 7:37 PM, [email-masked] wrote:

>>> The history of email agents parsing and processing time values isn't
>>> exactly trouble free.
> 
>> While true, that hardly seems like the thing that's most likely to break.
> 
> Past experience says otherwise.
> 
>> My guess is that most MTAs aren't set up to be able to (re)attempt delivery
>> within a particular time window.
> 
> Which is exactly my point. This is a tricky thing to add to an existing
> queuing strategy.

yes, I agree.  parsing the dates isn't a problem, but changing a queueing strategy in an existing product could be a huge problem.

Keith

Date: 13 Oct 2011 03:00:12 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Organization: 
Sender: [email-masked]

>The extension produces a potentially useful optimization, but there
>seems to be a lot less interest in doing that work than there is for
>doing the BCP.  It's not clear this is a major pain point that's worth
>the significant effort of producing a standards track extension and then
>expecting industry to begin adopting it.

The whole point of greylisting is to leverage something that
legitimate MTAs do anyway.  If we're going to add an extenstion to
tell MTAs how to look legit, I'd rather add "sign all your mail with
DKIM" than "try in five minutes."

It would probably make sense in a greylisting BCP to document the
retry times that a greylist is likely to see (occasionally a few
seconds, more often a few minutes to an hour, the old sendmail
interval.)  That doesn't change any specs.

R's,
John

From: [email-masked]
Cc: [email-masked], John Levine <[email-masked]>, [email-masked]
Date: Wed, 12 Oct 2011 16:37:39 -0700 (PDT)
Subject: Re: We need an IETF BCP  for GREY LISTING
To: Keith Moore <[email-masked]>
Sender: [email-masked]

> On Oct 12, 2011, at 1:15 AM, [email-masked] wrote:

> >>> I think the only real value in this extension would be to reward clients
> >>> that recognize it, by providing their users with more predictable
> >>> service (=less delay, more uniform delay).
> >
> >> When I wrote my CEAS paper on greylisting, I found that very little
> >> legit mail is affected by it.  A reasonable greylister only delays the
> >> first message seen from an IP, and after that whitelists it.  (I
> >> realize there are greylisters that delay every new [IP,from] or
> >> [IP,from,to] but the solution is not to do that.)  Very little legit
> >> mail comes from an IP you've never seen before.
> >
> >> Every discussion list I know comes from a single IP or a very small
> >> set of IPs, so the first message or more likely the subscription
> >> confirmation might be delayed, but nothing after that.
> >
> >> I really don't see anything to fix.
> >
> > There may not be anything to fix, but there's most certainly a lot to
> > break. The history of email agents parsing and processing time values isn't
> > exactly trouble free.

> While true, that hardly seems like the thing that's most likely to break.

Past experience says otherwise.

> My guess is that most MTAs aren't set up to be able to (re)attempt delivery
> within a particular time window.

Which is exactly my point. This is a tricky thing to add to an existing
queuing strategy.

> One reason that I suggested a subset of ISO 8601 format (and even a subset of
> that specified in RFC 3339) is that working ISO 8601 parsers are commonplace.

Which is effectively saying that the simplest and easiest part of this is
simple and easy. Which is obviously true but barely relevant.

				Ned

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Wed, 12 Oct 2011 18:01:53 -0400
Cc: John Levine <[email-masked]>, [email-masked]
To: [email-masked]
Sender: [email-masked]

On Oct 12, 2011, at 1:15 AM, [email-masked] wrote:

>>> I think the only real value in this extension would be to reward clients
>>> that recognize it, by providing their users with more predictable
>>> service (=less delay, more uniform delay).
> 
>> When I wrote my CEAS paper on greylisting, I found that very little
>> legit mail is affected by it.  A reasonable greylister only delays the
>> first message seen from an IP, and after that whitelists it.  (I
>> realize there are greylisters that delay every new [IP,from] or
>> [IP,from,to] but the solution is not to do that.)  Very little legit
>> mail comes from an IP you've never seen before.
> 
>> Every discussion list I know comes from a single IP or a very small
>> set of IPs, so the first message or more likely the subscription
>> confirmation might be delayed, but nothing after that.
> 
>> I really don't see anything to fix.
> 
> There may not be anything to fix, but there's most certainly a lot to
> break. The history of email agents parsing and processing time values isn't
> exactly trouble free.

While true, that hardly seems like the thing that's most likely to break.   My guess is that most MTAs aren't set up to be able to (re)attempt delivery within a particular time window.

One reason that I suggested a subset of ISO 8601 format (and even a subset of that specified in RFC 3339) is that working ISO 8601 parsers are commonplace.

If there were interest, it should also be possible to craft an SMTP extension so that the server would supply a token along with the 4xx message, to be used when retrying.  Then, if the client attempted to retransmit the file outside of the assigned window, the server could respond with a different 4xx error message:

4xx retry outside specified window, retry again between YYMMDDTHHMMSSZ-YYYYMMDDTHHMMSSZ

...so that the broken client would get fixed.   (The hardest thing about making mail work right, IMO, is arranging for the errors indications to get to the party most capable of fixing them).

Is any of this extra complexity actually worth the trouble?  I have no idea.

Keith

Date: Wed, 12 Oct 2011 12:23:48 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: Mailing lists and GREY LISTING
Sender: [email-masked]

On 10/12/2011 12:06 PM, Дилян Палаузов wrote:
> I know this mailing list is about SMTP, but when it comes to greylisting
> settings, apart from exposing them over SMTP, it would be nice, if every
> recipient has the possibility to tell his SMTP server the recipient's preferred
> greylisting - settings

doesn't scale.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

X-Mail-Sent-By-AEGEE.org-Account: didopalauzov
Date: Wed, 12 Oct 2011 21:06:37 +0200
From: =?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?= <[email-masked]>
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Mailing lists and GREY LISTING
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------090301090907000909050108
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hello,

On 11.10.2011 19:45, Steve Atkins wrote:

> For classic greylisting that's all quite true (he whole point of greylisting is the assumption that bad actors will not retry, while good actors will). Greylisting does damage some forms of email, though - discussion lists, where emails are delayed by seemingly arbitrary amounts, breaking up the flow of conversation are one - but I can't think of any good way to mitigate that damage other than telling servers not to use greylisting or telling clients to retry quickly, neither of which seems like a good idea.

Discussion list have often the property, that only a limited amount of 
email addresses can post to the list (if the mail is spread over the 
list depends on SMTP env from:/MIME Redirect-...:, From:, Sender:) and 
all other emails are not distributed over the list.  So when an email is 
evaluated for spaminess, apart from the usual criteria, the env 
from:/MIME From:, Sender, Redirect-(From,Sender): have to be correct. 
To my knowledge, spammers writing to mailing list do not have 
information what to put as From: in order to get the mail delivered. 
Spam over mailing lists arrives (most often) due to hacked 
mailbox-passwords and this problem is not avoided by greylisting.  For 
discussion lists I suggest to disable greylisting and instead 
permanently reject the emails (at SMTP level based on the MIME-Headers), 
which would not be distributed over the list.  Rejecting mails (for 
mailing lists) at SMTP level is anyway a godo idea, is it leads to 
decreasing the amount of bounces.

I know this mailing list is about SMTP, but when it comes to greylisting 
settings, apart from exposing them over SMTP, it would be nice, if every 
recipient has the possibility to tell his SMTP server the recipient's 
preferred greylisting - settings (if IP/From or IP/From/To shall be 
proceeded for this recipient, preferred parameters for time retries). 
This could be achieved by Sieve extension, that is evaluated at 
SMTP-Time-- the recipient stores there his greylisting preferences, and 
the server considers them when it proceeds a message for that recipient.

Със здраве
   Дилян

--------------090301090907000909050108
Content-Type: text/x-vcard; charset=utf-8;
 name="dilyan_palauzov.vcf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="dilyan_palauzov.vcf"

YmVnaW46dmNhcmQNCmZuO3F1b3RlZC1wcmludGFibGU6PUQwPTk0PUQwPUI4PUQwPUJCPUQx
PThGPUQwPUJEID1EMD05Rj1EMD1CMD1EMD1CQj1EMD1CMD1EMT04Mz1EMD1CNz1EMD1CRT0N
Cgk9RDA9QjINCm47cXVvdGVkLXByaW50YWJsZTtxdW90ZWQtcHJpbnRhYmxlOj1EMD05Rj1E
MD1CMD1EMD1CQj1EMD1CMD1EMT04Mz1EMD1CNz1EMD1CRT1EMD1CMjs9RDA9OTQ9RDA9Qjg9
RDA9QkI9RDE9OEY9RDA9QkQNCmVtYWlsO2ludGVybmV0OmRpbHlhbi5wYWxhdXpvdkBhZWdl
ZS5vcmcNCnRlbDtob21lOis0OS03MjEtNzU5NDk3NjUNCnRlbDtjZWxsOis0OS0xNjItNDA5
MTE3Mg0Kbm90ZTpzaXA6ODM3MkBhZWdlZS5vcmcNCnZlcnNpb246Mi4xDQplbmQ6dmNhcmQN
Cg0K
--------------090301090907000909050108--

Date: Wed, 12 Oct 2011 11:28:33 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/12/2011 11:15 AM, Steve Atkins wrote:
> I'm interested enough to at least put an individual draft together for people to kick around.

cool!

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Steve Atkins <[email-masked]>
Date: Wed, 12 Oct 2011 11:15:18 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 12, 2011, at 10:49 AM, Dave CROCKER wrote:

> 
> 
> 
> On 10/12/2011 10:36 AM, Murray S. Kucherawy wrote:
>>> Having thought about this a bit, I can't see any problem with a BCP or
>>> even with the server saying 'try again in 5 minutes' (or whatever)
>> 
>> You're blurring the line between two things though.  One of the things we're talking about, the BCP, documents existing, established, or even recommended practice.  The other one is an SMTP extension.  As far as I know, a BCP can't create an extension, so we're really talking about two different efforts here.
>> 
>> The extension produces a potentially useful optimization, but there seems to be a lot less interest in doing that work than there is for doing the BCP. It's not clear this is a major pain point that's worth the significant effort of producing a standards track extension and then expecting industry to begin adopting it.
> 
> 
> +1 on all points.

+1

I think there are people who'd be interested in doing the work for the extension if it were brought to their attention. I'm interested enough to at least put an individual draft together for people to kick around. There are people for whom it's a serious pain point, and one that could be mitigated right now with pretty narrow server-side adoption.

Cheers,
 Steve

Date: Wed, 12 Oct 2011 14:09:29 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

I concur on all points, and would also like to repeat and emphasize something
that Murray said:

	"The extension produces a potentially useful optimization [...]"

I think the word "potential" is important here.  It's not clear to me,
at this point, that optimization is necessary, desirable, or feasible.
I'm open to those possibilities, especially if they can be achieved
via a BCP and not by extending/modifying the protocol, but so far I'm
only partially convinced.  Let me phrase the question this way: can
proponents identify the problem(s) they see, their extent, their impacts,
and their causes?  I'd like to see discussion on that, so that I can
better understand what the underlying issues are.

---rsk

Date: Wed, 12 Oct 2011 10:49:58 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Murray S. Kucherawy" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/12/2011 10:36 AM, Murray S. Kucherawy wrote:
>> Having thought about this a bit, I can't see any problem with a BCP or
>> even with the server saying 'try again in 5 minutes' (or whatever)
>
> You're blurring the line between two things though.  One of the things we're talking about, the BCP, documents existing, established, or even recommended practice.  The other one is an SMTP extension.  As far as I know, a BCP can't create an extension, so we're really talking about two different efforts here.
>
> The extension produces a potentially useful optimization, but there seems to be a lot less interest in doing that work than there is for doing the BCP.  It's not clear this is a major pain point that's worth the significant effort of producing a standards track extension and then expecting industry to begin adopting it.

+1 on all points.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Wed, 12 Oct 2011 10:36:07 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Smith
> Sent: Wednesday, October 12, 2011 6:00 AM
> To: SMTP Interest Group
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> Having thought about this a bit, I can't see any problem with a BCP or
> even with the server saying 'try again in 5 minutes' (or whatever)

You're blurring the line between two things though.  One of the things we're talking about, the BCP, documents existing, established, or even recommended practice.  The other one is an SMTP extension.  As far as I know, a BCP can't create an extension, so we're really talking about two different efforts here.

The extension produces a potentially useful optimization, but there seems to be a lot less interest in doing that work than there is for doing the BCP.  It's not clear this is a major pain point that's worth the significant effort of producing a standards track extension and then expecting industry to begin adopting it.

Date: Wed, 12 Oct 2011 10:25:31 -0400
From: Hector <[email-masked]>
CC: IETF SMTP <[email-masked]>
Subject: Re: We need an IETF BCP for GREY LISTING
Sender: [email-masked]

[email-masked] wrote:
> On Wed, 12 Oct 2011 04:04:08 EDT, Hector said:
> 
>> When you add that many DNS records have very short TTLs, i.e. 5 
>> minutes, if you 2nd attempt more than 5 minutes, then any DNS caching 
>> benefits you might have are lost.  There are many parts to all this.
> 
> People with sub-optimal greylisting configurations often have sub-optimal DNS
> configurations as well, because cluelessness is often pervasive. Film at 11.
> 
> Move along, nothing to see.

Good point, but imagine if a BCP was developed, eventually even the 
layman operators will inherent the result and naturally their software 
will/can support a retry=hint automatically generated the dummies have 
nothing to do with (if only to just enable it if the software chooses 
to have it default off).

In the end, we network benefits or at least has the potential now to 
leverage new information.

> (Though I agree that an Informational RFC documenting the BCP might not be a
> bad idea at all)

+1

--
HLS

Date: Wed, 12 Oct 2011 10:15:13 -0400
From: Hector <[email-masked]>
To: Paul Smith <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

+1.

Yes, for some systems, GL seem to have become part of their existing 
load management (or the other way around). So the retry hint can be 
(optionally) implemented as part the servers overall management.  That 
would be certainly something to consider for our own system which is 
currently separate.  The load management is inherent part of the C/C++ 
SMTP receiver, and the Greylisting is an add-on p-code component hook 
into the DATA state. The scripting language has inherent API access 
the SMTP setup so this would be easy.  The C/C++ receiver can read the 
wcGreyList.INI too, so its all doable here. Thanks for this reminder. :)

Yes, a simple structured BCP response with keyword (retry: or retry=) 
to key in on is all that is needed.

Real Bad guys will never follow anything and that is what GL helped 
protect against. But even if some bad guys adapt from this, so what? 
As long as they had a normal MTA following SMTP retry recommendations, 
they will never have an issue anyway. I don't see any downside to this.

Thanks Paul.

--
HLS

Paul Smith wrote:
> 
> Having thought about this a bit, I can't see any problem with a BCP or 
> even with the server saying 'try again in 5 minutes' (or whatever)
> 
> Yes, you may say that this will help spammers - well, if spammers are 
> able/willing to 'try again in 5 minutes', they will already be doing so, 
> thus getting around greylisting. If they aren't (for whatever reason), 
> then they'll probably also ignore the 'try again in 5 minutes' response.
> 
> They obviously know about greylisting, so apparently many have decided 
> not to bother with it for some reason. Why would telling them the retry 
> time make them change their minds about this?
> 
> However, the 'try again in 5 minutes' information will help legitimate 
> senders/receivers who want to take advantage of the feature, so it has a 
> definite up-side, and a dubious down-side. As previously mentioned, the 
> 'try again in 5 minutes' response doesn't need to be limited to 
> grey-listing but could be used for load management as well.
> 
> It wouldn't even need to be much, just respond
> 
> 421 4.2.x [retry:300s] connection refused for now. Try again later
> 
> it wouldn't interfere with MTAs which don't understand it, and would 
> give a hint to those which do.

Date: 12 Oct 2011 13:29:32 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Organization: 
Sender: [email-masked]

>> Every discussion list I know comes from a single IP or a very small
>> set of IPs, so the first message or more likely the subscription
>> confirmation might be delayed, but nothing after that.
>
>
>mail from lists at google, yahoo and the like has a constant IP address for a 
>given list?

Good point.  It doesn't, but there's enough mail from Google and Yahoo
that their IPs all get whitelisted pretty quickly.

R's,
John

Date: Wed, 12 Oct 2011 14:00:15 +0100
From: Paul Smith <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Having thought about this a bit, I can't see any problem with a BCP or 
even with the server saying 'try again in 5 minutes' (or whatever)

Yes, you may say that this will help spammers - well, if spammers are 
able/willing to 'try again in 5 minutes', they will already be doing so, 
thus getting around greylisting. If they aren't (for whatever reason), 
then they'll probably also ignore the 'try again in 5 minutes' response.

They obviously know about greylisting, so apparently many have decided 
not to bother with it for some reason. Why would telling them the retry 
time make them change their minds about this?

However, the 'try again in 5 minutes' information will help legitimate 
senders/receivers who want to take advantage of the feature, so it has a 
definite up-side, and a dubious down-side. As previously mentioned, the 
'try again in 5 minutes' response doesn't need to be limited to 
grey-listing but could be used for load management as well.

It wouldn't even need to be much, just respond

421 4.2.x [retry:300s] connection refused for now. Try again later

it wouldn't interfere with MTAs which don't understand it, and would 
give a hint to those which do.

To: Hector <[email-masked]>
Cc: IETF SMTP <[email-masked]>
Subject: Re: We need an IETF BCP for GREY LISTING
From: [email-masked]
Date: Wed, 12 Oct 2011 09:04:48 -0400
Sender: [email-masked]

--==_Exmh_1318424688_4531P
Content-Type: text/plain; charset=us-ascii

On Wed, 12 Oct 2011 04:04:08 EDT, Hector said:

> When you add that many DNS records have very short TTLs, i.e. 5 
> minutes, if you 2nd attempt more than 5 minutes, then any DNS caching 
> benefits you might have are lost.  There are many parts to all this.

People with sub-optimal greylisting configurations often have sub-optimal DNS
configurations as well, because cluelessness is often pervasive. Film at 11.

Move along, nothing to see.

(Though I agree that an Informational RFC documenting the BCP might not be a
bad idea at all)

--==_Exmh_1318424688_4531P
Content-Type: application/pgp-signature

--==_Exmh_1318424688_4531P--

Date: Wed, 12 Oct 2011 05:49:47 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: John Levine <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/11/2011 9:20 PM, John Levine wrote:
> Every discussion list I know comes from a single IP or a very small
> set of IPs, so the first message or more likely the subscription
> confirmation might be delayed, but nothing after that.

mail from lists at google, yahoo and the like has a constant IP address for a 
given list?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Wed, 12 Oct 2011 14:35:53 +0200
From: Arnt Gulbrandsen <[email-masked]>
Organization: Me, http://arnt.gulbrandsen.priv.no
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Ned answered John:
>> I really don't see anything to fix.
>
> There may not be anything to fix, but there's most certainly a lot to
> break. The history of email agents parsing and processing time values isn't
> exactly trouble free.

+1

Date: Wed, 12 Oct 2011 07:44:10 -0400
To: SMTP Interest Group <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: RE: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

At 11:16 -0700 on 10/11/2011, Murray S. Kucherawy wrote about Re: We 
need an IETF BCP  for GREY LISTING:

>RFC3339 instead of ISO8601, perhaps?
>
>Of course, abusers will only pay attention to this if it benefits 
>them and it�s cheap to do so.  But they won�t be distinguishable 
>from legitimate clients that just don�t know about this extension 
>and retry by their own schedules, so one can�t penalize such clients 
>for not respecting the request.  On the other hand, you might be 
>able to identify �good� clients (for some value thereof) by 
>observing which ones do respect the request.

The way I understand the suggestion, the MTA offers the sender (in 
its 220 reply) the information that it can send such a reply and the 
sender acknowledges that it has support to respond to the 4xx reply. 
Absence of such an acknowledgement would suppress the sending of a 
4xx "retry at" message (you would get a different type of 4xx).

>On Oct 11, 2011, at 1:45 PM, Steve Atkins wrote:
>
>
>In that case, some minor extension to allow the SMTP server to 
>communicate something a bit more nuanced than "Go away, come back 
>later." might have some value.
>
>I could see value in that. �I could imagine an SMTP extension which, 
>if included, indicated that the server might send a response of the 
>form
>
>4xx please retry between <date-time>-<date-time>
>
>...in response to say the MAIL (or maybe DATA) command,�where 
><date-time> could be an ISO8601 date-time (the horror!) based on GMT 
>(Z) and with no punctuation. �If the client included a SIZE the 
>server could even do bandwidth reservation. �Of course there would 
>be no guarantee that the second attempt wouldn't result in some sort 
>of 4xx response for other reasons.
>
>Greylisting servers could certainly make use of it, though I don't 
>know if it would be a good idea to recommend that greylisting 
>servers use it.
>
>Keith
>

Date: Wed, 12 Oct 2011 04:04:08 -0400
From: Hector <[email-masked]>
To: IETF SMTP <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

John Levine wrote:
>> I think the only real value in this extension would be to reward clients
>> that recognize it, by providing their users with more predictable
>> service (=less delay, more uniform delay).
> 
> When I wrote my CEAS paper on greylisting, I found that very little
> legit mail is affected by it.  A reasonable greylister only delays the
> first message seen from an IP, and after that whitelists it.  (I
> realize there are greylisters that delay every new [IP,from] or
> [IP,from,to] but the solution is not to do that.)  Very little legit
> mail comes from an IP you've never seen before.
> 
> Every discussion list I know comes from a single IP or a very small
> set of IPs, so the first message or more likely the subscription
> confirmation might be delayed, but nothing after that.
> 
> I really don't see anything to fix.

The issue is that its no longer just the "first" but now extended 
further into more attempts where a particular SMTP recommended initial 
short frequency with a backoff is becoming less effective.

This is not just mailing list operations and discussion list, but GL 
servers are now being adopted at the business and corporate level.

The worst case I have seen so far is how a large group of different 
email domains are all served by the same GL hosting MX servers.  So if 
you had, for example, 100 different email destination domains, but 
they are all served by the same hosting service (maybe as part of a 
filtering service),  you now have a very complex queuing issue when 
the rejection is beyond the 1st attempt.

When you add that many DNS records have very short TTLs, i.e. 5 
minutes, if you 2nd attempt more than 5 minutes, then any DNS caching 
benefits you might have are lost.  There are many parts to all this.

Mind you, it you let it all fly and don't pay attention to it, 
eventually the mail will all get sent and unless anyone reported an 
issue, you may not even know it took 1 to 24+ hours with many wasted 
attempts in-between because of an unknown blocking time remote site 
requirement.

The overall issue is the growing wasted attempts (higher overhead) and 
far longer delivery times which in my view, defeats the main purpose 
behind Greylisting.

--
HLS

From: [email-masked]
Cc: [email-masked], [email-masked]
Date: Tue, 11 Oct 2011 22:15:10 -0700 (PDT)
Subject: Re: We need an IETF BCP  for GREY LISTING
To: John Levine <[email-masked]>
Sender: [email-masked]

> >I think the only real value in this extension would be to reward clients
> >that recognize it, by providing their users with more predictable
> >service (=less delay, more uniform delay).

> When I wrote my CEAS paper on greylisting, I found that very little
> legit mail is affected by it.  A reasonable greylister only delays the
> first message seen from an IP, and after that whitelists it.  (I
> realize there are greylisters that delay every new [IP,from] or
> [IP,from,to] but the solution is not to do that.)  Very little legit
> mail comes from an IP you've never seen before.

> Every discussion list I know comes from a single IP or a very small
> set of IPs, so the first message or more likely the subscription
> confirmation might be delayed, but nothing after that.

> I really don't see anything to fix.

There may not be anything to fix, but there's most certainly a lot to
break. The history of email agents parsing and processing time values isn't
exactly trouble free.

				Ned

From: "Tim Kehres" <[email-masked]>
To: "John Levine" <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Date: Wed, 12 Oct 2011 12:54:44 +0800
Sender: [email-masked]

> When I wrote my CEAS paper on greylisting, I found that very little
> legit mail is affected by it.  A reasonable greylister only delays the
> first message seen from an IP, and after that whitelists it.  (I
> realize there are greylisters that delay every new [IP,from] or
> [IP,from,to] but the solution is not to do that.)  Very little legit
> mail comes from an IP you've never seen before.

In the several years that we've been running GL on our servers I've yet to 
see a single legit message not get through.  The false positive rate is as 
close to zero as one can get as long as you are running a sane 
configuration.   And as the checks are done during the SMTP transaction, we 
save a lot by not having to queue on the backend.  HUGE savings on backend 
resources.

The problem is the initial delays in messages (we run [IP,from,to]) - takes 
a bit of education of the users, and the initial responses are not always 
positive.   I think we're running a 10 minute timeout for most systems, and 
have noticed delays in retries of an hour or more in some cases depending on 
the sending side configurations.

Some configurations that have been mentioned (greylisting *everything* and 
setting ridiculously long timeout windows for instance) in my opinion are 
terribly broken configurations.   If a BCP can help people understand how 
this fits into the grand scheme of things then perhaps it is a good thing. 
For those that run with a one day window - I'm surprised how they can keep 
their jobs - it would seem to me that with the progressive backoff 
algorighims used by senders this would result in regular and consistent loss 
of email, not to mention unwarranted message delays.   Protocol changes will 
not protect us from administrators lacking common sense and the ability to 
properly configure their systems.

> Every discussion list I know comes from a single IP or a very small
> set of IPs, so the first message or more likely the subscription
> confirmation might be delayed, but nothing after that.
>
> I really don't see anything to fix.

Completely agree!

Best Regards,

-- Tim 

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 11 Oct 2011 21:45:49 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of John Levine
> Sent: Tuesday, October 11, 2011 9:21 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> I really don't see anything to fix.

Ditto.

As far as I know, SMTP has always been a best-effort retry mechanism.  It's certainly convenient and cheap, but given the Wild West nature of Internet messaging, I'm not so sure I buy the "timely delivery" argument either.  Given the widely deployed and varied abuse countermeasures out there, greylisting being only one of them, highly important stuff should probably find another transport method if arbitrary delay is unacceptable.

Knowing when to retry would be an optimization, but I haven't seen much evidence that its absence is a real pain point worth spinning up an SMTP extension of some kind.

A BCP about greylisting seems to be a fine thing to do, however.

Date: 12 Oct 2011 04:20:43 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Organization: 
Sender: [email-masked]

>I think the only real value in this extension would be to reward clients
>that recognize it, by providing their users with more predictable
>service (=less delay, more uniform delay).

When I wrote my CEAS paper on greylisting, I found that very little
legit mail is affected by it.  A reasonable greylister only delays the
first message seen from an IP, and after that whitelists it.  (I
realize there are greylisters that delay every new [IP,from] or
[IP,from,to] but the solution is not to do that.)  Very little legit
mail comes from an IP you've never seen before.

Every discussion list I know comes from a single IP or a very small
set of IPs, so the first message or more likely the subscription
confirmation might be delayed, but nothing after that.

I really don't see anything to fix.

R's,
John

Date: 12 Oct 2011 04:16:18 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Organization: 
Sender: [email-masked]

>A BCP that merely documents what greylisting is, how to use it, why
>people like it, some of the pitfalls, etc., might not be a bad idea
>though.

I might be able to adapt my CEAS paper.

R's,
John

Date: Tue, 11 Oct 2011 20:31:10 -0400
From: Hector <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

+1

Actually, both sides benefits - The client now can better schedule its 
queued attempts for accepted sessions and not wasting multiple 
attempts on GL servers with *increasingly* unexpected, and IMV, 
unreasonable longer delays and the GL server will see less wasted 
connected sessions they will be blocking anyway, thus increasing their 
availability for better mail acceptability.

If a BCP is widely supported, we should see it contribute to lower DNS 
and SMTP wasted overhead.

In addition, it will MAY reduce the growing consideration to revisit 
queuing strategies to consider the more complex Queue-By-MX/IP and not 
just Queue-By-Email-Domain, which is part of the ordeal we are going 
through with the much higher transaction results seen with GL servers.

Consider, for a bulk mail trade-based newsletter business, many of the 
target email domains with different domains are being served by a 
remote MX service.  So for example, if among the large mailing list 
there are 100 email different domains  but they all hosted by a single 
company, once they begin implementing GL, all those queued by domains 
attempts will fail the first time.

And that all fine and the system has worked, until there is some 
unexpected delay algorithm being used and your wasted attempts:

        2nd attempt 5 mins later
        3rd attempt 15 mins later
        4rd attempt 30 mins later
        5rd attempt 60 mins later

and perhaps as in some cases, much later even as long as a day later, 
are all wasted attempts before it is finally accepted.

To add to this, even if if the client is using a connection sharing 
feature, perhaps limiting four transactions per session, you have 25 
groups that are greylisted.  Less pain, but still a pain.

If there was some structure GL response, such as (suggestion only):

    4xx [extended-code] GREYLIST=V1 [ID=sess-id] RETRY=retry-time

where

    extended-code      optional extended code for RFC2034 compliant 
servers
    GREYLIST=V1 	      BCP anchor for the structure format
    ID=sess-id         optional unique session-id the client can use
    RETRY=retry-time   server suggested time factor to reschedule

where

    sess-id            is opaque with no defined format, but maybe a
                       limited in size.

    retry-time         format can be of two forms:

                       yyyymmdd.hhmmss    GMT time stamp to try again
                       hhmmss             blocking time

But either form is fine since its:

     yyymmdd.hhmmss = current datetime + hhmmss

I just indicated the RETRY=hhmmss format because GL setups already 
have a "blocking time before acceptance" field concept as part of 
their setup.

--
HLS

Keith Moore wrote:
> I think the only real value in this extension would be to reward 
> clients that recognize it, by providing their users with more 
> predictable service (=less delay, more uniform delay).
> 
> Keith
> 
> On Oct 11, 2011, at 2:16 PM, Murray S. Kucherawy wrote:
> 
>> RFC3339 instead of ISO8601, perhaps?
>>  
>> Of course, abusers will only pay attention to this if it benefits them and it�s cheap to do so.  But they won�t be distinguishable from legitimate clients that just don�t know about this extension and retry by their own schedules, so one can�t penalize such clients for not respecting the request.  On the other hand, you might be able to identify �good� clients (for some value thereof) by observing which ones do respect the request.
>>  
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Keith Moore
>> Sent: Tuesday, October 11, 2011 11:06 AM
>> To: Steve Atkins
>> Cc: SMTP Interest Group
>> Subject: Re: We need an IETF BCP for GREY LISTING
>>  
>> On Oct 11, 2011, at 1:45 PM, Steve Atkins wrote:
>>
>>
>> In that case, some minor extension to allow the SMTP server to communicate something a bit more nuanced than "Go away, come back later." might have some value.
>>  
>> I could see value in that.  I could imagine an SMTP extension which, if included, indicated that the server might send a response of the form
>>  
>> 4xx please retry between <date-time>-<date-time>
>>  
>> ...in response to say the MAIL (or maybe DATA) command, where <date-time> could be an ISO8601 date-time (the horror!) based on GMT (Z) and with no punctuation.  If the client included a SIZE the server could even do bandwidth reservation.  Of course there would be no guarantee that the second attempt wouldn't result in some sort of 4xx response for other reasons.
>>  
>> Greylisting servers could certainly make use of it, though I don't know if it would be a good idea to recommend that greylisting servers use it.
>>  
>> Keith
>>  
> 
> 

Date: Tue, 11 Oct 2011 12:46:50 -0700
From: Douglas Otis <[email-masked]>
To: Steve Atkins <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On 10/11/11 11:35 AM, Steve Atkins wrote:
> On Oct 11, 2011, at 11:16 AM, Murray S. Kucherawy wrote:
>> RFC3339 instead of ISO8601, perhaps?
>>
>> Of course, abusers will only pay attention to this if it benefits them and it�s cheap to do so.
> Yup. But it's not the abusers that really matter here, it's the good actors who are happy to play nice with the receivers, but don't have the information to do so.
Rather than playing games that further burden receivers with added 
state, spammers already make themselves appear legit by performing the 
retries without actually doing it in a stateful manner, as campaigns 
behind the retry may change.  The advantage therefore goes to the spammer.

Just as Apple provides public kerberos services for all their millions 
of users, perhaps email could benefit from a similar approach.  
Something similar to that outlined in
http://tools.ietf.org/html/rfc6281

Recipients could then base acceptance on an authentication process that 
takes place once every 10 hours, for example.   Trust could then be 
placed in services that exclude spammers.  The approach would enable use 
of either IPv4 or IPv6 without reliance upon any single service provider 
or auxiliary reputation service.  This would require a number of 
conventions, but it should be pointed out their scheme uses existing 
protocols.

-Doug

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Steve Atkins <[email-masked]>
Date: Tue, 11 Oct 2011 11:35:51 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 11, 2011, at 11:16 AM, Murray S. Kucherawy wrote:

> RFC3339 instead of ISO8601, perhaps?
>  
> Of course, abusers will only pay attention to this if it benefits them and it�s cheap to do so.  

Yup. But it's not the abusers that really matter here, it's the good actors who are happy to play nice with the receivers, but don't have the information to do so.

> But they won�t be distinguishable from legitimate clients that just don�t know about this extension and retry by their own schedules, so one can�t penalize such clients for not respecting the request.  

You can reward senders who do respect the request

If I don't want any more connections from a particular sender for the next five minutes, I'll 4xx any connections from them for that period, then start letting mail through. A sender that pays attention to my "� in five minutes" message to them will simply go away, then come back in five minutes and be accepted.

A sender that didn't pay attention might retry after 15 minutes and get delivered, or they might retry after one minute (and get deferred), after two minutes (and get deferred), then four minutes (and get deferred), then eight minutes and be delivered. They'll get worse latency than the cooperative sender, even if the receiver pays no particular attention to their behaviour (and if I were an ISP I'd probably reward those senders by giving them priority over random bad senders when resources are constrained).

> On the other hand, you might be able to identify �good� clients (for some value thereof) by observing which ones do respect the request.

A lot of legitimate email is sent by legitimate bulk emailers. Mostly ESPs, but also a lot of corporations sending mail to their customers. They expend quite a lot of effort in traffic management already, so as to get their mail delivered in a timely fashion while not getting throttled by the recipient ISP. If they could check a box in their smarthost configuration that says "Do what the ISP asks", they'd do it - as it would reduce their overheads considerably, and likely lead to their email being delivered faster or within more predictable bounds.

I think that the issue of real-time traffic control feedback from server to client might be a bit broader than just this one example, but it's a simple enough case to start thinking about.

Cheers,
  Steve

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 11 Oct 2011 11:27:31 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Richard Kulawiec
> Sent: Tuesday, October 11, 2011 11:20 AM
> To: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> On the other hand, it's not clear to me what benefit we could achieve
> by doing so.  That is, I'm not sure I see many instances of greylisting
> implementations behaving badly -- thus, potentially benefitting from
> a document that would give some guidance.  I think that most of the ones
> I've observed are doing a reasonable (if, perhaps, sometimes annoying
> or inconvenient) job of deferring mail traffic for a sensible length
> of time.  Are others' logs reflecting a different picture?

No evidence here of a new or increasing problem, and I'm not convinced it's something that requires a change or extension to the protocol in any case.

A BCP that merely documents what greylisting is, how to use it, why people like it, some of the pitfalls, etc., might not be a bad idea though.

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Tue, 11 Oct 2011 14:24:20 -0400
Cc: SMTP Interest Group <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
Sender: [email-masked]

--Apple-Mail-15-747844312
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

I think the only real value in this extension would be to reward clients =
that recognize it, by providing their users with more predictable =
service (=3Dless delay, more uniform delay).

Keith

On Oct 11, 2011, at 2:16 PM, Murray S. Kucherawy wrote:

> RFC3339 instead of ISO8601, perhaps?
> =20
> Of course, abusers will only pay attention to this if it benefits them =
and it=92s cheap to do so.  But they won=92t be distinguishable from =
legitimate clients that just don=92t know about this extension and retry =
by their own schedules, so one can=92t penalize such clients for not =
respecting the request.  On the other hand, you might be able to =
identify =93good=94 clients (for some value thereof) by observing which =
ones do respect the request.
> =20
> From: [email-masked] =
[mailto:[email-masked]] On Behalf Of Keith Moore
> Sent: Tuesday, October 11, 2011 11:06 AM
> To: Steve Atkins
> Cc: SMTP Interest Group
> Subject: Re: We need an IETF BCP for GREY LISTING
> =20
> On Oct 11, 2011, at 1:45 PM, Steve Atkins wrote:
>=20
>=20
> In that case, some minor extension to allow the SMTP server to =
communicate something a bit more nuanced than "Go away, come back =
later." might have some value.
> =20
> I could see value in that.  I could imagine an SMTP extension which, =
if included, indicated that the server might send a response of the form
> =20
> 4xx please retry between <date-time>-<date-time>
> =20
> ...in response to say the MAIL (or maybe DATA) command, where =
<date-time> could be an ISO8601 date-time (the horror!) based on GMT (Z) =
and with no punctuation.  If the client included a SIZE the server could =
even do bandwidth reservation.  Of course there would be no guarantee =
that the second attempt wouldn't result in some sort of 4xx response for =
other reasons.
> =20
> Greylisting servers could certainly make use of it, though I don't =
know if it would be a good idea to recommend that greylisting servers =
use it.
> =20
> Keith
> =20

--Apple-Mail-15-747844312
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=windows-1252

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I =
think the only real value in this extension would be to reward clients =
that recognize it, by providing their users with more predictable =
service (=3Dless delay, more uniform =
delay).<div><br></div><div>Keith<br><div><br><div><div>On Oct 11, 2011, =
at 2:16 PM, Murray S. Kucherawy wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
font-family: Helvetica; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div =
lang=3D"EN-US" link=3D"blue" vlink=3D"purple" style=3D"word-wrap: =
break-word; -webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space; "><div class=3D"WordSection1" style=3D"page: =
WordSection1; "><div style=3D"margin-top: 0in; margin-right: 0in; =
margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; font-family: =
'Times New Roman', serif; "><span style=3D"font-size: 11pt; font-family: =
Calibri, sans-serif; color: rgb(31, 73, 125); ">RFC3339 instead of =
ISO8601, perhaps?<o:p></o:p></span></div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: =
11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); =
"><o:p>&nbsp;</o:p></span></div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; "><span style=3D"font-size: =
11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Of =
course, abusers will only pay attention to this if it benefits them and =
it=92s cheap to do so.&nbsp; But they won=92t be distinguishable from =
legitimate clients that just don=92t know about this extension and retry =
by their own schedules, so one can=92t penalize such clients for not =
respecting the request.&nbsp; On the other hand, you might be able to =
identify =93good=94 clients (for some value thereof) by observing which =
ones do respect the request.<o:p></o:p></span></div><div =
style=3D"margin-top: 0in; margin-right: 0in; margin-left: 0in; =
margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New =
Roman', serif; "><span style=3D"font-size: 11pt; font-family: Calibri, =
sans-serif; color: rgb(31, 73, 125); =
"><o:p>&nbsp;</o:p></span></div><div style=3D"border-top-style: none; =
border-right-style: none; border-bottom-style: none; border-width: =
initial; border-color: initial; border-left-style: solid; =
border-left-color: blue; border-left-width: 1.5pt; padding-top: 0in; =
padding-right: 0in; padding-bottom: 0in; padding-left: 4pt; "><div><div =
style=3D"border-right-style: none; border-bottom-style: none; =
border-left-style: none; border-width: initial; border-color: initial; =
border-top-style: solid; border-top-color: rgb(181, 196, 223); =
border-top-width: 1pt; padding-top: 3pt; padding-right: 0in; =
padding-bottom: 0in; padding-left: 0in; "><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; "><b><span =
style=3D"font-size: 10pt; font-family: Tahoma, sans-serif; =
">From:</span></b><span style=3D"font-size: 10pt; font-family: Tahoma, =
sans-serif; "><span class=3D"Apple-converted-space">&nbsp;</span><a =
href=3D"mailto:[email-masked]" style=3D"color: blue; =
text-decoration: underline; ">[email-masked]</a><span =
class=3D"Apple-converted-space">&nbsp;</span>[mailto:owner-ietf-smtp@mail.=
imc.org]<span class=3D"Apple-converted-space">&nbsp;</span><b>On Behalf =
Of<span class=3D"Apple-converted-space">&nbsp;</span></b>Keith =
Moore<br><b>Sent:</b><span =
class=3D"Apple-converted-space">&nbsp;</span>Tuesday, October 11, 2011 =
11:06 AM<br><b>To:</b><span =
class=3D"Apple-converted-space">&nbsp;</span>Steve =
Atkins<br><b>Cc:</b><span =
class=3D"Apple-converted-space">&nbsp;</span>SMTP Interest =
Group<br><b>Subject:</b><span =
class=3D"Apple-converted-space">&nbsp;</span>Re: We need an IETF BCP for =
GREY LISTING<o:p></o:p></span></div></div></div><div style=3D"margin-top: =
0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
font-size: 12pt; font-family: 'Times New Roman', serif; =
"><o:p>&nbsp;</o:p></div><div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; ">On Oct 11, 2011, at 1:45 =
PM, Steve Atkins wrote:<o:p></o:p></div></div><div style=3D"margin-top: =
0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
font-size: 12pt; font-family: 'Times New Roman', serif; =
"><br><br><o:p></o:p></div><div style=3D"margin-top: 0in; margin-right: =
0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 12pt; =
font-family: 'Times New Roman', serif; "><span =
class=3D"apple-style-span"><span style=3D"font-size: 13.5pt; =
font-family: Helvetica, sans-serif; ">In that case, some minor extension =
to allow the SMTP server to communicate something a bit more nuanced =
than "Go away, come back later." might have some =
value.</span></span><o:p></o:p></div></div><div style=3D"margin-top: =
0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; =
font-size: 12pt; font-family: 'Times New Roman', serif; =
"><o:p>&nbsp;</o:p></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; ">I could see value in =
that. &nbsp;I could imagine an SMTP extension which, if included, =
indicated that the server might send a response of the =
form<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; =
"><o:p>&nbsp;</o:p></div></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; ">4xx please retry between =
&lt;date-time&gt;-&lt;date-time&gt;<o:p></o:p></div></div><div><div =
style=3D"margin-top: 0in; margin-right: 0in; margin-left: 0in; =
margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New =
Roman', serif; "><o:p>&nbsp;</o:p></div></div><div><div =
style=3D"margin-top: 0in; margin-right: 0in; margin-left: 0in; =
margin-bottom: 0.0001pt; font-size: 12pt; font-family: 'Times New =
Roman', serif; ">...in response to say the MAIL (or maybe DATA) =
command,&nbsp;where &lt;date-time&gt; could be an ISO8601 date-time (the =
horror!) based on GMT (Z) and with no punctuation. &nbsp;If the client =
included a SIZE the server could even do bandwidth reservation. &nbsp;Of =
course there would be no guarantee that the second attempt wouldn't =
result in some sort of 4xx response for other =
reasons.<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; =
"><o:p>&nbsp;</o:p></div></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; ">Greylisting servers could =
certainly make use of it, though I don't know if it would be a good idea =
to recommend that greylisting servers use =
it.<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; =
"><o:p>&nbsp;</o:p></div></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; =
">Keith<o:p></o:p></div></div><div><div style=3D"margin-top: 0in; =
margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: =
12pt; font-family: 'Times New Roman', serif; =
"><o:p>&nbsp;</o:p></div></div></div></div></div></span></blockquote></div=
><br></div></div></body></html>=

--Apple-Mail-15-747844312--

Date: Tue, 11 Oct 2011 14:19:59 -0400
From: Richard Kulawiec <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

On Tue, Oct 11, 2011 at 08:06:16AM -0400, Keith Moore wrote:
> I'm not sure that it's worth the trouble.   The whole point of
> greylisting is to marginalize naive client implementations on the
> assumption (largely valid to this point) that they're likely to
> be spambots.   But I expect that spambots will start to deal with
> greylisting very soon.  If IETF were to document greylisting it would
> only accelerate that process.

I don't think documenting greylisting will accelerate the
(already-underway) process of spammer adaptation.  I think the prime
mover there is the adoption rate of greylisting.  This isn't unique
to greylisting -- we've seen spammers adapt to other countermeasures
when they've (apparently) judged that those measures are sufficiently
widely-used to merit their attention and effort.

On the other hand, it's not clear to me what benefit we could achieve
by doing so.  That is, I'm not sure I see many instances of greylisting
implementations behaving badly -- thus, potentially benefitting from
a document that would give some guidance.  I think that most of the ones
I've observed are doing a reasonable (if, perhaps, sometimes annoying
or inconvenient) job of deferring mail traffic for a sensible length
of time.  Are others' logs reflecting a different picture?

---rsk

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Tue, 11 Oct 2011 11:16:03 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

--_000_F5833273385BB34F99288B3648C4F06F19C45D9F61EXCHC2corpclo_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

RFC3339 instead of ISO8601, perhaps?

Of course, abusers will only pay attention to this if it benefits them and =
it's cheap to do so.  But they won't be distinguishable from legitimate cli=
ents that just don't know about this extension and retry by their own sched=
ules, so one can't penalize such clients for not respecting the request.  O=
n the other hand, you might be able to identify "good" clients (for some va=
lue thereof) by observing which ones do respect the request.

From: [email-masked] [mailto:[email-masked]] On=
 Behalf Of Keith Moore
Sent: Tuesday, October 11, 2011 11:06 AM
To: Steve Atkins
Cc: SMTP Interest Group
Subject: Re: We need an IETF BCP for GREY LISTING

On Oct 11, 2011, at 1:45 PM, Steve Atkins wrote:

In that case, some minor extension to allow the SMTP server to communicate =
something a bit more nuanced than "Go away, come back later." might have so=
me value.

I could see value in that.  I could imagine an SMTP extension which, if inc=
luded, indicated that the server might send a response of the form

4xx please retry between <date-time>-<date-time>

...in response to say the MAIL (or maybe DATA) command, where <date-time> c=
ould be an ISO8601 date-time (the horror!) based on GMT (Z) and with no pun=
ctuation.  If the client included a SIZE the server could even do bandwidth=
 reservation.  Of course there would be no guarantee that the second attemp=
t wouldn't result in some sort of 4xx response for other reasons.

Greylisting servers could certainly make use of it, though I don't know if =
it would be a good idea to recommend that greylisting servers use it.

Keith

--_000_F5833273385BB34F99288B3648C4F06F19C45D9F61EXCHC2corpclo_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:p=3D"urn:schemas-m=
icrosoft-com:office:powerpoint" xmlns:a=3D"urn:schemas-microsoft-com:office=
:access" xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s=3D"=
uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs=3D"urn:schemas-microsof=
t-com:rowset" xmlns:z=3D"#RowsetSchema" xmlns:b=3D"urn:schemas-microsoft-co=
m:office:publisher" xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadshee=
t" xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns=
:odc=3D"urn:schemas-microsoft-com:office:odc" xmlns:oa=3D"urn:schemas-micro=
soft-com:office:activation" xmlns:html=3D"http://www.w3.org/TR/REC-html40" =
xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc=3D"http://m=
icrosoft.com/officenet/conferencing" xmlns:D=3D"DAV:" xmlns:Repl=3D"http://=
schemas.microsoft.com/repl/" xmlns:mt=3D"http://schemas.microsoft.com/share=
point/soap/meetings/" xmlns:x2=3D"http://schemas.microsoft.com/office/excel=
/2003/xml" xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd" xmlns:ois=
=3D"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir=3D"http://=
schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds=3D"http://www.w3=
.org/2000/09/xmldsig#" xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint=
/dsp" xmlns:udc=3D"http://schemas.microsoft.com/data/udc" xmlns:xsd=3D"http=
://www.w3.org/2001/XMLSchema" xmlns:sub=3D"http://schemas.microsoft.com/sha=
repoint/soap/2002/1/alerts/" xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"=
 xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" xmlns:sps=3D"http://=
schemas.microsoft.com/sharepoint/soap/" xmlns:xsi=3D"http://www.w3.org/2001=
/XMLSchema-instance" xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/so=
ap" xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udc=
p2p=3D"http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf=3D"http:/=
/schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss=3D"http://sche=
mas.microsoft.com/office/2006/digsig-setup" xmlns:dssi=3D"http://schemas.mi=
crosoft.com/office/2006/digsig" xmlns:mdssi=3D"http://schemas.openxmlformat=
s.org/package/2006/digital-signature" xmlns:mver=3D"http://schemas.openxmlf=
ormats.org/markup-compatibility/2006" xmlns:m=3D"http://schemas.microsoft.c=
om/office/2004/12/omml" xmlns:mrels=3D"http://schemas.openxmlformats.org/pa=
ckage/2006/relationships" xmlns:spwp=3D"http://microsoft.com/sharepoint/web=
partpages" xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/20=
06/types" xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/200=
6/messages" xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/Sli=
deLibrary/" xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortal=
Server/PublishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" xmlns:=
st=3D"&#1;" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUI=
V=3D"Content-Type" CONTENT=3D"text/html; charset=3Dus-ascii"><meta name=3DG=
enerator content=3D"Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:Helvetica;
	panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
	{font-family:Helvetica;
	panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.apple-style-span
	{mso-style-name:apple-style-span;}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue vli=
nk=3Dpurple style=3D'word-wrap: break-word;-webkit-nbsp-mode: space;-webkit=
-line-break: after-white-space'><div class=3DWordSection1><p class=3DMsoNor=
mal><span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";colo=
r:#1F497D'>RFC3339 instead of ISO8601, perhaps?<o:p></o:p></span></p><p cla=
ss=3DMsoNormal><span style=3D'font-size:11.0pt;font-family:"Calibri","sans-=
serif";color:#1F497D'><o:p>&nbsp;</o:p></span></p><p class=3DMsoNormal><spa=
n style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497=
D'>Of course, abusers will only pay attention to this if it benefits them a=
nd it&#8217;s cheap to do so.&nbsp; But they won&#8217;t be distinguishable=
 from legitimate clients that just don&#8217;t know about this extension an=
d retry by their own schedules, so one can&#8217;t penalize such clients fo=
r not respecting the request.&nbsp; On the other hand, you might be able to=
 identify &#8220;good&#8221; clients (for some value thereof) by observing =
which ones do respect the request.<o:p></o:p></span></p><p class=3DMsoNorma=
l><span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:=
#1F497D'><o:p>&nbsp;</o:p></span></p><div style=3D'border:none;border-left:=
solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style=3D'border:none;=
border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=3DMsoNor=
mal><b><span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>F=
rom:</span></b><span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-s=
erif"'> [email-masked] [mailto:[email-masked]] =
<b>On Behalf Of </b>Keith Moore<br><b>Sent:</b> Tuesday, October 11, 2011 1=
1:06 AM<br><b>To:</b> Steve Atkins<br><b>Cc:</b> SMTP Interest Group<br><b>=
Subject:</b> Re: We need an IETF BCP for GREY LISTING<o:p></o:p></span></p>=
</div></div><p class=3DMsoNormal><o:p>&nbsp;</o:p></p><div><div><p class=3D=
MsoNormal>On Oct 11, 2011, at 1:45 PM, Steve Atkins wrote:<o:p></o:p></p></=
div><p class=3DMsoNormal><br><br><o:p></o:p></p><p class=3DMsoNormal><span =
class=3Dapple-style-span><span style=3D'font-size:13.5pt;font-family:"Helve=
tica","sans-serif"'>In that case, some minor extension to allow the SMTP se=
rver to communicate something a bit more nuanced than &quot;Go away, come b=
ack later.&quot; might have some value.</span></span><o:p></o:p></p></div><=
p class=3DMsoNormal><o:p>&nbsp;</o:p></p><div><p class=3DMsoNormal>I could =
see value in that. &nbsp;I could imagine an SMTP extension which, if includ=
ed, indicated that the server might send a response of the form<o:p></o:p><=
/p></div><div><p class=3DMsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=
=3DMsoNormal>4xx please retry between &lt;date-time&gt;-&lt;date-time&gt;<o=
:p></o:p></p></div><div><p class=3DMsoNormal><o:p>&nbsp;</o:p></p></div><di=
v><p class=3DMsoNormal>...in response to say the MAIL (or maybe DATA) comma=
nd,&nbsp;where &lt;date-time&gt; could be an ISO8601 date-time (the horror!=
) based on GMT (Z) and with no punctuation. &nbsp;If the client included a =
SIZE the server could even do bandwidth reservation. &nbsp;Of course there =
would be no guarantee that the second attempt wouldn't result in some sort =
of 4xx response for other reasons.<o:p></o:p></p></div><div><p class=3DMsoN=
ormal><o:p>&nbsp;</o:p></p></div><div><p class=3DMsoNormal>Greylisting serv=
ers could certainly make use of it, though I don't know if it would be a go=
od idea to recommend that greylisting servers use it.<o:p></o:p></p></div><=
div><p class=3DMsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=3DMsoNorm=
al>Keith<o:p></o:p></p></div><div><p class=3DMsoNormal><o:p>&nbsp;</o:p></p=
></div></div></div></body></html>=

--_000_F5833273385BB34F99288B3648C4F06F19C45D9F61EXCHC2corpclo_--

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Tue, 11 Oct 2011 14:05:30 -0400
Cc: SMTP Interest Group <[email-masked]>
To: Steve Atkins <[email-masked]>
Sender: [email-masked]

--Apple-Mail-8-746714071
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Oct 11, 2011, at 1:45 PM, Steve Atkins wrote:

> In that case, some minor extension to allow the SMTP server to =
communicate something a bit more nuanced than "Go away, come back =
later." might have some value.

I could see value in that.  I could imagine an SMTP extension which, if =
included, indicated that the server might send a response of the form

4xx please retry between <date-time>-<date-time>

...in response to say the MAIL (or maybe DATA) command, where =
<date-time> could be an ISO8601 date-time (the horror!) based on GMT (Z) =
and with no punctuation.  If the client included a SIZE the server could =
even do bandwidth reservation.  Of course there would be no guarantee =
that the second attempt wouldn't result in some sort of 4xx response for =
other reasons.

Greylisting servers could certainly make use of it, though I don't know =
if it would be a good idea to recommend that greylisting servers use it.

Keith

--Apple-Mail-8-746714071
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><div>On Oct 11, 2011, at 1:45 PM, Steve Atkins wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
font-family: Helvetica; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; ">In that case, =
some minor extension to allow the SMTP server to communicate something a =
bit more nuanced than "Go away, come back later." might have some =
value.</span></blockquote></div><br><div>I could see value in that. =
&nbsp;I could imagine an SMTP extension which, if included, indicated =
that the server might send a response of the =
form</div><div><br></div><div>4xx please retry between =
&lt;date-time&gt;-&lt;date-time&gt;</div><div><br></div><div>...in =
response to say the MAIL (or maybe DATA) command,&nbsp;where =
&lt;date-time&gt; could be an ISO8601 date-time (the horror!) based on =
GMT (Z) and with no punctuation. &nbsp;If the client included a SIZE the =
server could even do bandwidth reservation. &nbsp;Of course there would =
be no guarantee that the second attempt wouldn't result in some sort of =
4xx response for other reasons.</div><div><br></div><div>Greylisting =
servers could certainly make use of it, though I don't know if it would =
be a good idea to recommend that greylisting servers use =
it.</div><div><br></div><div>Keith</div><div><br></div></body></html>=

--Apple-Mail-8-746714071--

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Tue, 11 Oct 2011 11:03:29 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Steve Atkins
> Sent: Tuesday, October 11, 2011 10:46 AM
> To: SMTP Interest Group
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> For classic greylisting that's all quite true (he whole point of
> greylisting is the assumption that bad actors will not retry, while
> good actors will). Greylisting does damage some forms of email, though
> - discussion lists, where emails are delayed by seemingly arbitrary
> amounts, breaking up the flow of conversation are one - but I can't
> think of any good way to mitigate that damage other than telling
> servers not to use greylisting or telling clients to retry quickly,
> neither of which seems like a good idea.

That's true, though the disruption should be minor and not persistent as the delayed sender is only delayed the first time he/she gets greylisted.  In my experience so far this hasn't been a major problem.

> There's a vaguely related situation, though, where the current protocol
> isn't ideal, and that's when an SMTP server wants to put some control
> on the volume of mail it's being sent (by a specific SMTP client). The
> scarce resource is number of concurrent sessions, not total bandwidth,
> so the appropriate level at which to throttle is the SMTP session
> level. Yahoo, as one example, uses 4xx deferrals in order to shed load
> or to reserve capacity for high value mail at the expense of low value
> mail. That's not an ideal approach, as the SMTP clients have no
> consistent retry policy, so it's a fairly crude tool for capacity
> management. In that case, some minor extension to allow the SMTP server
> to communicate something a bit more nuanced than "Go away, come back
> later." might have some value.

I think "4xx" does exactly that.  It sounds like what you're after is "Go away, come back at time X" but only tell the good guys that.  Perhaps this would be a combination of 4xx load shedding coupled with greylisting merely as a way of giving the hint to the good guys.  This is where something actually specified might be handy.

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Steve Atkins <[email-masked]>
Date: Tue, 11 Oct 2011 10:45:38 -0700
To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

On Oct 11, 2011, at 10:32 AM, Murray S. Kucherawy wrote:

>> 
>> 
>>> I think someone should take up the effort to begin/draft an GreyList
>>> BCP for systems to follow.
>>> 
>>> Comments?
>> 
>> I'm not sure that it's worth the trouble.   The whole point of
>> greylisting is to marginalize naive client implementations on the
>> assumption (largely valid to this point) that they're likely to be
>> spambots.   But I expect that spambots will start to deal with
>> greylisting very soon.  If IETF were to document greylisting it would
>> only accelerate that process.
> 
> I agree.  I don't see what additional interoperability needs to be specified here that isn't already handled by RFC5321.  4xx means retry, and that's it.  If, as a server, I give a hint as to when a retry will succeed, I'm giving the bad guys information that I don't want them to have, while a compliant RFC5321 implementation will do just fine as it is.

For classic greylisting that's all quite true (he whole point of greylisting is the assumption that bad actors will not retry, while good actors will). Greylisting does damage some forms of email, though - discussion lists, where emails are delayed by seemingly arbitrary amounts, breaking up the flow of conversation are one - but I can't think of any good way to mitigate that damage other than telling servers not to use greylisting or telling clients to retry quickly, neither of which seems like a good idea.

There's a vaguely related situation, though, where the current protocol isn't ideal, and that's when an SMTP server wants to put some control on the volume of mail it's being sent (by a specific SMTP client). The scarce resource is number of concurrent sessions, not total bandwidth, so the appropriate level at which to throttle is the SMTP session level. Yahoo, as one example, uses 4xx deferrals in order to shed load or to reserve capacity for high value mail at the expense of low value mail. That's not an ideal approach, as the SMTP clients have no consistent retry policy, so it's a fairly crude tool for capacity management. In that case, some minor extension to allow the SMTP server to communicate something a bit more nuanced than "Go away, come back later." might have some value.

Cheers,
  Steve

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 11 Oct 2011 10:36:53 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Keith Moore
> Sent: Tuesday, October 11, 2011 9:43 AM
> To: Stephane Bortzmeyer
> Cc: Hector; [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> No, it doesn't.   But history shows that spammers have adapted to a
> wide variety of spam countermeasures.   What is it about greylisting
> that makes it unlikely that they'll adapt to it?

It requires increased long-term storage and a retry scheduler.  Not hard to implement, to be sure, but it becomes a little tougher to remain undetected and leave no footprint on an infected system.

From: "Murray S. Kucherawy" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 11 Oct 2011 10:32:40 -0700
Subject: RE: We need an IETF BCP  for GREY LISTING
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Keith Moore
> Sent: Tuesday, October 11, 2011 5:06 AM
> To: Hector
> Cc: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> > I think someone should take up the effort to begin/draft an GreyList
> > BCP for systems to follow.
> >
> > Comments?
> 
> I'm not sure that it's worth the trouble.   The whole point of
> greylisting is to marginalize naive client implementations on the
> assumption (largely valid to this point) that they're likely to be
> spambots.   But I expect that spambots will start to deal with
> greylisting very soon.  If IETF were to document greylisting it would
> only accelerate that process.

I agree.  I don't see what additional interoperability needs to be specified here that isn't already handled by RFC5321.  4xx means retry, and that's it.  If, as a server, I give a hint as to when a retry will succeed, I'm giving the bad guys information that I don't want them to have, while a compliant RFC5321 implementation will do just fine as it is.

Date: Tue, 11 Oct 2011 09:56:45 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: Stephane Bortzmeyer <[email-masked]>, Hector <[email-masked]>, [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------040104080405070001050201
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Keith Moore wrote:
>> I here that since I use greylisting (and I started many years ago,
>> with people saying "it will work only for a few months"). Repeating
>> this mantra does not make it true.
>>     
>
> No, it doesn't.   But history shows that spammers have adapted to a wide variety of spam countermeasures.   What is it about greylisting that makes it unlikely that they'll adapt to it?

Anecdotally -- some already have. There was one 'bot I used to see 
regularly that would probe my SMTP servers for gray listing, source IP 
Blocks, and tarpitting. I think it has also measured my rate limit for 
detecting DHA. It noticed if a particular IP address was getting 
blocked, and would switch to another; as soon as the block on an IP 
address expired, it would be back trying from that IP address again. It 
seemed to be almost exclusively targeting financial institutions; if it 
hadn't been for that I might not have even noticed it.

I've often wondered if the sophistication of the 'bot reflected the 
strategic significance of the target. The 'bots that tried to get 
through to my ISP customers were dumb as a brick.

<csg>

--------------040104080405070001050201
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Keith Moore wrote:<br>
<blockquote
 cite="mid:[email-masked]"
 type="cite">
  <blockquote type="cite">
    <pre wrap="">I here that since I use greylisting (and I started many years ago,
with people saying "it will work only for a few months"). Repeating
this mantra does not make it true.
    </pre>
  </blockquote>
  <pre wrap=""><!---->
No, it doesn't.   But history shows that spammers have adapted to a wide variety of spam countermeasures.   What is it about greylisting that makes it unlikely that they'll adapt to it?</pre>
</blockquote>
<br>
Anecdotally -- some already have. There was one 'bot I used to see
regularly that would probe my SMTP servers for gray listing, source IP
Blocks, and tarpitting. I think it has also measured my rate limit for
detecting DHA. It noticed if a particular IP address was getting
blocked, and would switch to another; as soon as the block on an IP
address expired, it would be back trying from that IP address again. It
seemed to be almost exclusively targeting financial institutions; if it
hadn't been for that I might not have even noticed it.<br>
<br>
I've often wondered if the sophistication of the 'bot reflected the
strategic significance of the target. The 'bots that tried to get
through to my ISP customers were dumb as a brick.<br>
<br>
&lt;csg&gt;<br>
</body>
</html>

--------------040104080405070001050201--

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Tue, 11 Oct 2011 12:42:58 -0400
Cc: Hector <[email-masked]>, [email-masked]
To: Stephane Bortzmeyer <[email-masked]>
Sender: [email-masked]

On Oct 11, 2011, at 11:40 AM, Stephane Bortzmeyer wrote:

> I here that since I use greylisting (and I started many years ago,
> with people saying "it will work only for a few months"). Repeating
> this mantra does not make it true.

No, it doesn't.   But history shows that spammers have adapted to a wide variety of spam countermeasures.   What is it about greylisting that makes it unlikely that they'll adapt to it?

Keith

Date: Tue, 11 Oct 2011 17:40:02 +0200
From: Stephane Bortzmeyer <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: Hector <[email-masked]>, [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Organization: NIC France
Sender: [email-masked]

On Tue, Oct 11, 2011 at 08:06:16AM -0400,
 Keith Moore <[email-masked]> wrote 
 a message of 12 lines which said:

> But I expect that spambots will start to deal with greylisting very
> soon.  

I here that since I use greylisting (and I started many years ago,
with people saying "it will work only for a few months"). Repeating
this mantra does not make it true.

On Tue, Oct 11, 2011 at 01:59:28PM -0000,
 John Levine <[email-masked]> wrote 
 a message of 12 lines which said:

> My logs don't show any recent increase in greylisting.

Same thing for me. This is probably the reason why spammers do not
adapt: few people are using greylisting. 

--
To escape the bear, you do not need to run faster than the bear, just
to run faster than your friends.

Date: Tue, 11 Oct 2011 10:32:05 -0400
From: Hector <[email-masked]>
CC: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Patrik Faltstrom (pfaltstr) wrote:
> Difficult...the whole idea with grey listing is that smtp server 
> that receive mail is using corners of the protocol that smtp servers 
> have implemented but not spam serving software.

Difficult but in my view doable with a possible high payoff for everyone.

> Exactly where and what that difference is will always differ over 
> time, and by definition can not be standardized.
> 
> So, what to include in such a draft?
> 
> How to act in the case of 4xx? How to format 4xx responses?
> 

Yes, I think that would be the only reasonable "major benefit" to
consider in such a draft - i.e. a "method" that can really help systems.

But I can also see where we (the industry) now has enough years into
GL implementation and we have history and recommendations to be outlined.

Quick outline:

    - Intro and background

    - Remind people that TIMELY DELIVERY is still important

    - Mapping GL Entities (i.e. the triplet vs other factors)
        - Class C issues
        - Sender vs RFC5322 content (i.e. Message-ID)

    - Where in the SMTP state machine 4yz GL rejections
         - Connection vs RCPT vs increasingly at DATA

    - Recommended Delays

    - Recommended Structured Formats for Responses
        - Extended Reply codes RFC2034 Considerations

It might be interesting technical reading for all and it might even
useful for new people implementing Greylisting in their operations.

--
HLS

Date: 11 Oct 2011 13:59:28 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Organization: 
Sender: [email-masked]

In article <[email-masked]> you write:
>According to my statistic, this year marks a massive growth in SMTP 
>receiver implementing greylisting ...

My logs don't show any recent increase in greylisting.  For the stuff I
see, the usual retry schedule works as well as it ever did.

Perhaps Hector is misreading his statistic.

R's,
John

From: "Tim Kehres" <[email-masked]>
To: "Hector" <[email-masked]>, <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Date: Tue, 11 Oct 2011 21:55:17 +0800
Sender: [email-masked]

I think a BCP that outlines the advantages and disadvantages of various 
timeout values can be a good thing.  The most we've run for a timeout is 30 
minutes but now typically run with much shorter timeouts.  On the other hand 
recently we have been seeing increases in spam getting through the GL so I 
suspect that some might feel compelled to adjust the timeout window to 
compensate.   Of course making this window too large will start to have an 
adverse effect on the legit sending site queues and frequency in which they 
attempt a retry.

On the other hand if we're trying to standardize on GL response codes, I 
don't really see the utility in this.  GL's are there for one reason - to 
effect a simple and cheap (something that can be done at connection time 
rather than after queuing) anti-spam measure.  Giving the bad guys more 
information on how you are protecting yourself seems contradictory to the 
primary objective.   The other systems trying to communicate with you (the 
good guys) likely won't bother to do anything with this additional 
information anyway as they will be governed by their own internal backoff 
algorithms and timing.

What am I missing here?

Best Regards,

-- Tim

----- Original Message ----- 
From: "Hector" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, October 11, 2011 8:39 PM
Subject: Re: We need an IETF BCP for GREY LISTING

>
> That might be the "ideal" but it will be more complex (bigger effort). The 
> problem with basing it on extended reply codes (RFC2034) that it would 
> mean the server itself will need to support it and also advertise it.  It 
> may implement greylisting and not support extended reply codes, nor the 
> client support it.
>
> I think we need something for Greylist that does not require RFC2034, but 
> a GreyList "response format" that can be parse because today, I don't 
> think there is any "official standard" consideration for parsing the text 
> part of the greylist response.  You see the attempts at layman "english" 
> strings but no structure to it and I think that is probably the best one 
> can get - some structure.
>
> But the BCP can also touch based with current methods in implementing 
> Greylist, with recommendations, etc.
>
> --
> HLS
>
> Rosenwald, Jordan wrote:
>> I might be missing your point, so correct me if I am, but it sounds like 
>> a BCP for extended response codes more than a position/policy/BCP for 
>> greylisting.
>>
>> ----- Reply message -----
>> From: "Hector" <[email-masked]>
>> To: "[email-masked]" <[email-masked]>
>> Subject: We need an IETF BCP for GREY LISTING
>> Date: Tue, Oct 11, 2011 7:47 am
>>
>>
>>
>>
>> Folks,
>>
>> According to my statistic, this year marks a massive growth in SMTP
>> receiver implementing greylisting to the point that some systems are
>> becoming more "elaborate" with their responses, including multi-line
>> response such as this as a response to the DATA EOD response:
>>
>> 451-DEFER - TB3 - Try a lower numbered MX record - S=1  FakeMX - FAKE-MX
>> 451-I=[xxx.xxx.xxx.xxx] X=tarbaby H=xxx.xxxxx.com [xxx.xxx.xxx.x]
>> 451-HELO=[xxx.xxxxx.com] F=[[email-masked]]
>> 451 T=[[email-masked]]
>>
>> More so, the delays requires to retry vary greatly and sometimes it
>> seems like it can a day.
>>
>> I think we finally need an official IETF "BCP" written up to begin
>> maybe coordinate the client/server operation to help to alleviate the
>> increasing delays and wasted retries to the point where i am now
>> hearing complaints with the ATTEMPTS are exhausted and the destination
>> is "flagged" as bad domain or address.
>>
>> One suggestion for the GREYLIST "BCP" is if a receiver is going to
>> block for an extensive time, it should maybe provide that feedback in
>> should official response format.
>>
>> For use, our stock retry attempt table has initial 5, 10, 15, 30
>> minutes delays before it fall backs to 1 hour for the remaining
>> attempts up to 4 days.
>>
>> Greylisting has become essentially a "pseudo" standard today and too
>> many systems are becoming too aggressive with delays.   This needs to
>> be coordinated better.
>>
>> I think someone should take up the effort to begin/draft an GreyList
>> BCP for systems to follow.
>>
>> Comments?
>>
>
> 

Date: Tue, 11 Oct 2011 09:50:19 -0400
From: Hector <[email-masked]>
To: "Rosenwald, Jordan" <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Rosenwald, Jordan wrote:
> I agree structure = good, but in this case what is the endgame?  What's 
> the value?  What would you do with the information if MTAs returned it?

Teach the MTA when they should reschedule to not waste attempts.

When we first implemented GL in early 2003, the two immediate needs were:

    - Shorter 2nd and 3rd Retries (within 5 mins)
    - Automated whitelisting of customers

So our implementation comes with auto white listing of recipients you 
directly write mail to with the idea you know them and would expect 
responses.  This is done at the SMTP receiver for authorized mail 
authors of the local host.   When anonymous senders, the greylist 
process checks this auto-whitelist.

And shorting the initial retries help address the delivery delays 
against remote sites now implementing Greylist.

While the history has been there were systems that wanted you to retry 
much later, maybe 1 hour, what I have been seeing increasing it has 
beyond 1 hour into days now.  Today, this has caused some false 
positive bounces and flagging of "bad destination" address.

What will help is a structured response that will help the MTA to 
reschedule at the appropriate time and now waste all the attempts.

Further, there are customers with large mailing list to different 
destination email domains but are hosted by the same remote host. 
i.e. many email domains dealing in the same trade might be using the 
same set of MX hosting domains.  They are now greylisting too and 
there are now a tremendous amount of waste in sending attempts.

Some don't even accept mail until after 5:00pm - end of the business.

Some TAKE down their server at 5:00pm  and only bring it back at 9am 
the next morning.

I am seeing far too much waste in sending mail with GREYLIST 
implementations or those that "behave" like so with non-FULL time 
operations.

Something can be done.

--
HLS

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Tue, 11 Oct 2011 09:23:04 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

--Apple-Mail-6-729767783
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Oct 11, 2011, at 9:08 AM, Hector wrote:

> IMO, I think that is good because the main logic behind GL is that =
clients are expected to retry and IMO, it works because legacy Bad Guys =
will never adapt anyway, regardless of what you do.=20

to me it seems like spammers have been adapting to countermeasures =
fairly well, for at least 20 years.

--Apple-Mail-6-729767783
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><div>On Oct 11, 2011, at 9:08 AM, Hector wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
font-family: Helvetica; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; ">IMO, I think =
that is good because the main logic behind GL is that clients are =
expected to retry and IMO, it works because legacy Bad Guys will never =
adapt anyway, regardless of what you do.<span =
class=3D"Apple-converted-space">&nbsp;</span></span></blockquote></div><br=
><div>to me it seems like spammers have been adapting to countermeasures =
fairly well, for at least 20 years.</div><div><br></div></body></html>=

--Apple-Mail-6-729767783--

Date: Tue, 11 Oct 2011 09:08:35 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Keith Moore wrote:
> On Oct 11, 2011, at 7:43 AM, Hector wrote:
> 
>> I think someone should take up the effort to begin/draft an GreyList 
>> BCP for systems to follow.
>>
>> Comments?
> 
> I'm not sure that it's worth the trouble.   The whole point of greylisting 
> is to marginalize naive client implementations on the assumption (largely valid 
> to this point) that they're likely to be spambots.   But I expect that 
> spambots will start to deal with greylisting very soon.  If IETF were to 
> document greylisting it would only accelerate that process.

IMO, I think that is good because the main logic behind GL is that 
clients are expected to retry and IMO, it works because legacy Bad 
Guys will never adapt anyway, regardless of what you do.  These are 
the ones GL tries to get and its most effective against.  We don't 
want to hurt anyone else - even if you are a "spammer" - the rule is 
simple - use a Standard SMTP expectation for retrying.

The problem is that everyone is now doing it and in varying ways to 
delay you.  Now you need smarter Retry Attempts table at the DOMAIN 
level but you won't have that information until after the fact.

What I am finding are a few new basic issues this year

    - More greylisting even at the BUSINESS and CORPORATE level,
    - Longer delays
    - "False positive" bad domain flagging of exhausted attempts

The latter is what was presented to me, however, in this case, the 
customer got a bounce after the total retry attempts and stated he 
never had a problem before sending to this remote host.  It appeared 
the remote implemented Greylisting.  We took the original archived 
message, forced a retry and PUFF - it was accepted.  That was 1 day 
later - Go Figure!   Looking at the logs, there were similar sessions 
traces where it took nearly a day before it was accepted.

That is ridiculous in my book.

--
HLS

From: "Rosenwald, Jordan" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: RE: We need an IETF BCP  for GREY LISTING
Date: Tue, 11 Oct 2011 12:50:26 +0000
Accept-Language: en-US
Sender: [email-masked]

I agree structure = good, but in this case what is the endgame?  What's the value?  What would you do with the information if MTAs returned it?

> -----Original Message-----
> From: [email-masked] [mailto:owner-ietf-
> [email-masked]] On Behalf Of Hector
> Sent: Tuesday, October 11, 2011 8:40 AM
> To: [email-masked]
> Subject: Re: We need an IETF BCP for GREY LISTING
> 
> 
> That might be the "ideal" but it will be more complex (bigger effort).
>   The problem with basing it on extended reply codes (RFC2034) that it would
> mean the server itself will need to support it and also advertise it.  It may
> implement greylisting and not support extended reply codes, nor the client
> support it.
> 
> I think we need something for Greylist that does not require RFC2034, but a
> GreyList "response format" that can be parse because today, I don't think
> there is any "official standard" consideration for parsing the text part of the
> greylist response.  You see the attempts at layman "english" strings but no
> structure to it and I think that is probably the best one can get - some
> structure.
> 
> But the BCP can also touch based with current methods in implementing
> Greylist, with recommendations, etc.
> 
> --
> HLS
> 
> Rosenwald, Jordan wrote:
> > I might be missing your point, so correct me if I am, but it sounds
> > like a BCP for extended response codes more than a position/policy/BCP
> > for greylisting.
> >
> > ----- Reply message -----
> > From: "Hector" <[email-masked]>
> > To: "[email-masked]" <[email-masked]>
> > Subject: We need an IETF BCP for GREY LISTING
> > Date: Tue, Oct 11, 2011 7:47 am
> >
> >
> >
> >
> > Folks,
> >
> > According to my statistic, this year marks a massive growth in SMTP
> > receiver implementing greylisting to the point that some systems are
> > becoming more "elaborate" with their responses, including multi-line
> > response such as this as a response to the DATA EOD response:
> >
> > 451-DEFER - TB3 - Try a lower numbered MX record - S=1  FakeMX -
> > FAKE-MX 451-I=[xxx.xxx.xxx.xxx] X=tarbaby H=xxx.xxxxx.com
> > [xxx.xxx.xxx.x] 451-HELO=[xxx.xxxxx.com] F=[[email-masked]]
> > 451 T=[[email-masked]]
> >
> > More so, the delays requires to retry vary greatly and sometimes it
> > seems like it can a day.
> >
> > I think we finally need an official IETF "BCP" written up to begin
> > maybe coordinate the client/server operation to help to alleviate the
> > increasing delays and wasted retries to the point where i am now
> > hearing complaints with the ATTEMPTS are exhausted and the destination
> > is "flagged" as bad domain or address.
> >
> > One suggestion for the GREYLIST "BCP" is if a receiver is going to
> > block for an extensive time, it should maybe provide that feedback in
> > should official response format.
> >
> > For use, our stock retry attempt table has initial 5, 10, 15, 30
> > minutes delays before it fall backs to 1 hour for the remaining
> > attempts up to 4 days.
> >
> > Greylisting has become essentially a "pseudo" standard today and too
> > many systems are becoming too aggressive with delays.   This needs to
> > be coordinated better.
> >
> > I think someone should take up the effort to begin/draft an GreyList
> > BCP for systems to follow.
> >
> > Comments?
> >

Date: Tue, 11 Oct 2011 08:39:46 -0400
From: Hector <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

That might be the "ideal" but it will be more complex (bigger effort). 
  The problem with basing it on extended reply codes (RFC2034) that it 
would mean the server itself will need to support it and also 
advertise it.  It may implement greylisting and not support extended 
reply codes, nor the client support it.

I think we need something for Greylist that does not require RFC2034, 
but a GreyList "response format" that can be parse because today, I 
don't think there is any "official standard" consideration for parsing 
the text part of the greylist response.  You see the attempts at 
layman "english" strings but no structure to it and I think that is 
probably the best one can get - some structure.

But the BCP can also touch based with current methods in implementing 
Greylist, with recommendations, etc.

--
HLS

Rosenwald, Jordan wrote:
> I might be missing your point, so correct me if I am, but it 
> sounds like a BCP for extended response codes more than a 
> position/policy/BCP for greylisting.
> 
> ----- Reply message -----
> From: "Hector" <[email-masked]>
> To: "[email-masked]" <[email-masked]>
> Subject: We need an IETF BCP for GREY LISTING
> Date: Tue, Oct 11, 2011 7:47 am
> 
> 
> 
> 
> Folks,
> 
> According to my statistic, this year marks a massive growth in SMTP
> receiver implementing greylisting to the point that some systems are
> becoming more "elaborate" with their responses, including multi-line
> response such as this as a response to the DATA EOD response:
> 
> 451-DEFER - TB3 - Try a lower numbered MX record - S=1  FakeMX - FAKE-MX
> 451-I=[xxx.xxx.xxx.xxx] X=tarbaby H=xxx.xxxxx.com [xxx.xxx.xxx.x]
> 451-HELO=[xxx.xxxxx.com] F=[[email-masked]]
> 451 T=[[email-masked]]
> 
> More so, the delays requires to retry vary greatly and sometimes it
> seems like it can a day.
> 
> I think we finally need an official IETF "BCP" written up to begin
> maybe coordinate the client/server operation to help to alleviate the
> increasing delays and wasted retries to the point where i am now
> hearing complaints with the ATTEMPTS are exhausted and the destination
> is "flagged" as bad domain or address.
> 
> One suggestion for the GREYLIST "BCP" is if a receiver is going to
> block for an extensive time, it should maybe provide that feedback in
> should official response format.
> 
> For use, our stock retry attempt table has initial 5, 10, 15, 30
> minutes delays before it fall backs to 1 hour for the remaining
> attempts up to 4 days.
> 
> Greylisting has become essentially a "pseudo" standard today and too
> many systems are becoming too aggressive with delays.   This needs to
> be coordinated better.
> 
> I think someone should take up the effort to begin/draft an GreyList
> BCP for systems to follow.
> 
> Comments?
> 

From: "Rosenwald, Jordan" <[email-masked]>
To: Hector <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Date: Tue, 11 Oct 2011 12:09:25 +0000
Accept-Language: en-US
Sender: [email-masked]

--_000_BAD7FD4EC8F54F8BAC2E12DAC8C89DCEcablecomcastcom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

--_000_BAD7FD4EC8F54F8BAC2E12DAC8C89DCEcablecomcastcom_
Content-Type: text/html; charset="utf-8"
Content-ID: <[email-masked]>
Content-Transfer-Encoding: base64

--_000_BAD7FD4EC8F54F8BAC2E12DAC8C89DCEcablecomcastcom_--

From: "Rosenwald, Jordan" <[email-masked]>
To: Hector <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: We need an IETF BCP  for GREY LISTING
Date: Tue, 11 Oct 2011 12:08:42 +0000
Accept-Language: en-US
Sender: [email-masked]

--_000_1CCA36778A744A0280053F93D31E1CEBcablecomcastcom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I might be missing your point, so correct me if I am, but it sounds like a =
BCP for extended response codes more than a position/policy/BCP for greylis=
ting.

----- Reply message -----
From: "Hector" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: We need an IETF BCP for GREY LISTING
Date: Tue, Oct 11, 2011 7:47 am

Folks,

According to my statistic, this year marks a massive growth in SMTP
receiver implementing greylisting to the point that some systems are
becoming more "elaborate" with their responses, including multi-line
response such as this as a response to the DATA EOD response:

451-DEFER - TB3 - Try a lower numbered MX record - S=3D1  FakeMX - FAKE-MX
451-I=3D[xxx.xxx.xxx.xxx] X=3Dtarbaby H=3Dxxx.xxxxx.com [xxx.xxx.xxx.x]
451-HELO=3D[xxx.xxxxx.com] F=3D[[email-masked]]
451 T=3D[[email-masked]]

More so, the delays requires to retry vary greatly and sometimes it
seems like it can a day.

I think we finally need an official IETF "BCP" written up to begin
maybe coordinate the client/server operation to help to alleviate the
increasing delays and wasted retries to the point where i am now
hearing complaints with the ATTEMPTS are exhausted and the destination
is "flagged" as bad domain or address.

One suggestion for the GREYLIST "BCP" is if a receiver is going to
block for an extensive time, it should maybe provide that feedback in
should official response format.

For use, our stock retry attempt table has initial 5, 10, 15, 30
minutes delays before it fall backs to 1 hour for the remaining
attempts up to 4 days.

Greylisting has become essentially a "pseudo" standard today and too
many systems are becoming too aggressive with delays.   This needs to
be coordinated better.

I think someone should take up the effort to begin/draft an GreyList
BCP for systems to follow.

Comments?

--_000_1CCA36778A744A0280053F93D31E1CEBcablecomcastcom_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; pad=
ding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<style>
<!--
.x_EmailQuote
	{margin-left:1pt;
	padding-left:4pt;
	border-left:#800000 2px solid}
-->
</style>
<div>I might be missing your point, so correct me if I am, but it sounds li=
ke a BCP for extended response codes more than a position/policy/BCP for gr=
eylisting.<br>
<br>
<div id=3D"x_htc_header" style=3D"">----- Reply message -----<br>
From: &quot;Hector&quot; &lt;[email-masked]&gt;<br>
To: &quot;[email-masked]&quot; &lt;[email-masked]&gt;<br>
Subject: We need an IETF BCP for GREY LISTING<br>
Date: Tue, Oct 11, 2011 7:47 am<br>
<br>
</div>
<br>
<br>
</div>
<font size=3D"2"><span style=3D"font-size:10pt;">
<div class=3D"PlainText"><br>
Folks,<br>
<br>
According to my statistic, this year marks a massive growth in SMTP <br>
receiver implementing greylisting to the point that some systems are <br>
becoming more &quot;elaborate&quot; with their responses, including multi-l=
ine <br>
response such as this as a response to the DATA EOD response:<br>
<br>
451-DEFER - TB3 - Try a lower numbered MX record - S=3D1&nbsp; FakeMX - FAK=
E-MX<br>
451-I=3D[xxx.xxx.xxx.xxx] X=3Dtarbaby H=3Dxxx.xxxxx.com [xxx.xxx.xxx.x]<br>
451-HELO=3D[xxx.xxxxx.com] F=3D[[email-masked]]<br>
451 T=3D[[email-masked]]<br>
<br>
More so, the delays requires to retry vary greatly and sometimes it <br>
seems like it can a day.<br>
<br>
I think we finally need an official IETF &quot;BCP&quot; written up to begi=
n <br>
maybe coordinate the client/server operation to help to alleviate the <br>
increasing delays and wasted retries to the point where i am now <br>
hearing complaints with the ATTEMPTS are exhausted and the destination <br>
is &quot;flagged&quot; as bad domain or address.<br>
<br>
One suggestion for the GREYLIST &quot;BCP&quot; is if a receiver is going t=
o <br>
block for an extensive time, it should maybe provide that feedback in <br>
should official response format.<br>
<br>
For use, our stock retry attempt table has initial 5, 10, 15, 30 <br>
minutes delays before it fall backs to 1 hour for the remaining <br>
attempts up to 4 days.<br>
<br>
Greylisting has become essentially a &quot;pseudo&quot; standard today and =
too <br>
many systems are becoming too aggressive with delays.&nbsp;&nbsp; This need=
s to <br>
be coordinated better.<br>
<br>
I think someone should take up the effort to begin/draft an GreyList <br>
BCP for systems to follow.<br>
<br>
Comments?<br>
<br>
</div>
</span></font>
</body>
</html>

--_000_1CCA36778A744A0280053F93D31E1CEBcablecomcastcom_--

Subject: Re: We need an IETF BCP  for GREY LISTING
From: Keith Moore <[email-masked]>
Date: Tue, 11 Oct 2011 08:06:16 -0400
Cc: [email-masked]
To: Hector <[email-masked]>
Sender: [email-masked]

On Oct 11, 2011, at 7:43 AM, Hector wrote:

> I think someone should take up the effort to begin/draft an GreyList BCP for systems to follow.
> 
> Comments?

I'm not sure that it's worth the trouble.   The whole point of greylisting is to marginalize naive client implementations on the assumption (largely valid to this point) that they're likely to be spambots.   But I expect that spambots will start to deal with greylisting very soon.  If IETF were to document greylisting it would only accelerate that process.

Keith

Date: Tue, 11 Oct 2011 07:43:30 -0400
From: Hector <[email-masked]>
To: [email-masked]
Subject: We need an IETF BCP  for GREY LISTING
Sender: [email-masked]

Folks,

According to my statistic, this year marks a massive growth in SMTP 
receiver implementing greylisting to the point that some systems are 
becoming more "elaborate" with their responses, including multi-line 
response such as this as a response to the DATA EOD response:

451-DEFER - TB3 - Try a lower numbered MX record - S=1  FakeMX - FAKE-MX
451-I=[xxx.xxx.xxx.xxx] X=tarbaby H=xxx.xxxxx.com [xxx.xxx.xxx.x]
451-HELO=[xxx.xxxxx.com] F=[[email-masked]]
451 T=[[email-masked]]

More so, the delays requires to retry vary greatly and sometimes it 
seems like it can a day.

I think we finally need an official IETF "BCP" written up to begin 
maybe coordinate the client/server operation to help to alleviate the 
increasing delays and wasted retries to the point where i am now 
hearing complaints with the ATTEMPTS are exhausted and the destination 
is "flagged" as bad domain or address.

One suggestion for the GREYLIST "BCP" is if a receiver is going to 
block for an extensive time, it should maybe provide that feedback in 
should official response format.

For use, our stock retry attempt table has initial 5, 10, 15, 30 
minutes delays before it fall backs to 1 hour for the remaining 
attempts up to 4 days.

Greylisting has become essentially a "pseudo" standard today and too 
many systems are becoming too aggressive with delays.   This needs to 
be coordinated better.

I think someone should take up the effort to begin/draft an GreyList 
BCP for systems to follow.

Comments?

Date: Mon, 10 Oct 2011 14:00:01 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: =?windows-1252?Q?Julien_=C9LIE?= <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: How aggressively to reject Pipelining errors?
Sender: [email-masked]

Julien �LIE wrote:
>> In order to deal with the STARTTLS plaintext command injection
>> vulnerability (CERT VU#555316), I added a check in my commands parser to
>> error out on "illegal pipelining"....
>>
>> At the moment, I'm only performing this check after the STARTTLS
>> command. But that got me to wondering: would it be wise/unwise to check
>> after other commands, e.g., EHLO, DATA, or AUTH? RFC 2920 is silent on
>> the topic of what the server may do if the client violates the rules.
>
> For what is worth, RFC 3977 (about NNTP) says:
> ....
>    If the specific description of a command says it "MUST NOT be
>    pipelined", that command MUST end any pipeline of commands.  That is,
>    the client MUST NOT send any following command until it receives the
>    CRLF at the end of the response from the command.  The server MAY
>    ignore any data received after the command and before the CRLF at the
>    end of the response is sent to the client.

I appreciate the thought, but I'm not sure the NNTP behavior has much 
relevance here. There's no provision in SMTP for ever throwing away any 
commands under any circumstances, and the strong historical precedent is 
to return an unambiguous error message any time the server doesn't 
understand the client.

> Anyway, NNTP does not provide a generic response code to reject an 
> unwanted command.  Existing response codes have other meanings, and it 
> does not seem great to re-use one of them, if not standardized.

Right, whereas SMTP does.

I think the differences here between NNTP and SMTP are so deep that the 
NNTP behavior doesn't really provide useful guidance for SMTP. But 
thanks, it's been too long since I looked at NNTP.

<csg>

From: [email-masked]
Cc: "Carl S. Gutekunst" <[email-masked]>, SMTP Interest Group <[email-masked]>
Date: Sat, 08 Oct 2011 12:22:17 -0700 (PDT)
Subject: Re: How aggressively to reject Pipelining errors?
To: Paul Smith <[email-masked]>
Sender: [email-masked]

> On 07/10/2011 01:43, Carl S. Gutekunst wrote:
> >
> > In order to deal with the STARTTLS plaintext command injection
> > vulnerability (CERT VU#555316), I added a check in my commands parser
> > to error out on "illegal pipelining": if there is anything in the read
> > I/O buffer when there shouldn't be, I return a 421 response and drop
> > the connection.
> Hang on, I'm worrying now - we don't do anything special about checking
> the I/O buffer here, but I'm not sure how it could be abused.

> If the client sends STARTTLS and other commands, then how would it get
> through the STARTTLS negotiation, since it won't be able to pipeline the
> negotiation data (not knowing what any challenges will be), so the
> STARTTLS will fail, and the other pipelined commands will be processed
> outside the TLS session.

> Or, am I missing something?

It depends on your buffering and SSL/TLS implementation. If, say, your SSL/TLS
implementation only reads from a socket, when you turn it on it cannot see
data in application-layer buffers. So that data sits there until the SSL/TLS
negotiation is complete, and then gets consumed by the application and it
appears to have come in under the security layer.

Even then this probably doesn't amount to all that much in the case of SMTP,
but the combination of IMAP or POP and client certs can present a real
exposure.

				Ned

Date: Sat, 08 Oct 2011 19:36:34 +0100
From: Paul Smith <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: How aggressively to reject Pipelining errors?
Sender: [email-masked]

On 07/10/2011 01:43, Carl S. Gutekunst wrote:
>
> In order to deal with the STARTTLS plaintext command injection 
> vulnerability (CERT VU#555316), I added a check in my commands parser 
> to error out on "illegal pipelining": if there is anything in the read 
> I/O buffer when there shouldn't be, I return a 421 response and drop 
> the connection.
Hang on, I'm worrying now - we don't do anything special about checking 
the I/O buffer here, but I'm not sure how it could be abused.

If the client sends STARTTLS and other commands, then how would it get 
through the STARTTLS negotiation, since it won't be able to pipeline the 
negotiation data (not knowing what any challenges will be), so the 
STARTTLS will fail, and the other pipelined commands will be processed 
outside the TLS session.

Or, am I missing something?

Date: Sat, 08 Oct 2011 12:56:17 +0200
From: =?windows-1252?Q?Julien_=C9LIE?= <[email-masked]>
Organization: TrigoFACILE -- http://www.trigofacile.com/
To: SMTP Interest Group <[email-masked]>
CC: "Carl S. Gutekunst" <[email-masked]>
Subject: Re: How aggressively to reject Pipelining errors?
Sender: [email-masked]

Hi Carl,

> In order to deal with the STARTTLS plaintext command injection
> vulnerability (CERT VU#555316), I added a check in my commands parser to
> error out on "illegal pipelining": if there is anything in the read I/O
> buffer when there shouldn't be, I return a 421 response and drop the
> connection. My reasoning was that I don't want to allow the session to
> continue if the client appears to be testing for that vulnerability.
>
> At the moment, I'm only performing this check after the STARTTLS
> command. But that got me to wondering: would it be wise/unwise to check
> after other commands, e.g., EHLO, DATA, or AUTH? RFC 2920 is silent on
> the topic of what the server may do if the client violates the rules.

For what is worth, RFC 3977 (about NNTP) says:

3.5.  Pipelining

    Except where stated otherwise, a client MAY use pipelining.  That is,
    it may send a command before receiving the response for the previous
    command.  The server MUST allow pipelining and MUST NOT throw away
    any text received after a command.  Irrespective of whether
    pipelining is used, the server MUST process commands in the order
    they are sent.

    If the specific description of a command says it "MUST NOT be
    pipelined", that command MUST end any pipeline of commands.  That is,
    the client MUST NOT send any following command until it receives the
    CRLF at the end of the response from the command.  The server MAY
    ignore any data received after the command and before the CRLF at the
    end of the response is sent to the client.

STARTTLS, defined in RFC 4642 for NNTP, does not allow pipelining.
News servers can therefore just ignore the unwanted data.

The problem with returning an answer is that it can hinder a bug in the 
client (which may never see the bug because it will for instance send 
its command again, instead of exiting after a timeout) or cause 
desynchronization issues in case the server does not answer as many 
responses as the number of unwanted commands sent by the client.

Anyway, NNTP does not provide a generic response code to reject an 
unwanted command.  Existing response codes have other meanings, and it 
does not seem great to re-use one of them, if not standardized.

-- 
Julien �LIE

� J'adore �tre pris en flagrant d�lire. � (Raymond Devos)

Date: Fri, 7 Oct 2011 11:35:57 +0100
From: Tony Finch <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: Re: How aggressively to reject Pipelining errors?
Sender: [email-masked]

Carl S. Gutekunst <[email-masked]> wrote:
>
> At the moment, I'm only performing this check after the STARTTLS command. But
> that got me to wondering: would it be wise/unwise to check after other
> commands, e.g., EHLO, DATA, or AUTH? RFC 2920 is silent on the topic of what
> the server may do if the client violates the rules.

Exim does fairly aggressive synchronization checks by default and has done
for many years. It does not cause interop problems in my experience.

See the "pipelining_advertise_hosts", "smtp_enforce_sync", and
"smtp_max_synprot_errors" options at
http://exim.org/exim-html-current/doc/html/spec_html/ch14.html

(sorry, the options don't have individual anchors)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Lundy, Fastnet, Irish Sea: Northwest backing west 5 to 7, decreasing 4 for a
time. Moderate or rough, but very rough in Fastnet at first. Squally showers,
rain later. Good, occasionally moderate.

Date: Fri, 07 Oct 2011 10:29:13 +0200
From: Alessandro Vesely <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 07/Oct/11 02:39, Dave CROCKER wrote:
> On 10/5/2011 11:47 AM, Carl S. Gutekunst wrote:
>
>> The problem is nearly all of our anti-spam measures are empirical.
> 
> The real problem is that you think that empirical foundation matters
> in these sorts of discussions...

Doesn't it?  IMHO, an empirically-founded method is likely to be a
better candidate for standardizing an anti-spam technique than a
purely theoretical work.  After this rationale, e.g., I'll be
pondering on whether or not to propose

   192.0.2.3.abuse-contact.arpa. TXT "[email-masked]" [*]

based on the pros and cons that have been highlighted here, as well as
the interest that this topic seems to elicit.

[*] see http://www.imc.org/ietf-smtp/mail-archive/msg06625.html
(Actually, RIRs are rather skeptical on the effectiveness of this
tool.  Quite few abuse-mailboxes are currently defined in the whois
databases, and their reliability will be worsen by making them
mandatory.  The outcome of my pondering is currently "no".)

Date: Thu, 06 Oct 2011 17:43:17 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: How aggressively to reject Pipelining errors?
Sender: [email-masked]

In order to deal with the STARTTLS plaintext command injection 
vulnerability (CERT VU#555316), I added a check in my commands parser to 
error out on "illegal pipelining": if there is anything in the read I/O 
buffer when there shouldn't be, I return a 421 response and drop the 
connection. My reasoning was that I don't want to allow the session to 
continue if the client appears to be testing for that vulnerability.

At the moment, I'm only performing this check after the STARTTLS 
command. But that got me to wondering: would it be wise/unwise to check 
after other commands, e.g., EHLO, DATA, or AUTH? RFC 2920 is silent on 
the topic of what the server may do if the client violates the rules.

<csg>

Date: Thu, 06 Oct 2011 17:41:54 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Murray S. Kucherawy" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 10/5/2011 7:23 AM, Murray S. Kucherawy wrote:
> It's probably not a good idea to standardize such a practice (yet?), but it does seem like a useful tool to have around.

Perhaps it would be worth a brief MAAWG White Paper with a title along the lines of:

    Small, Worthy Anti-Abuse Hacks, for Small, Worthy Benefits ?

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: Thu, 06 Oct 2011 17:39:41 -0700
From: Dave CROCKER <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: "Carl S. Gutekunst" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 10/5/2011 11:47 AM, Carl S. Gutekunst wrote:
> The problem is nearly all of our anti-spam measures are empirical.

The real problem is that you think that empirical foundation matters in these 
sorts of discussions...

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

Date: 7 Oct 2011 00:27:33 -0000
From: "John Levine" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: The anti-abuse rDNS check that FTP gave up
Organization: 
Sender: [email-masked]

>I don't know which ones are responsible and which ones aren't.  But
>the idea that some third party gets to say, effectively, whether your
>mail gets delivered, and you have no recourse against that third
>party, is fundamentally broken and intolerable.

Hi, Keith.  My spam filters catch about 70,000 spams a day, mostly
using DNSBLs, with a false positive rate well under 1%.  If you're
offering to sort them, and can promise a similar error rate, I'll
send them along.

If not, I'd appreciate it if you'd explain exactly what operational
advice you're offering here.

R's,
John

Date: Thu, 06 Oct 2011 12:29:49 -0700
From: Douglas Otis <[email-masked]>
To: Keith Moore <[email-masked]>
CC: "Derek J. Balling" <[email-masked]>, "Rosenwald, Jordan" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 10/6/11 6:48 AM, Keith Moore wrote:
>> Perhaps you need to re-examine your understanding of how DNSBLs actually operate.
> I understand them quite well.  It's called dilution of responsibility.  It's a great mechanism for causing harm while escaping blame.  The destination MTA blames the RBL, the RBL blames the MTA, the end users get screwed.
Keith,

IMHO, the strongest motivator for changing email stems from a provider's 
desire to redirect complaints elsewhere.  As a result, schemes like DKIM 
allow any number of signatures that also fail to capture either who sent 
the message or their intended recipients.  From a domain name 
perspective, these missing details cut both directions, in either 
dodging responsibility or in allowing actions beyond their control held 
against them by a reputation service.  With the uncertainty involved, 
challenges by large domains are likely to find issues resolved in their 
favor.  The same applies for SPF, where such records authorize the 
shared providers.  With no certainty which message element selects SPF 
records, or what assurances were offered to assure against SPF related 
spoofing, it would be risky to even use SPF as a basis for directing 
feedback, rather than depending upon who controls the IP address.  Of 
course, this also ignores DDoS threats related to SPF macro processing.

MAPS attempted to categorize the entire IPv4 address space, often in 
conjunction with assistance from larger ISPs.  When a large ISP emitted 
a significant amount of spam, they would be encouraged to share the 
addresses deployed for access with a ULA prohibiting public servers (the 
DUL), or to simply block port 25.  This type of information was 
essential to combat spammers employing asymmetric strategies, where even 
modem dial-ups appeared to emit a volume normally requiring T3 
connections.  To assist in maintaining the DUL list, ISPs were asked how 
they assigned reverse DNS for this space.  Conventions established 
better ensure against DUL listing mistakes.  In the era of IPv4, these 
addresses were considered sufficiently limited and this permitted 
aggressive categorization to re-actively block abusive sources.  Due to 
the unlimited nature of domain names, control primarily focused on IP 
addresses to conserve resources.  That era ends with the Internet 
adopting IPv6.

Even MTAs refusing to accept IPv6 connections will still be affected.  
There are any number of translation schemes where an IPv4 address no 
longer represents a single entity.  The announced IPv6 address prefix 
space, ignoring the lower 64 bits, already represents 56,000 times that 
of the entire IPv4 address space, and growing exponentially.  It seems 
growth of IPv6 prefixes should flatten out at about 340,000 times that 
of the IPv4 address space.  This increased size roughly reflects the 
resources that will be needed to extend reactive blocking of abusive 
sources.

Changing reputation to that of recognition from that of reactive 
blocking could dramatically reduce the resources required to convey 
whether a connection is likely non-abusive.  For this to happen, there 
is a need to authenticate outbound MTAs.  This might be done by using a 
challenge that gets signed and confirmed by a public certificate.  
Something like DANE seems well positioned at offering a service that 
could work in conjunction with both IPv6 or IPv4 environments.

With such a scheme, large providers should not have any trouble being 
recognized, but they will then need to deal with complaints.  Hopefully, 
these complaints would also be shared with related customers who may 
have had their system compromised.  What will it take to convince ISPs 
they need to accept this responsibility and that doing so is in their 
best interest?

-Doug

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: "Derek J. Balling" <[email-masked]>
Date: Thu, 6 Oct 2011 09:51:41 -0400
Cc: "Rosenwald, Jordan" <[email-masked]>, SMTP Interest Group <[email-masked]>
To: Keith Moore <[email-masked]>
Sender: [email-masked]

On Oct 6, 2011, at 9:48 AM, Keith Moore wrote:
> I understand them quite well.  It's called dilution of responsibility.  It's a great mechanism for causing harm while escaping blame.  The destination MTA blames the RBL, the RBL blames the MTA, the end users get screwed.

Look, we're all here in a group of people who understand the SMTP spec pretty well. The accepting server is the one for ultimate responsibility for delivering or rejecting the mail. Anything outside of that is local policy that the accepting-server operators use to determine what they want to do, but they maintain responsibility for their policies.

D

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: Keith Moore <[email-masked]>
Date: Thu, 6 Oct 2011 09:48:00 -0400
Cc: "Rosenwald, Jordan" <[email-masked]>, SMTP Interest Group <[email-masked]>
To: "Derek J. Balling" <[email-masked]>
Sender: [email-masked]

On Oct 6, 2011, at 9:43 AM, Derek J. Balling wrote:

> 
> On Oct 6, 2011, at 9:28 AM, Keith Moore wrote:
>> I don't know which ones are responsible and which ones aren't.   But the idea that some third party gets to say, effectively, whether your mail gets delivered, and you have no recourse against that third party, is fundamentally broken and intolerable.
> 
> That third party doesn't "get to say" whether your mail gets delivered.

I did say "effectively".

> The recipient of your message asks a third party for their opinion of you, and then your recipient makes a decision on whether your mail gets delivered/accepted or not.
> 
> Perhaps you need to re-examine your understanding of how DNSBLs actually operate.

I understand them quite well.  It's called dilution of responsibility.  It's a great mechanism for causing harm while escaping blame.  The destination MTA blames the RBL, the RBL blames the MTA, the end users get screwed.

Keith

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: "Derek J. Balling" <[email-masked]>
Date: Thu, 6 Oct 2011 09:43:24 -0400
Cc: "Rosenwald, Jordan" <[email-masked]>, SMTP Interest Group <[email-masked]>
To: Keith Moore <[email-masked]>
Sender: [email-masked]

On Oct 6, 2011, at 9:28 AM, Keith Moore wrote:
> I don't know which ones are responsible and which ones aren't.   But the idea that some third party gets to say, effectively, whether your mail gets delivered, and you have no recourse against that third party, is fundamentally broken and intolerable.

That third party doesn't "get to say" whether your mail gets delivered.

The recipient of your message asks a third party for their opinion of you, and then your recipient makes a decision on whether your mail gets delivered/accepted or not.

Perhaps you need to re-examine your understanding of how DNSBLs actually operate.

D

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: Keith Moore <[email-masked]>
Date: Thu, 6 Oct 2011 09:28:29 -0400
Cc: SMTP Interest Group <[email-masked]>
To: "Rosenwald, Jordan" <[email-masked]>
Sender: [email-masked]

On Oct 6, 2011, at 9:00 AM, Rosenwald, Jordan wrote:

>> ... and to kiss the assess of the RBL operators, and to pay tribute in any
>> number of other arbitrary ways to others who threaten to sabotage your
>> outgoing mail.
> 
> 
> Seriously?  You still view that DNSBL operators function in that fashion?  

I don't know which ones are responsible and which ones aren't.   But the idea that some third party gets to say, effectively, whether your mail gets delivered, and you have no recourse against that third party, is fundamentally broken and intolerable.

Keith

From: "Rosenwald, Jordan" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: RE: The anti-abuse rDNS check that FTP gave up
Date: Thu, 6 Oct 2011 13:00:46 +0000
Accept-Language: en-US
Sender: [email-masked]

> It's ridiculous to put in meaningless PTR records (the kind that just have an IP
> address embedded in them) just so that stupid filters that check for the
> existence of PTR records will think they're valid.   It just gives more
> appearance of credibility to a check that never was meaningful in the first
> place.

That's why most receivers put in logic to ensure the PTR was valid AND meaningful long ago.

> > He got that resolved, but today, if you want to setup an MTA to send out
> mail, you have no choice but to make sure the machine IP has an PTR record.
> 
> ... and to kiss the assess of the RBL operators, and to pay tribute in any
> number of other arbitrary ways to others who threaten to sabotage your
> outgoing mail.

Seriously?  You still view that DNSBL operators function in that fashion?  

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: Keith Moore <[email-masked]>
Date: Thu, 6 Oct 2011 08:31:40 -0400
Cc: SMTP Interest Group <[email-masked]>
To: Hector <[email-masked]>
Sender: [email-masked]

On Oct 6, 2011, at 2:13 AM, Hector wrote:

> Keith Moore wrote:
> 
>>> Right, but this mechanism is not stupid, instead it is clever ;-)
> 
>> nope.  it makes no sense whatsoever.  it is using a completely irrelevant test to decide whether mail is legitimate.  and it's easily defeated by spammers.  it's not only a complete waste of time, it's worse in that it causes legitimate mail to be dropped.
> 
> or at least initially cause transactions to be rejected at specifics receivers doing IP PTR checking.
> 
>> rejecting mail for this reason should be a criminal offense.
> 
> IMO, if the ISP is providing the name servers for a business tier account and they fail to add ARPA records for the IPs, I would consider this MAL-PRACTICE today - certainly tortious interference.

IMO, if an ISP is providing name servers for any customer and they mess with that customers PTR records without specific direction from that customer, that should be considered fraud on the part of the ISP.   Whether those records are populated, and what goes in those that are populated, should be determined by the party to whom those IP addresses are assigned.  

It's ridiculous to put in meaningless PTR records (the kind that just have an IP address embedded in them) just so that stupid filters that check for the existence of PTR records will think they're valid.   It just gives more appearance of credibility to a check that never was meaningful in the first place.   

> He got that resolved, but today, if you want to setup an MTA to send out mail, you have no choice but to make sure the machine IP has an PTR record.

... and to kiss the assess of the RBL operators, and to pay tribute in any number of other arbitrary ways to others who threaten to sabotage your outgoing mail.   

Keith

From: "Storz, Michael" <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>, Keith Moore <[email-masked]>
CC: "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: RE: The anti-abuse rDNS check that FTP gave up
Date: Thu, 6 Oct 2011 11:56:28 +0000
Accept-Language: de-DE, en-US
Sender: [email-masked]

> From: Carl S. Gutekunst [mailto:[email-masked]]
> Sent: Wednesday, October 05, 2011 8:55 PM
> To: Keith Moore
> Cc: Storz, Michael; [email-masked]; SMTP Interest Group
> Subject: Re: The anti-abuse rDNS check that FTP gave up
> 
> Keith Moore wrote:
> > Just imagine how many wrongly rejected emails aren't reported.
> >
> > Stupid spam filtering mechanisms are a DoS attack on email.
> 
> The problem is nearly all of our anti-spam measures are empirical. We
> all know a lot of people who swear by this, that, or the other check,
> even if the only supporting evidence is that using that particular
> mechanism cut down the number of complaints. And what works on one
> stream fails miserably on another.
> 
> FWIW, I last looked at this problem about three years ago. I
> specifically wanted to know if some form of RDNS checks might be useful
> in cutting down the load on the content-based spam filters. (I was also
> checking effect and utilization of SPF and TLS.) Note that the purpose
> here was not to improve the catch rate. This was on a stream that was
> already filtered by a short-lived (60 minute) IP reputation filter, so
> that would reduce the message count from some known spam sources; and
> most of the recipients were business users.
> 
> As I recall, my sample size was something around 10 million E-mails
> from
> a single MX server in a load balanced cluster over a 24-hour period.
> There was a weak correlation between spam and a simple existence check
> for the PTR record. There was no correlation at all for a stronger
> check, e.g., A record matches; a message that failed the strong check
> was as likely to be judged ham as spam by the content filter.
> 
> <csg>

Hi Carl,

this is really interesting, because we got a totally different result when analyzed our data before we introduced the FCrDNS check. To verify our data, I took all rejected IP addresses which had a PTR record but did not have a matching A record from the log of the last 12 hours from one of our servers and run it against the Spamhaus ZEN DNSBL. The result is 

- 420 different IP addresses not on ZEN
- 61722 different IP addresses are on ZEN = 99,3 % of all rejected IP addresses

The distribution to the different parts of ZEN is:

     1 XBL-NJABL
     6 SBLCSS
    24 SBL
  4825 PBL-ISP
 25608 PBL-Spamhaus
 31258 XBL-CBL

I would say, that means there is a strong correlation between spam and at least the second part of the FCrDNS check.

Michael

Date: Thu, 06 Oct 2011 02:13:34 -0400
From: Hector <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

Keith Moore wrote:

>> Right, but this mechanism is not stupid, instead it is clever ;-)

> nope.  it makes no sense whatsoever.  it is using a completely 
> irrelevant test to decide whether mail is legitimate.  and it's 
> easily defeated by spammers.  it's not only a complete waste of time, 
> it's worse in that it causes legitimate mail to be dropped.

or at least initially cause transactions to be rejected at specifics 
receivers doing IP PTR checking.

> rejecting mail for this reason should be a criminal offense.

IMO, if the ISP is providing the name servers for a business tier 
account and they fail to add ARPA records for the IPs, I would 
consider this MAL-PRACTICE today - certainly tortious interference.

I think Otis can fill us in more (MAPS), but as I recall, ISP's did 
add ARPA records or they had provisioned it (outsourced) it or rather 
it wasn't an issue because AOL.COM did not start to do this yet.

But that stopped around 2000-2003 when MAPs which no longer servicing 
ISPs or the ISP decided to save money.   But as the customer support 
issue increased, they realized they needed to do add the ARPA records 
because others were beginning to follow AOL.COM who started this crap.

I agree with you Keith - there is no logic to it, at least I don't 
see, it's moronic and I recall my brother's business getting hurt by 
it when he moved his business to a different provider and they put his 
business tier account on the same trunk as Home based tiers - so he 
didn't get the BIZ privileges he expected.

He got that resolved, but today, if you want to setup an MTA to send 
out mail, you have no choice but to make sure the machine IP has an 
PTR record.
In other words, like it or not, stupid, moronic or not, forget Mom and 
Pop shops, if you want to send to email to aol.com accounts, you need 
a PTR record when connecting to them.

C'est la vie.

---

Date: Thu, 06 Oct 2011 02:44:03 +0100
From: Paul Smith <[email-masked]>
To: "Storz, Michael" <[email-masked]>
CC: "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 05/10/2011 16:28, Storz, Michael wrote:

> Another name for the iprev test is "Forward Confirmed reverse DNS" (FCrDNS). With Postfix you configure it with the two commands
>
>     reject_unknown_reverse_client_hostname
>     reject_unknown_client_hostname
>
> We use this check since years as our first defense against botnet spam with great success. In the last 7 days we rejected emails for nearly 22.000.000 recipients. 49% did not have a PTR record, 29% did not have a matching A record.

Where does RFC 5321 say that a sending MTA needs a PTR record? (or even 
an A record?)

If it doesn't, then the lack of a PTR record does not indicate that the 
MTA is 'wrongly configured'.

Failing FCrDNS shouldn't be sufficient to reject mail. Lots of MTAs 
can't have a 'correct' reverse DNS entry, even if they have one at all. 
Use valid FCrDNS as a way of validating whitelist entries, but surely 
not for more than that.

Date: Wed, 05 Oct 2011 16:55:41 -0700
From: Douglas Otis <[email-masked]>
To: "Carl S. Gutekunst" <[email-masked]>
CC: Keith Moore <[email-masked]>, "Storz, Michael" <[email-masked]>, "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 10/5/11 11:47 AM, Carl S. Gutekunst wrote:
>
> Keith Moore wrote:
>> Just imagine how many wrongly rejected emails aren't reported.
>>
>> Stupid spam filtering mechanisms are a DoS attack on email.
>
> The problem is nearly all of our anti-spam measures are empirical. We 
> all know a lot of people who swear by this, that, or the other check, 
> even if the only supporting evidence is that using that particular 
> mechanism cut down the number of complaints. And what works on one 
> stream fails miserably on another.
>
> FWIW, I last looked at this problem about three years ago. I 
> specifically wanted to know if some form of RDNS checks might be 
> useful in cutting down the load on the content-based spam filters. (I 
> was also checking effect and utilization of SPF and TLS.) Note that 
> the purpose here was not to improve the catch rate. This was on a 
> stream that was already filtered by a short-lived (60 minute) IP 
> reputation filter, so that would reduce the message count from some 
> known spam sources; and most of the recipients were business users.
>
> As I recall, my sample size was something around 10 million E-mails 
> from a single MX server in a load balanced cluster over a 24-hour 
> period. There was a weak correlation between spam and a simple 
> existence check for the PTR record. There was no correlation at all 
> for a stronger check, e.g., A record matches; a message that failed 
> the strong check was as likely to be judged ham as spam by the content 
> filter.
Some will decide not to accept IPv6 connection.  The size of this space 
(~56,000 x 2^32) prefixes growing exponentially inhibits effective 
address management.  It seems this number is likely to flatten out at 
about 340,000 x 2^32 prefixes.  IMHO, the only method that might scale 
would use something like DANE.  Use of reverse DNS is simply nonsensical 
at these numbers.  What is worse, Dual-Stack Lite is likely to expose 
IPv4 only MTAs to IPv6 through Dual-Stack Lite NAPT or 6rd.  In which 
case, what would a reverse pointer offer within private address space?

-Doug

Date: Wed, 05 Oct 2011 11:47:58 -0700
From: "Carl S. Gutekunst" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: "Storz, Michael" <[email-masked]>, "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

Keith Moore wrote:
> Just imagine how many wrongly rejected emails aren't reported.
>
> Stupid spam filtering mechanisms are a DoS attack on email.

The problem is nearly all of our anti-spam measures are empirical. We 
all know a lot of people who swear by this, that, or the other check, 
even if the only supporting evidence is that using that particular 
mechanism cut down the number of complaints. And what works on one 
stream fails miserably on another.

FWIW, I last looked at this problem about three years ago. I 
specifically wanted to know if some form of RDNS checks might be useful 
in cutting down the load on the content-based spam filters. (I was also 
checking effect and utilization of SPF and TLS.) Note that the purpose 
here was not to improve the catch rate. This was on a stream that was 
already filtered by a short-lived (60 minute) IP reputation filter, so 
that would reduce the message count from some known spam sources; and 
most of the recipients were business users.

As I recall, my sample size was something around 10 million E-mails from 
a single MX server in a load balanced cluster over a 24-hour period. 
There was a weak correlation between spam and a simple existence check 
for the PTR record. There was no correlation at all for a stronger 
check, e.g., A record matches; a message that failed the strong check 
was as likely to be judged ham as spam by the content filter.

<csg>

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: Keith Moore <[email-masked]>
Date: Wed, 5 Oct 2011 14:22:32 -0400
Cc: SMTP Interest Group <[email-masked]>
To: Alessandro Vesely <[email-masked]>
Sender: [email-masked]

On Oct 5, 2011, at 1:49 PM, Alessandro Vesely wrote:

>> IMO, any time you're basing an abuse test on something that is
>> fundamentally irrelevant, it's of short-term value at best.  rDNS
>> is such a check.
> 
> The most relevant obstacle seems to be caused by uncooperative network
> providers that don't set PTR RRs.

how about the uncooperative spam filters that reject legitimate mail for no good reason, like this one?

Keith

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: Keith Moore <[email-masked]>
Date: Wed, 5 Oct 2011 14:18:56 -0400
Cc: "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
To: "Storz, Michael" <[email-masked]>
Sender: [email-masked]

> 
> in which world do you live? Many if not most of the major ISPs reject mail servers which do not have a PTR record. From the view of a customer this makes a mail server without a PTR record wrongly configured, even if you do not like this.

mediocrity is excellence at pursuing the mean.   just because "major ISPs" reject mail for bad reasons is not a justification for others to do so.
>> 
>> Stupid spam filtering mechanisms are a DoS attack on email.
> 
> Right, but this mechanism is not stupid, instead it is clever ;-)

nope.  it makes no sense whatsoever.  it is using a completely irrelevant test to decide whether mail is legitimate.  and it's easily defeated by spammers.  it's not only a complete waste of time, it's worse in that it causes legitimate mail to be dropped.

rejecting mail for this reason should be a criminal offense.

Keith

Date: Wed, 05 Oct 2011 19:49:36 +0200
From: Alessandro Vesely <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 05/Oct/11 16:35, Keith Moore wrote:
> On Oct 5, 2011, at 10:23 AM, Murray S. Kucherawy wrote:
>>> -----Original Message-----
>>> From: mail.imc.org On Behalf Of Rosenwald, Jordan
>>> 
>>> True statement on both accounts.  I'd add to it, that invalid rDNS is
>>> still a viable anti-abuse mechanism.  There are plenty of compromised
>>> machine operators that are "asleep at the wheel."  Checking rDNS
>>> scrapes off that chaff.
>> 
>> I generally agree, but implementing it on my small site with only
>> a handful of users did lead to a number of false positive
>> complaints that then needed to be handled.
>> 
>> It's probably not a good idea to standardize such a practice
>> (yet?), but it does seem like a useful tool to have around.
> 
> IMO, any time you're basing an abuse test on something that is
> fundamentally irrelevant, it's of short-term value at best.  rDNS
> is such a check.

The most relevant obstacle seems to be caused by uncooperative network
providers that don't set PTR RRs.  Whether they do so for some savings
or because they are "asleep at the wheel", they can afford evading
that setup because there is low market pressure for it.

Mailbox providers never had to look for some specially cooperative
kind of ISP in order to set up a reliable MTA.  However, it seems that
there will be a relatively short period of time during which network
providers will mainly do IPv6, while MTAs will still need IPv4
addresses.  During such period, mailbox providers will have to look
for some special kind of ISP anyway.  After all, IPv4s scarceness
should increase their value, but such niche-market pressure will be
different.  Am I wrong?

Date: Wed, 05 Oct 2011 19:45:49 +0200
From: "Rolf E. Sonneveld" <[email-masked]>
To: Keith Moore <[email-masked]>
Cc: "Storz, Michael" <[email-masked]>, "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

On 10/5/11 6:24 PM, Keith Moore wrote:
> On Oct 5, 2011, at 11:28 AM, Storz, Michael wrote:
>
>> Another name for the iprev test is "Forward Confirmed reverse DNS" (FCrDNS). With Postfix you configure it with the two commands
>>
>>    reject_unknown_reverse_client_hostname
>>    reject_unknown_client_hostname
>>
>> We use this check since years as our first defense against botnet spam with great success. In the last 7 days we rejected emails for nearly 22.000.000 recipients. 49% did not have a PTR record, 29% did not have a matching A record. Therefore the FCrDNS was responsible for 78% of all rejections. This means your statement, that this check is not working, is definitely not true.
> This is a pretty ridiculous statement.  You use a dubious criterion to reject 78% of messages, and then you claim that because you did that, the check "works".
>
>> However you have to live with a moderately false positive rate. Before we implemented the check, we analyzed out traffic for 3 months and build an automatic whitelist with 4.000 wrongly configured MTAs.
> There's absolutely nothing "wrongly configured" about an MTA that doesn't have a PTR record.
>
>> Since the beginning of the check we get about 1-2 false positives per week reported by our users. This second whitelist has 230 entries at the moment. This means about 4% of the MTAs we accept emails from are wrongly configured. We can live with that.
> Just imagine how many wrongly rejected emails aren't reported.

Right. How many recipients know they should have received mail but 
didn't? And of those who notice, how many take the time/effort to report 
it? Multiply those percentages and you get an idea how many false 
positives you're missing.

>
> Stupid spam filtering mechanisms are a DoS attack on email.

+1

/rolf

From: "Storz, Michael" <[email-masked]>
To: Keith Moore <[email-masked]>
CC: "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
Subject: RE: The anti-abuse rDNS check that FTP gave up
Date: Wed, 5 Oct 2011 17:15:29 +0000
Accept-Language: de-DE, en-US
Sender: [email-masked]

> On Oct 5, 2011, at 11:28 AM, Storz, Michael wrote:
> 
> > Another name for the iprev test is "Forward Confirmed reverse DNS"
> (FCrDNS). With Postfix you configure it with the two commands
> >
> >   reject_unknown_reverse_client_hostname
> >   reject_unknown_client_hostname
> >
> > We use this check since years as our first defense against botnet
> spam with great success. In the last 7 days we rejected emails for
> nearly 22.000.000 recipients. 49% did not have a PTR record, 29% did
> not have a matching A record. Therefore the FCrDNS was responsible for
> 78% of all rejections. This means your statement, that this check is
> not working, is definitely not true.
> 
> This is a pretty ridiculous statement.  You use a dubious criterion to
> reject 78% of messages, and then you claim that because you did that,
> the check "works".

Read the email, the statement of Valdis was,

"so most of Vint Cerf's famous 140 million compromised machines have an rDNS entry, which means it's not that effective anymore"

If thiswould be true, the FCrDNS check should only reject a very low percentage of "compromised machines". My statistic proves that this is not the case and on the contrary it is rejecting 78% which is a very high number. Therefore I say the check (still) works. BTW, if we would use the Spamhaus PBL as the first criterion, the rejection rate would be nearly 70% and the FCrDNS check would go down to a 20% rejection rate.

> 
> > However you have to live with a moderately false positive rate.
> Before we implemented the check, we analyzed out traffic for 3 months
> and build an automatic whitelist with 4.000 wrongly configured MTAs.
> 
> There's absolutely nothing "wrongly configured" about an MTA that
> doesn't have a PTR record.
> 

Keith, 

in which world do you live? Many if not most of the major ISPs reject mail servers which do not have a PTR record. From the view of a customer this makes a mail server without a PTR record wrongly configured, even if you do not like this.

> > Since the beginning of the check we get about 1-2 false positives per
> week reported by our users. This second whitelist has 230 entries at
> the moment. This means about 4% of the MTAs we accept emails from are
> wrongly configured. We can live with that.
> 
> Just imagine how many wrongly rejected emails aren't reported.

If one of our user wants an email they are pretty fast in reporting a reject.

> 
> Stupid spam filtering mechanisms are a DoS attack on email.

Right, but this mechanism is not stupid, instead it is clever ;-) It does not relate to spam, it does not relate to IP reputation etc., it is just a configuration issue which in most cases can be corrected easily by the admin of the mail server. 

> 
> Keith
> 
> 

Michael

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: Keith Moore <[email-masked]>
Date: Wed, 5 Oct 2011 12:24:07 -0400
Cc: "[email-masked]" <[email-masked]>, SMTP Interest Group <[email-masked]>
To: "Storz, Michael" <[email-masked]>
Sender: [email-masked]

On Oct 5, 2011, at 11:28 AM, Storz, Michael wrote:

> Another name for the iprev test is "Forward Confirmed reverse DNS" (FCrDNS). With Postfix you configure it with the two commands
> 
>   reject_unknown_reverse_client_hostname
>   reject_unknown_client_hostname
> 
> We use this check since years as our first defense against botnet spam with great success. In the last 7 days we rejected emails for nearly 22.000.000 recipients. 49% did not have a PTR record, 29% did not have a matching A record. Therefore the FCrDNS was responsible for 78% of all rejections. This means your statement, that this check is not working, is definitely not true.

This is a pretty ridiculous statement.  You use a dubious criterion to reject 78% of messages, and then you claim that because you did that, the check "works".

> However you have to live with a moderately false positive rate. Before we implemented the check, we analyzed out traffic for 3 months and build an automatic whitelist with 4.000 wrongly configured MTAs.

There's absolutely nothing "wrongly configured" about an MTA that doesn't have a PTR record.

> Since the beginning of the check we get about 1-2 false positives per week reported by our users. This second whitelist has 230 entries at the moment. This means about 4% of the MTAs we accept emails from are wrongly configured. We can live with that.

Just imagine how many wrongly rejected emails aren't reported.

Stupid spam filtering mechanisms are a DoS attack on email.

Keith

From: "Storz, Michael" <[email-masked]>
To: "[email-masked]" <[email-masked]>
CC: SMTP Interest Group <[email-masked]>
Subject: RE: The anti-abuse rDNS check that FTP gave up
Date: Wed, 5 Oct 2011 15:28:08 +0000
Accept-Language: de-DE, en-US
Sender: [email-masked]

> On Fri, 23 Sep 2011 20:50:27 +0200, Alessandro Vesely said:
> 
> > Most SMTP servers duly lookup the client's IP and annotate the
> > resulting name as comment in Received fields.  However, I don't
> recall
> > denying SMTP access based on the "iprev" test (as RFC 5451 named it.)
> >  Was it ever � la mode to do so?
> 
> At one time, the net was still small enough that it was a safe
> assumption that
> if you got mail from an IP address that didn't have a valid rDNS, it
> was (a) a rare
> event because (b) a missing rDNS meant the provider was asleep at the
> wheel.
> 
> Now-a-days, most providers have automatic provisioning systems that
> assign rDNS
> for customer addresses, so most of Vint Cerf's famous 140 million
> compromised
> machines have an rDNS entry, which means it's not that effective
> anymore.
> 
> (What *is* used a lot today is 'rDNS looks like a customer
> cablemodem/adsl connection')

Another name for the iprev test is "Forward Confirmed reverse DNS" (FCrDNS). With Postfix you configure it with the two commands

   reject_unknown_reverse_client_hostname
   reject_unknown_client_hostname

We use this check since years as our first defense against botnet spam with great success. In the last 7 days we rejected emails for nearly 22.000.000 recipients. 49% did not have a PTR record, 29% did not have a matching A record. Therefore the FCrDNS was responsible for 78% of all rejections. This means your statement, that this check is not working, is definitely not true.

However you have to live with a moderately false positive rate. Before we implemented the check, we analyzed out traffic for 3 months and build an automatic whitelist with 4.000 wrongly configured MTAs. Since the beginning of the check we get about 1-2 false positives per week reported by our users. This second whitelist has 230 entries at the moment. This means about 4% of the MTAs we accept emails from are wrongly configured. We can live with that.

Michael

Date: Wed, 5 Oct 2011 16:08:34 +0100
From: Tony Finch <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
cc: SMTP Interest Group <[email-masked]>
Subject: RE: The anti-abuse rDNS check that FTP gave up
Sender: [email-masked]

Murray S. Kucherawy <[email-masked]> wrote:
>
> I generally agree, but implementing it on my small site with only a
> handful of users did lead to a number of false positive complaints that
> then needed to be handled.

I haven't deployed this check myself, but I understand that it is fairly
common for MXs to require the existence of reverse DNS but not require
that it is valid. For example AOL has done this for many years.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Lundy, Fastnet, Irish Sea, Shannon: Southwest veering west 5 to 7,
occasionally gale 8. Very rough or high in Shannon, otherwise moderate or
rough. Occasional rain then showers. Moderate or poor, becoming good.

Subject: Re: The anti-abuse rDNS check that FTP gave up
From: Keith Moore <[email-masked]>
Date: Wed, 5 Oct 2011 10:35:19 -0400
Cc: SMTP Interest Group <[email-masked]>
To: "Murray S. Kucherawy" <[email-masked]>
Sender: [email-masked]

On Oct 5, 2011, at 10:23 AM, Murray S. Kucherawy wrote:

> 
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Rosenwald, Jordan
>> Sent: Wednesday, October 05, 2011 6:40 AM
>> To: SMTP Interest Group
>> Subject: RE: The anti-abuse rDNS check that FTP gave up
>> 
>> True statement on both accounts.  I'd add to it, that invalid rDNS is
>> still a viable anti-abuse mechanism.  There are plenty of compromised
>> machine operators that are "asleep at the wheel."  Checking rDNS
>> scrapes off that chaff.
> 
> I generally agree, but implementing it on my small site with only a handful of users did lead to a number of false positive complaints that then needed to be handled.
> 
> It's probably not a good idea to standardize such a practice (yet?), but it does seem like a useful tool to have around.

IMO, any time you're basing an abuse test on something that is fundamentally irrelevant, it's of short-term value at best.  rDNS is such a check.

Keith

From: "Murray S. Kucherawy" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Date: Wed, 5 Oct 2011 07:23:50 -0700
Subject: RE: The anti-abuse rDNS check that FTP gave up
Accept-Language: en-US
acceptlanguage: en-US
Sender: [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Rosenwald, Jordan
> Sent: Wednesday, October 05, 2011 6:40 AM
> To: SMTP Interest Group
> Subject: RE: The anti-abuse rDNS check that FTP gave up
> 
> True statement on both accounts.  I'd add to it, that invalid rDNS is
> still a viable anti-abuse mechanism.  There are plenty of compromised
> machine operators that are "asleep at the wheel."  Checking rDNS
> scrapes off that chaff.

I generally agree, but implementing it on my small site with only a handful of users did lead to a number of false positive complaints that then needed to be handled.

It's probably not a good idea to standardize such a practice (yet?), but it does seem like a useful tool to have around.

From: "Rosenwald, Jordan" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: RE: The anti-abuse rDNS check that FTP gave up
Date: Wed, 5 Oct 2011 13:39:32 +0000
Accept-Language: en-US
Sender: [email-masked]

> 
> On Fri, 23 Sep 2011 20:50:27 +0200, Alessandro Vesely said:
> 
> > Most SMTP servers duly lookup the client's IP and annotate the
> > resulting name as comment in Received fields.  However, I don't recall
> > denying SMTP access based on the "iprev" test (as RFC 5451 named it.)
> > Was it ever � la mode to do so?
> 
> At one time, the net was still small enough that it was a safe assumption that
> if you got mail from an IP address that didn't have a valid rDNS, it was (a) a
> rare event because (b) a missing rDNS meant the provider was asleep at the
> wheel.
> 
> Now-a-days, most providers have automatic provisioning systems that assign
> rDNS for customer addresses, so most of Vint Cerf's famous 140 million
> compromised machines have an rDNS entry, which means it's not that
> effective anymore.
> 
> (What *is* used a lot today is 'rDNS looks like a customer cablemodem/adsl
> connection')

True statement on both accounts.  I'd add to it, that invalid rDNS is still a viable anti-abuse mechanism.  There are plenty of compromised machine operators that are "asleep at the wheel."  Checking rDNS scrapes off that chaff.

To: Alessandro Vesely <[email-masked]>
Cc: SMTP Interest Group <[email-masked]>
Subject: Re: The anti-abuse rDNS check that FTP gave up
From: [email-masked]
Date: Wed, 05 Oct 2011 00:57:55 -0400
Sender: [email-masked]

--==_Exmh_1317790675_4646P
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, 23 Sep 2011 20:50:27 +0200, Alessandro Vesely said:

> Most SMTP servers duly lookup the client's IP and annotate the
> resulting name as comment in Received fields.  However, I don't recall
> denying SMTP access based on the =22iprev=22 test (as RFC 5451 named it=
.)
>  Was it ever =E0 la mode to do so?

At one time, the net was still small enough that it was a safe assumption=
 that
if you got mail from an IP address that didn't have a valid rDNS, it was =
(a) a rare
event because (b) a missing rDNS meant the provider was asleep at the whe=
el.

Now-a-days, most providers have automatic provisioning systems that assig=
n rDNS
for customer addresses, so most of Vint Cerf's famous 140 million comprom=
ised
machines have an rDNS entry, which means it's not that effective anymore.=

(What *is* used a lot today is 'rDNS looks like a customer cablemodem/ads=
l connection')

--==_Exmh_1317790675_4646P
Content-Type: application/pgp-signature

--==_Exmh_1317790675_4646P--