Subject: [ietf-smtp] SMTP Greylisting Retry Hints

From: Hector Santos <[email-masked]>
Date: Tue, 05 Feb 2019 11:11:23 -0500

Which smtp servers and client follow or honor Greylist SMTP retry hint 
for rescheduling retries?

https://tools.ietf.org/html/draft-santos-smtpgrey-02

I wrote this draft a while ago, wondering if any software beside mine, 
Wildcat! SMTP, supports it?

It has been every efficient and it has helped minimized the retry 
attempts for delivery of mail to 2 with the shortest period.

-- 
HLS

---

From: Claus Assmann <[email-masked]>
Date: Tue, 5 Feb 2019 08:24:36 -0800

On Tue, Feb 05, 2019, Hector Santos wrote:
> Which smtp servers and client follow or honor Greylist SMTP retry hint for
> rescheduling retries?

MeTA1 issues the retry time (server),
the client doesn't use it yet.

Quoting ChangeLog:

2018-03-17
Add a retry=delay part to a graylisting reply, e.g.,
421 4.7.0 Come back later. retry=123
(see also draft-santos-smtpgrey-02)

---

From: Paul Smith <[email-masked]>
Date: Tue, 5 Feb 2019 16:31:11 +0000

> for rescheduling retries?

Well, I thought our server indicated the retry time, but it appears that 
we were using a format from a previous discussion about it, not the 
proper draft, so we'll fix the return format to match the current draft.

The client doesn't use the hint though.

(To be honest, I thought the idea had disappeared, so we returned the 
retry time because that was a trivial change, but actually using the 
hint is a bit more complicated so we were waiting to see if it would be 
useful).

-- 

Paul Smith Computer Services
Tel: 01484 855800
Vat No: GB 685 6987 53

Sign up for news & updates at http://www.pscs.co.uk/go/subscribe

---

From: Hector Santos <[email-masked]>
Date: Tue, 05 Feb 2019 12:59:40 -0500

On 2/5/2019 11:31 AM, Paul Smith wrote:
>> Which smtp servers and client follow or honor Greylist SMTP retry
>> hint for rescheduling retries?
>
> Well, I thought our server indicated the retry time, but it appears
> that we were using a format from a previous discussion about it, not
> the proper draft, so we'll fix the return format to match the current
> draft.
>
> hint is a bit more complicated so we were waiting to see if it would
> be useful).

As far as minimizing the impact of greylisting delays, it has proven 
to be very useful over the years.

Server side:

The default has been set to 55 seconds.  My client will honor it and 
works as expected among similar setups.  Many other clients will try 
immediate within the 55 secs and of course, they are blocked. 
Yesterday, I was exploring signing up with InstaCart and I was waiting 
for their (confirmation/acknowledgement) email.  It got in at the 3rd 
attempts, 40, then 90 seconds apart.  Hence why I posted this to see 
if modern services are learning from the greylisting delays.

Client Side:

Yes, you have to adjust your client retry queuing/timing/schedule 
logic.  The SMTP client code would check for any possible server 4yz 
"retry hints" and extracts/returns the recommended time/delta.  The 
optional delta is then passed your scheduler and used to adjust/set 
the next retry.  It does work very work to minimize the retries to 2 
with the minimum time.

If it helps, I have this C code (port as required for *nix) with the 
functions used to extract the time from a response string checking the 
learned common strings seen:

// smtpgrey-example.cpp

#include <stdio.h>
#include <windows.h>

//-----------------------------------------------------------
// int GetRetrySecs(const char *psz)
//
// return # of seconds from formats
//
//  [DD-]HH:MM:SS  formal syntax
//  # seconds
//  # minutes
//  # hours
//
//-----------------------------------------------------------

int GetRetrySecs(const char *psz)
{
     char *pOut = NULL;
     int days = 0;
     int secs = strtol(psz,&pOut,10);
     // check for formal syntax [DD-]HH:MM:SS
     if (*pOut == '-') {
        days = secs;
        secs = strtol(pOut+1,&pOut,10);
     }
     if (*pOut == ':') {
        secs = days*24*3600+secs*3600+strtol(pOut+1,&pOut,10)*60;
        if (*pOut == ':') secs += atoi(pOut+1);
        return secs;
     }
     // check for informal syntax
     if (secs) {
        if (*pOut == ' ') pOut++;
        switch (*pOut) {
          case 's': break;
          case 'm': secs=secs*60; break;
          case 'h': secs=secs*3600; break;
        }
     }
     return secs;
}

//-----------------------------------------------------------
// int ExtractRetryHint(const char *psz)
//
// For given SMTP 4yz response, return any known retry time
// hint in seconds. Return 0 for no time hint or non 4yz
// response.  The routine checks for the formal syntax and
// for non-formal existing syntax:
//
//  retry=[DD-]HH:MM:SS                        proposed draft std
//  greylisted for # seconds|minutes|hours     common, non-std
//  try again in # seconds|minutes|hours       common, non-std
//  please come back in HH:MM::SS              common, non-std
//-----------------------------------------------------------

int ExtractRetryHint(const char *psz)
{
     int nsecs = 0;
     if (!psz || !psz[0] || ((atoi(psz)/100*100) != 400)) {
         return nsecs;
     }
     const char *tags[] = {
         "retry=",
         "greylisted for ",
         "try again in ",
         "please come back in ",
         NULL};
     const char **p = tags;
     char *lpsz = _strlwr(_strdup(psz));
     while (*p) {
       char *pos = strstr(lpsz,*p);
       if (pos) {
          nsecs = GetRetrySecs(pos+strlen(*p));
          break;
       }
       p++;
     }
     free(lpsz);
     return nsecs;
}

void test(const char *sz)
{
     int secs = ExtractRetryHint(sz);
     printf("secs: %5d | %s\n",secs, sz);
}

void main(char argc, char *argv[])
{
     test("421 This server implements greylisting, please try again in 
90 seconds");
     test("450 4.7.1 <RCPT>: Recipient address rejected: Greylisted 
for 5 minutes");
     test("450 4.7.1 <RCPT>: Recipient address rejected: Greylisted 
for 120 seconds");
     test("451 4.7.1 Greylisting in action, please come back in 
00:30:00");
     test("451 4.7.1 Greylisting in action, PLEASE COME BACK in 
01:05:20");
     test("451 Greylisted for 55 seconds");
     test("451 Greylisted, please try again in 167 seconds");
     test("451 Greylisting enabled, try again in 22 minutes");
     test("451 Greylisting. Try again later.");
     test("451 Greylisting. Try again later. retry=5 mins");
     test("471 Greylisting. Try again later. retry=300 secs");
     test("471 Greylisting. Try again later. retry=2 hours");
     test("471 Greylisting. Try again later. retry=2h");
     test("471 Greylisting. Try again later. retry=120");
     test("471 Greylisting. Try again later. retry=01:01:02");
     test("471 retry=01:01:02 Greylisting. Try again later.");
     test("471 Greylist: RETRY=1M Try again later.");
     test("550 Permanent Reject");
}

-- 
HLS

---

From: Дилян Палаузов <[email-masked]>
Date: Tue, 05 Feb 2019 21:03:40 +0000

Hello Hector,

I have not read your proposal, but I guess its purpose is to communicate to the other party (smpt-client) when to retry
transmission, in order to mitigate to minimum the delays from grey listing.  This is great, but there is one more
application of the proposed extension, that is not called “grey listing”:

When a mail for many recipients is send and different recipients have different accept/reject policies, PRDR is the non-
plus-ultra.  When the SMTP parties do not agree on PRDR, the server can accept the email for the first recipient and all
other recipients of the mail, that have the same accept/reject policy as the first recipient.  All other RCPT TO:s are
told to retry later *immediately*.

This enables, to apply different accept/delay policies per email recipient without having PRDR and without causing
noticeble delays for any recipient.

Consider updating the “Introduction” and “Abstract” of your draft accordingly.

Regards
  Дилян

---

From: Hector Santos <[email-masked]>
Date: Wed, 06 Feb 2019 10:23:07 -0500

Hi,

This draft is nothing more than an attempt to "IETF formalize" the 
functional and technical specification for an existing "technology" 
used for SMTP, and hopefully agree on the "retry hint" 4yz response 
string format to assist servers and clients in minimizing delivery 
delays.  Adding more to it was not the goal.

But how does Multiple RCPT TO fields apply in a transaction?

The typical Greylist "Triplet" are the three SMTP process parameters; 
the connecting IP, the Return Path (MAIL FROM) and the forwarding 
address (RCPT). Together, they make up the hash index to the greylist 
database.

Offhand, I would have to see how current code works with the idea of 
multiple triplets per transaction, one for each RCPT TO received. 
For our system, the GreyList Hook, is done at the DATA level where a 
bunch of p-code filters are run in sequence.  The Greylist hook grabs 
the first RCPT TO, I believe, but one key reason to do this at the 
DATA level to allow 3rd party developers a p-code reprogramming chance 
to add the Message-ID to the "triplet" as some GreyList systems are 
known to do, maybe in place of the RCPT and the possibility of having 
multiple addresses.  I think that makes technical sense.  But there 
are greylist systems who run immediately at the RCPT TO state and 
don't have the payload to work with.

Another point to consider, even if you did a GreyList response per 
RCPT TO and recorded multiple triplets,  I am not sure how clients 
will deal with this.  Will they retry the 4yz responses and skip the 
250 and 5yz responses?

Another point to consider is that Greylisting addresses anonymous 
senders, i.e. you don't know who they are, don't have info on them, 
not white listed, etc.  So sender authorization may be a local policy 
consideration in allowing multiple RCPT TO per transaction which would 
preempt greylisting and other filters a system may have.  For example, 
in our package, if the connecting client/sender is authorized either 
by IP or ESMTP AUTH, most restrictions are removed and most 
hooks/shims filters are pretty much skipped, i.e. RBL, SPF, CBV and 
Greylisting.

Thanks

-- 
HLS

---

From: Дилян Палаузов <[email-masked]>
Date: Wed, 06 Feb 2019 21:47:58 +0000

Hello Hector,

irrespective of SMTP transaction, grey-listing and IPaddr-MAILfrom-RCPTto triplet, describing more use cases for your
draft will increase its usefulness and more sites will implement it.

In any case, your draft shall state, if not included already, is that on

421|451 … retry=00:00:00

after RCPT TO: the client is supposed really soon to send another MAIL FROM: followed by that RCPT.

Like this:

Abstract: … Retry-time=00:00:00 can be send by a server, to accept emails for some recipients of an email, and reject it
for others.

Introduction/new subsection:
==Accepting the email for some recipients and rejecting it for others==
In order to avoid backscatters, or mitigate the effects of false positives, a site can reject suspicious emails during
the SMTP dialog, depending on recipient’s preferences.  When an email to such a site has several recipients in the
envelope and on the mail host these recipients have different policy for rejecting emails, the site can accept the first
RCPT TO: for the first recipient and any other recipient, who shares the mail rejection policy with the first recipient.
After the DATA-DOT the email for those recipients is accepted or rejected.  For the other recipients, the site can
communicate to the sender, that it shall retry immediately with a new transaction.

Amendment to section 2.3.4 RCTO TO:
For the purposes of applying different mail reject policies for the different recipients, a server communicates to the
client to retry immediately a new transaction with the failed recipient.  

Seciton 3 SMTP Service Keyword
I see no advantages for the 250-GREYLISTS EHLO reply, as with or without it nothing changes.

Section 4 Examples:

I propose replacing serverdomain.com with servdom.example.com

add:

Example of a site, where the first and third recipients share the same rejection policy, and all other recipients share
a different mail rejection policy:

S: a.example.com, welcome ESMTP v2.0
C: EHLO mail.example.org
S: 250-HELP
C: MAIL FROM:<[email-masked]>
S: 250 User OK
C: RCPT TO:<[email-masked]>
S: 250 OK
C: RCPT TO:<[email-masked]>
S: 451 This recipient has a different mail rejection policy than the first one.  retry=00:00:00
C: RCPT TO:<[email-masked]>
S: 250 OK
C: RCPT TO:<[email-masked]>
S: 451 This recipient has a different mail rejecton policy than the first one. retry=00:00:00
C: RCPT TO:<[email-masked]>
S: 451 This recipient has a different mail rejecton policy than the first one. retry=00:00:01
C: QUIT
S: 221 Goodbye

Note, by using a different retry for the last recipient, the server hints to retry [email-masked] and [email-masked]
in a new transaction, and [email-masked] in another transaction.  The server can do so in order to signal, that 
[email-masked] and [email-masked] have a separate, common mail-rejection policy, while [email-masked] has again
another mail rejection policy.

> Another point to consider, even if you did a … response per RCPT TO …,  I am not sure how clients will deal with
this.  Will they retry the 4yz responses and skip the 250 and 5yz responses?

This is what I and clients do now.

Regards
  Дилян

---

From: [email-masked]
Date: Wed, 06 Feb 2019 19:36:27 -0500

> after RCPT TO: the client is supposed really soon to send another MAIL FROM: 

After 4 decades of seeing programmers do unexpected things with sloppy specs, I
have to ask.. :)

Is the retry permitted to happen *later in the same connection*? And if so, is a
new EHLO or RSET required before doing so?

(and yes I'm sure that the same spamware that doesn't wait for the 220 before
sending an EHLO will retry in the same connection given half a chance...)

---

From: Дилян Палаузов <[email-masked]>
Date: Thu, 07 Feb 2019 07:19:00 +0000

Hello Valdis,

On Wed, 2019-02-06 at 19:36 -0500, [email-masked] wrote:
> On Wed, 06 Feb 2019 21:47:58 +0000,   said:
> 
> > 421|451  retry=00:00:00

The purpose I am talking about is adding possibility for the same mail message with several recipienst, to accept the
message for some recipients and reject it for others.

In the above example, it does not really matter, if retry=00:00:00 or retry=00:00:02 is written.

Retrying the same RCPT TO: before doing DATA/MAIL FROM: will lead to the same 421/451 reply.  This might be different
for grey listing.

Doing RCPT TO: after EHLO/RSET/another MAIL TO will lead to different results (accepting after RCPT).

I do not propose in general to change SMTP anyhow.  retry=… is a hint that can accelerate the delivery.

Regards
  Дилян

---

From: Hector Santos <[email-masked]>
Date: Thu, 07 Feb 2019 10:23:41 -0500

Hi

The draft spec as written does not restrict multiple triplets designs.

If the Greylist server issues different retry hint times for RCPT TO 
address, then it should can be presumed it is designed to record 
different triplets, and with such a Greylist server, it is assuming 
clients are prepared to behave properly, as expected, to reschedule 
the delivery to different people based on the 4yz reply code.

However, the question is are they prepared to handle different reply 
hints "reply=" to reschedule at different times?

Note there is a complexity and uniqueness for SMTP outbound systems, 
the queuing, the loading, the scheduling logics, etc.  This is why the 
other commentators indicated only the server response "reply=" hint 
were added, but not the client side.  Its "easier" to set server 
responses. Client side adoption means code changes to advanced queuing 
logics.

Anyway, I think I can update the draft to add protocol notes that say:

     GreyListing Servers MAY issue multiple and different reply hints per
     RCPT TO in a transaction.

     Compliant Clients SHOULD be prepared to reschedule retries using
     different reply hint times per transaction when issuing multiple
     RCPT TO addresses.

I can add your example as well. A reply hint of zero basically tells 
the client, they can perform a normal reschedule.

Thanks

On 2/6/2019 4:47 PM, Дилян Палаузов wrote:
> _______________________________________________
> ietf-smtp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-smtp
>

-- 
HLS

---

From: [email-masked]
Date: Thu, 07 Feb 2019 11:30:05 -0500

On Thu, 07 Feb 2019 07:19:00 +0000, Дилян Палаузов said:

> Retrying the same RCPT TO: before doing DATA/MAIL FROM: will lead to the same
> 421/451 reply.  This might be different for grey listing.

> Doing RCPT TO: after EHLO/RSET/another MAIL TO will lead to different results

Are we sure that's the semantics we wanted?  After all, if the server was
willing to accept a retry later on in the same TCP connection, why didn't it
just go ahead and accept the mail the first time?

---

From: Dilyan Palauzov <[email-masked]>
Date: Thu, 07 Feb 2019 18:14:12 +0100

Hello Valdis,

imagine a mail with an empty Subject: for two recipients.  The first recipient does not want to receive emails without subject, the second recipient has no problem with this.

This is what happens at SMTP level:

EHLO xx
250 blah blah
MAIL FROM:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 Okay
RCPT TO:<[email-masked]>
421 Retry soon retry=00:00:00
DATA
Subject:
From: [email-masked]
Message-Id: 12a
To: [email-masked], [email-masked]
....
.
550  You forgot to fill the Subject!
MAIL FROM:<[email-masked]>
250 ok
RCPT TO:<[email-masked]>
250 okay
DATA
Subject:
From: [email-masked]
Message-Id: 12a
To: [email-masked], [email-masked]
....
.
250 accepted

>After all, if the server was

Because the server wants to accept the email only for the second recipient.  At the moment RCPT is communicated the server does not know yet, if all recipents are willing to accept the message.

Regards
  Дилян

On February 7, 2019 5:30:05 PM GMT+01:00, [email-masked] wrote:
>On Thu, 07 Feb 2019 07:19:00 +0000, ���������� ���������������� said:
>
>> Doing RCPT TO: after EHLO/RSET/another MAIL TO will lead to different
>results
>> (accepting after RCPT).
>

---

From: Paul Smith <[email-masked]>
Date: Thu, 7 Feb 2019 17:27:03 +0000

> 550  You forgot to fill the Subject!

How would the server know to give the two different responses to the 
RCPT TO?

Or does it just always temporarily reject any messages for 
[email-masked]', just in case?

--

---

From: Jeremy Harris <[email-masked]>
Date: Thu, 7 Feb 2019 17:46:56 +0000

> How would the server know to give the two different responses to the

It checks the rules needed for the recipient, and tmp-rejects any
after the first that need different rules.
-- 
Jeremy

---

From: [email-masked]
Date: Thu, 07 Feb 2019 12:47:26 -0500

> clients are prepared to behave properly, as expected, to reschedule 

I find the phrase "assuming clients are prepared to behave properly"
rather amusing when we're talking about a technique for dealing
with clients that are maliciously trying to misbehave.

>     RCPT TO in a transaction.

So I can end you a MAIL FROM and 1 RCPT TO, and get back two or
more *different* hints?

1) Why is this at all useful enough to be allowed? What's the use case?

2) Must the hints be on the same line of a 451, or are continuation lines
allowed/required? (I'm willing to bet that even in 2019, multi-line replies to a
RCPT are rare enough that some mail clients *still* get this wrong)

3) If the two hints are different, should the client use the first, the last,
the shortest,the longest, or treat it as "the client must reply inside
this window for the mail to be accepted"?

---

From: [email-masked]
Date: Thu, 07 Feb 2019 12:50:41 -0500

> [email-masked]', just in case?

It took me a while to figure out the intent was to *defer* any RCPT TO that you
planned to actually accept, but *accept* any that may or may not be acceptable,
so it can 550 it further down the line if needed. In this case, it knows that
'user2' will accept the message, so it defers it. It doesn't know if 'user1'
will accept it, so it 250's it, gets the message, and then either 250 or 550
after DATA as needed.

---

From: Jeremy Harris <[email-masked]>
Date: Thu, 7 Feb 2019 17:56:00 +0000

> So I can end you a MAIL FROM and 1 RCPT TO, and get back two or

No.  One hint per RPCT.
-- 
Cheers,
  Jeremy

---

From: [email-masked]
Date: Thu, 07 Feb 2019 13:03:01 -0500

> > So I can end you a MAIL FROM and 1 RCPT TO, and get back two or

Pay attention.. That's not what Hector's text actually says.  It specifically
says that multiple hints *per rcpt to* are allowed.  It it was one per rcpt,
it should say something like:

MAY issue a different hint for each RCPT TO received.

---

From: Dilyan Palauzov <[email-masked]>
Date: Thu, 07 Feb 2019 19:21:39 +0100

Hello Paul,

you want to know how the server knows, whether two distinct mail addresses, have the equivalent mai-rejection-policy.

There is no definitive answer.  The way I do it, a system where all users share the same spam training, if two addresses share the Sieve script, then they share the mail rejection policy.  The handling of all other RCPT handling is postponed.

Strictly speaking, the bytewise equivalent Sieve script for two recipients does not imply equivalent mail-rejection-policy for both recipients: an IF ENVELOPE "to" test can cause different mail rejection policy, but in practice this is not relevant.

Otherwise, always postponing the handling of all recipients, except the first one, is a possible approach.

Regards
  Дилян

On February 7, 2019 6:27:03 PM GMT+01:00, Paul Smith <[email-masked]> wrote:
>> imagine a mail with an empty Subject: for two recipients.  The first
>recipient does not want to receive emails without subject, the second
>recipient has no problem with this.
>>

---

From: "John Levine" <[email-masked]>
Date: 7 Feb 2019 14:09:08 -0500

In article <[email-masked]> you write:
>imagine a mail with an empty Subject: for two recipients.  The first recipient does not want to receive
>emails without subject, the second recipient has no problem with this.

This issue, separate rejections after DATA, has come up many many
times before.  I've seen proposals to add new status codes after
the data with per-address reports, but they never went anywhere.

The only standard compliant way to accept some addresses and reject
others is to accept the message and send a bounce about the addresses
that didn't work.  Trying to do that with greylisting could never work
very well because you cannot reliably tell when a later message is the
same one you greylisted before.  In particular, they often come from
different IPs.

We realize that in the current cruddy mail environment it is fairly likely
that the bounce address is a fake and a bounce will go back to soemone
who didn't send the message, but that's another issue we've been looking
at without much progress.

R's,
John

---

From: Dilyan Palauzov <[email-masked]>
Date: Thu, 07 Feb 2019 23:20:24 +0100

Hello John,

the problem with bounces is, that they can lead to blacklisting the server, that generated them.

What I drafted, which has nothing to do with grey listing, is not necessary non-standard and avoids the problem with backscatters.  The “retry=” hint mitigates the side effect of delayed arrival.

Strictly speaking the delivery for all recipients does not have to be delayed.  The server can deliver promptly the messages to all deserving recipients, memorize the Message-Id and the per recipient response, and in subsequent transactions communicate the status per recipient.  The only side effect is then the “Mail not delivered within four hours” status message returned to the sender.

The proposals with separate responses per recipient after data (PRDR) are stuck in a sitation, where there is hardly any software capable on deciding on rejection per recipient. (vicious circle)

Regards
  Дилян

On February 7, 2019 8:09:08 PM GMT+01:00, John Levine <[email-masked]> wrote:
>recipient does not want to receive

---

From: Dilyan Palauzov <[email-masked]>
Date: Thu, 07 Feb 2019 23:32:32 +0100

Hello Valdis,

the intent is not to accept more recipients at RCPT TO:, which can disagree on the overall acceptance/rejection after DATA.

> In this case, it knows that 'user2' will accept the message, so it defers it.

No. In this case it does not know whether 'user2' will accept the message, so it defers it.

Or more correct reading:
In this case it does not know whether 'user2' will have the same opinion as 'user1' on accepting/rejecting the message, so it defers 'user2'.

Regards
  Дилян

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 8 Feb 2019 09:43:38 +0100

> server, that generated them.

Right. Bouncing spam is never a good idea, IMHO.

> What I drafted, which has nothing to do with grey listing, is not
> necessary non-standard and avoids the problem with backscatters.  The
> “retry=” hint mitigates the side effect of delayed arrival.

Yes, the retry hint can mitigate this. I've implemented a similar scheme
in 2005. I thought about delaying all recipients except the first but it
was impractical. The RFC allows up to 100 recipients and at the time
this was actually quite common. With typical retry times between 15
minutes and 2 hours, the last recipient would have gotten the mail
between 25 and 200 hours later (and probably not at all in the latter
case since 200 hours is longer than the typical queue timeout). A retry
hint "try again in 2 seconds" would have made this practical *if* it had
existed *and* been widely implemented. 

What I ended up doing was implementing a heuristic to guess which
recipients would produce the same result and delaying only those which I
expected to differ from the first one. Ugly, but worked actually quite
well. 

        hp

-- 
   _  | Peter J. Holzer    | we build much bigger, better disasters now
|_|_) |                    | because we have much more sophisticated
| |   | [email-masked]         | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson <https://www.edge.org/>

---

From: Paul Smith <[email-masked]>
Date: Fri, 8 Feb 2019 09:37:56 +0000

On 07/02/2019 22:20, Dilyan Palauzov wrote:
> Strictly speaking the delivery for all recipients does not have to be delayed.  The server can deliver promptly the messages to all deserving recipients, memorize the Message-Id and the per recipient response, and in subsequent transactions communicate the status per recipient.

So, you're suggesting delivering the message to the recipient, even 
though you've said it's been rejected?

That's not standards compliant... You don't KNOW that the sender is 
going to retry it, and if it does, you don't know if it'll use the same 
Message-Id, so the recipient may receive a message that the sender 
thinks they didn't, or they may receive multiple duplicates, or 
something else. You can't deliver it until/unless you say you'll deliver it.

The problems with the idea of 4xx rejecting recipients with different 
rules are that you really don't know what the sender is going to do, you 
are essentially removing the ability to send mail to multiple recipients 
at once, and you are adding load everywhere - a message sent to 100 
people at the same domain potentially has to be sent 100 times rather 
than just once (even if all recipients will accept it), and there have 
to be 5050 recipient checks rather than just 100, etc.

Is it that big a deal? If the recipient doesn't want the message, just 
accept it and quarantine it... That'll may even cut down on support 
calls - when a message is sent 'To: [email-masked], [email-masked]', 
and Kate receives the message, but Bob, who is sitting next to her, 
doesn't and has no idea why because you PRDR rejected it because of a 
rule that Bob turned on and didn't understand. (Or doesn't receive it 
until a few hours later (if at all, if the sender times-out), because he 
HASN'T got a *relevant* rule enabled, but does have some irrelevant 
slightly different setting from Kate).

--

---

From: Tony Finch <[email-masked]>
Date: Fri, 8 Feb 2019 12:33:25 +0000

John Levine <[email-masked]> wrote:
> that didn't work.

Another alternative is to split the envelope: if you know at RCPT TO time
that the addresses have different filtering policies which might lead to
unwanted bounces later on, 450 any subsequent addresses that have
stricter policies than the first one. The downside is that this can lead
to undesirable delays especially if the envelope is large.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
public services available on equal terms to all

---

From: "John R Levine" <[email-masked]>
Date: 8 Feb 2019 10:21:17 -0500

> to undesirable delays especially if the envelope is large.

How reliably does that work?  If I were writing an MTA I'd think a mix of 
2xx and 5xx is likely but 2xx and 4xx is unexpected.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Jeremy Harris <[email-masked]>
Date: Fri, 8 Feb 2019 15:32:24 +0000

On 08/02/2019 15:21, John R Levine wrote:
> How reliably does that work?  If I were writing an MTA I'd think a mix
> of 2xx and 5xx is likely but 2xx and 4xx is unexpected.

Works fine with Exim; I'd call your MTA nonconforming if
if it didn't handle it.
-- 
Cheers,
  Jeremy

---

From: Tony Finch <[email-masked]>
Date: Fri, 8 Feb 2019 15:40:49 +0000

> Works fine with Exim; I'd call your MTA nonconforming if

Yes.

There are situations where this can happen that are nothing to do with
disgusting filtering hacks :-) e.g. if the server is validating recipient
addresses and some of them depend on DNS lookups which fail.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Forties, Cromarty, Forth: South, veering west later, 6 to gale 8, perhaps
severe gale 9 later. Rough or very rough. Rain or showers. Good occasionally
poor.

---

From: Ned Freed <[email-masked]>
Date: Fri, 08 Feb 2019 07:33:24 -0800 (PST)

> 2xx and 5xx is likely but 2xx and 4xx is unexpected.

We've always referred to this as the good (2YZ), the bad (5YZ), and the ugly
(4YZ). It's a pain to handle because you have to reenqueue with a subset of
recipients, and the bookkeeping sounds like it should be straightforward, but
really isn't.

Nevertheless, the standards have effectively required that MTAs fully support it
since at least RFC 821, if not before. As I have noted many times, SMTP
just isn't that simple.

As for depending on other MTAs getting it right, while there are various ways
to get our MTA to provide a 2/4 or 2/4/5 mix of recipient responses, we don't
do so out of the box unless, say, an LDAP server failure occurs in the middle
of an SMTP session. (Extremely rare.) The few customers who have configured
things to return result mixtures involving temporary failures have generally
reported success, although there have been a few cases where it caused
problems.

All that said, when an SMTP client does get this wrong, it's not something that
easily observed from the vantage of the SMTP server returning the mixture of
results. You're basically depending on senders reporting the issue, which
almost certainly means problems are likely underreported.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 8 Feb 2019 17:37:49 +0100

On Friday 8 February 2019 16:33:24 CET, Ned Freed wrote:
> The few customers who have configured

I've seen it in practice, and I don't send that much mail, so I don't think 
this is obscure. Postfix has done it since 2.1, released in 2003.

http://www.postfix.org/postconf.5.html#reject_unverified_recipient

If you enable that on a secondary MX or SMTP-to-LMTP MTA, Postix will 
usually issue 250/550 but if the LMTP server or primary MX is unreachable, 
Postfix will issue a mixture of 250, 450 and 550.

Arnt

---

From: Dilyan Palauzov <[email-masked]>
Date: Fri, 08 Feb 2019 17:56:15 +0100

Hello Paul,

I am suggesting to tell the sender that the message for a recipient is temporary rejected, why delivering the message.

Why would an mta-client change the message-id of an email having several recipients on each retry?

Consider this example, I wrote to a friend an email stating “Tomorrow is my last day in this neighbourhood, I would enjoy meeting you once again before my departure”.  This email had empty Subject: and two weeks later the recipient told me, that she read only now the email, because it was delivered in the Spam folder (quarantined).  In this example, the software/AI/ML decided that there will be no meeting.  The real term is “false positive”.

To avoid side effects of false positives, somebody has to be involved in manually dealing with them.  You have to involve the recipient, the sender or special staff.

Obliging the recipient to deal with false positives, leads to a cancelled meeting in the example here and it “is nobody’s fault”.  Besides, when the recipient address is an alias for a mailbox on another provider, there is no way the aliasing server can communicate to the recipient the suspicious state.

Having special staff to deal with false positives is not an option in my case.

So the senders must be concerned with false positives, by hard rejecting the email, what I proposed.

There is no ideal way to go.  If the options are:
- good delivery for a server with a lot of aliases to other service providers, by ensuring good IP-repitation (implies no-bounces policy),
- no cancelled meetings or alike, because the software decided to speed down the communication flow by quarantining a message (false positives),
- mail delivery with unnecessary delays

you have to choose your priorities. Any choice has drawbacks. Arranging software towards avoiding the impacts of false positives (meetings, which do not happen) is justified.

Corollary: avoiding quarantines at any price, e.g. by rejecting emails with p=quarantine, when DMARC fails, can be justified depending on the priorities.

Regards
  Дилян

On February 8, 2019 10:37:56 AM GMT+01:00, Paul Smith <[email-masked]> wrote:
>> Strictly speaking the delivery for all recipients does not have to be
>delayed.  The server can deliver promptly the messages to all deserving
>recipients, memorize the Message-Id and the per recipient response, and
>in subsequent transactions communicate the status per recipient.
>

---

From: Paul Smith <[email-masked]>
Date: Fri, 8 Feb 2019 17:19:28 +0000

> Why would an mta-client change the message-id of an email having several recipients on each retry?

I don't know - but I've seen it. Basically, the client generates the 
Message-ID anew every time it tries to send it. It's not good, and 
arguably non-compliant, but it happens. With a 
'faked-defer-but-delivery-anyway' scheme, the recipient could end up 
with very many duplicates in that case.

Also, what if the send times-out, and the sender human retries? Or what 
if there was a mistake and the sender human cancels the send and sends a 
corrected message to the people who received it, but not to those who 
"didn't" (but actually did)? So, don't do a 'fake-defer'.

Real defer is fine, but could end up with very long delays if the 
message is being retried for each recipient in a big list of recipients. 
If you don't know how long the sender will wait between retries, you 
could end up with messages with over 20 or so recipients just "randomly" 
being delivered to some recipients and not others (because of timeouts), 
for no obvious reason to the sender or recipients.

Probably the best thing would be for the receiving server to be able to 
instruct the sender to degenerate to an 'only one RCPT TO per message' 
condition. It'd be inefficient (but possibly less so that an arbitrary 
mix of defers/rejects/accepts), but at least it'd be reliable, 
determinate and simpler.

> So the senders must be concerned with false positives, by hard rejecting the email, what I proposed.

Similarly, in my experience, human senders rarely look at bounce 
messages, and if they do, they rarely understand them or know how to 
react. So, rejecting a message (resulting in a bounce message) is at 
least as likely to end up with a message 'mysteriously' disappearing, as 
putting into a quarantine with a regular "quarantined message summary" 
being sent to the recipient.

--

---

From: Дилян Палаузов <[email-masked]>
Date: Fri, 08 Feb 2019 18:50:37 +0100

Hello Paul,

dynamic message-id for the same messsage means broken double-delivery-suppression anyway, when the message is delivered to the same mailbox many times over different paths.  Perhaps specifying that the message-id must be kept upon retrying after 421/451 in the next SMTP RFC edition will help.

As I acknowledged, any approach has drawbacks and at the end one has to weight the drawbacks.  I do not offer a panacea and in my case for a mail with many recipients, most recipients share the mail-rejection-policy.  So the delays appear only for the few others.

In long term PRDR will help, in middle term interpreting a terminating "retry=00:00:00" in the RCPT TO: reply for reusing the TCP connection will be an improvement.

If senders cannot read mails generated after RCPT TO 421/451 about delays, they will likely not be able to read fully-formatted pleasant boince messsages.

Regards
  Дилян

---

From: S Moonesamy <[email-masked]>
Date: Fri, 08 Feb 2019 09:55:10 -0800

Hi John,
At 07:21 AM 08-02-2019, John R Levine wrote:
>How reliably does that work?  If I were writing an MTA I'd think a 
>mix of 2xx and 5xx is likely but 2xx and 4xx is unexpected.

Both the 4xx and 5xx can occur.  The latter might be for a "mailbox 
not found" reply code.  The receiver might not want the sending MTA 
to retry in such a case.  The mix usually works.

Regards,
S. Moonesamy

---

From: Дилян Палаузов <[email-masked]>
Date: Fri, 08 Feb 2019 19:45:21 +0000

Some more thoughts:

What kind of emails have a lot of recipients?  Newsletters and spam.  When the spam delivery is postponed, then even
better, as there will be more knowledge about that mail in the DNSBL/DCC later.  If the email is a newsletter (MLM),
then its delivery is likely not urgent and the delivery statūs will be handled by the MLM or the corresponding
postmasters will do some “accelerations” to deal with the case, when a server accepts few RCPTs per try.

My “alternative” delivery mode will impact “Distributed Checksum Clearinghouse”, as the same mail will be hit once per
recipient on each transaction, not once per receiving server.

Here comes one more reason against fake temporary rejection: postponing the delivery, can evaluate an email as spam for
subsequent recipients, that was not evaluated as spam in the past for other recipienst.  (because whether a mail is
evaluated as spam is also a function of the time)

>mix of 2xx and 5xx is likely but 2xx and 4xx is unexpected.

If the some RCPTs have enough space in the mailbox they communicate 2xx.  If other RCPTs have currently not enough space
for new mails it can answer with 4xx in the same SMTP transaction. (https://tools.ietf.org/html/rfc1870 Section 6.4 Per-
recipient rejection based on message size)

> being sent to the recipient.

Not exactly.  Going back to the original example with the cancelled meeting, it is not anymore “nobody’s fault”: the
senders can do something once they are notified that RCPTs (permanently) rejected a message.  When the message (false
positive) is put in quarantine, meetings are cancelled and it is “nobody’s fault”.

On Fri, 2019-02-08 at 13:01 -0500, John Levine wrote:
> In article <[email-masked]> you write:
> I've seen it happen, in systems that regenerate the message from a database.
>

---

From: [email-masked]
Date: Fri, 08 Feb 2019 17:14:09 -0500

On Fri, 08 Feb 2019 19:45:21 +0000, Дилян Палаузов said:

> then its delivery is likely not urgent 

You'd be surprised.  In particular, emergency alert systems that try multiple media
(phone, SMS text, email, sirens, etc) because it's *really* *REALLY* important that
every recipient see it in one form or another, can generate a *lot* of emails that
need to go out *now*.

Google "April 16" for a worked example.

---

From: Hector Santos <[email-masked]>
Date: Fri, 08 Feb 2019 17:41:36 -0500

> MAY issue a different hint for each RCPT TO received.

Thanks for the correction. I did write "Multiple and Different" but of 
course, not what I meant. Just different per response.

But imagine a "Work vs After-work" or "Busy vs Non-Busy" hours retry 
load control thing:

   421 After=9am Before=5pm Retry=00:00:55; After=5pm Retry=00:02:00

Lets not go there, but the draft can also suggest to developers to use 
the first "reply hint" found.

-- 
HLS

---

From: "John Levine" <[email-masked]>
Date: 8 Feb 2019 17:46:44 -0500

>every recipient see it in one form or another, can generate a *lot* of emails that

Yes, but do they send one message with a zillion recipients or a zillion
messages with one recipient?

Most newsletters are now customized per recipient and everyone gets a
separate copy.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: [email-masked]
Date: Fri, 08 Feb 2019 18:15:11 -0500

> Most newsletters are now customized per recipient and everyone gets a

I'm specifically not talking about customized newsletters.

I'm talking "active shooter on the loose" notifications, where you need
to get thousands of identical messages out the door as fast as possible.

Basically, I'm pushing back on the notion that mass mailings are either
newsletters or spam.  There's other use cases...

(Yes, I know the mantra that email isn't instant messaging.  But when
all the cell phone towers just crashed under load, you end up hoping it's
close enough to make a difference....)

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sat, 9 Feb 2019 11:28:14 +0100

On 2019-02-08 17:56:15 +0100, Dilyan Palauzov wrote:
> So the senders must be concerned with false positives, by hard
> rejecting the email,

I agree with this (Although bounce messages are in practice hard to
understand by normal users and therefore often ignored).

> what I proposed.

But I don't think this is what you proposed. 

As I understood it, you proposed delivering a message while returning a
4xx reply to the sender. 

So the recpipient gets the message, but the sender might get a "your
message couldn't be delivered for 4 hours" notification even though it
was actually delivered. 

I think this would be very confusing.

It also doesn't help with reducing the number of delivery rounds since
the message Id isn't available at the time RCPT is received.

        hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sat, 9 Feb 2019 11:39:39 +0100

On 2019-02-08 19:45:21 +0000, Дилян Палаузов wrote:
> What kind of emails have a lot of recipients?  Newsletters and spam.

I disagree. 

Newsletters are almost always sent to individual subscribers, because
they contain personalized content (maybe only the envelope-from for
tracking bounces, but most likely URLs with unique Ids for tracking
clicks and maybe a personalized salutation).

Spam is (or was, last time I looked, may have changed again) also mostly
sent to one victim per message. I assume also mostly for tracxking
purposes.

What is still sent to many recipients are messages which are manually
adressed to many people - either because somebody manually enters the
recipients or - more likely - selects a group ("all employees",
"classmates", "people I share jokes with") from their address book.

        hp

---

From: Дилян Палаузов <[email-masked]>
Date: Sat, 09 Feb 2019 12:53:09 +0000

Hello,

back to draft-santos-smtpgrey-02, in section 2.3.5 DATA the draft says:

„A GreyListing Server rejecting at the DATA may be recording more information besides the triplet information,
i.e.  Message-Id header.”

The discussion here concluded, that tracking the Message-Id after DATA End of DATA and a 451 reply is not reliable means
to achieve anything.

In section 2.4.  4yz Format Structure > Examples: consider adding an example with totalsecs, like
451 4.7.1 Retry after end-of-data, possibly in this TCP connection. retry=0

My reading of the example:
  421 Your connection is greylisted. Please try again later (retry=00:01:00)
is that it is not conformant to the specification, as retry= is not at the very end, a ) follows.

I propose inserting in the draft under section “3.  SMTP Service Keyword” the text:

When the service keyword 250-GREYLIST is announced, it indicates that the client SHALL keep the IP address on retries.

draft-santos-smtpgrey-02 is not clear, whether clients are supposed to honour the retry= hint, if the server has not
announced 250-GREYLIST keyword.  This gets tricky, when the server accepts one RCPT per transaction, announces retry=
for the remaining RCPTs and does not care, whether the IP address is reused by the client on retries.

draft-santos-smtpgrey-02 wants to insert in multiple line responses the retry= hint on the last line.  What will happen
with other SMTP enhancements/hints?  Can they also insist on being present on the last line?  Can they insist to be also
the very last text in the response?

“If a SMTP client is rejected by the Greylisting Server during the
   session, the client SHOULD NOT attempt to start a new transaction
   during the same session and SHOULD immediately issue a QUIT command
   to end the session. ”

I do not like this text, as it is against the idea of accepting one/few RCPT per transaction, and telling all the other
recipients “421 … retry=0”.

„If a SMTP server offers a "retry=time-delay" hint which results in a
   wasted 2nd attempt and requires additional attempts, the SMTP client
   MAY begin to ignore the server's "retry=time-delay" hint after the
   2nd wasted retry. ”

Likewise.  What means “wasted”?  If the amount of remaing recipients is reduced on each retry, then the SMPT client
shall not begin to ignore the server's retry= hint after the 2nd wasted retry.

On 7th February I wrote:

> Strictly speaking the delivery for all recipients does not have to be delayed.  The server can deliver promptly the
messages to all deserving recipients, memorize the Message-Id and the per recipient response, and in subsequent
transactions communicate the status per recipient.  The only side effect is then the “Mail not delivered within four
hours” status message returned to the sender.

After some subscribers stated, that on retries the Message-Id can be changed, I have not insisted anymore, that
delivering promptly to all recipients, while communicating to the sender on RCPT “421 Retry later” does work as
expected.  It is a matter of weigthing all drawbacks.  I proposed a no-bounces policy, where the senders are concerned
with false positives.  As a matter of fact, I have never implemented this “cheating”, what I implemeted now is replying
on subsequent RCPT TO with

451 4.7.1 Retry immediately, possibly in the same TCP connection, after DATA End of data; retry=0

while not returning 250-GREYLIST on EHLO.

> So the recepient gets the message, but the sender might get a "your

Absolutely!  False positives are also very confusing and whatever is done to fix one very confusing situation leads to
another very confusing situation.  If there is interest I can describe all confusing situations and what options there
are to choose the right very confusing situation for a server.

> the message Id isn't available at the time RCPT is received.

Sure.  It helps with reducing the side effects of false positives and the side effects of delayed email delivery.

In the very simple case, when there is one recipient, and the mail is put in quarantine/spam/false positive, the SMTP
protocol allows shifting the work on dealing with false positives from the recipint to the sender.  This is largely not
done nowadays.  The people who decide on this are, if I remember correctly, not subscribed on any IETF mailing list.

The PRDR-after-DATA proposal will make progress, if providers are willing to switch the work on dealing with spam
folders to the sender.  As user I want not to deal with Spam folder, and this switch makes it possible.

Regards
  Дилян

---

From: [email-masked]
Date: Sat, 09 Feb 2019 14:11:15 -0500

> When the service keyword 250-GREYLIST is announced, it indicates that the
> client SHALL keep the IP address on retries.

So riddle me this:  What then happens to an e-mail if it's next attempt is from
a different IP address?  (Quite possible if the outbound client is a cluster of
machines with a shared mail queue, or an HA pair that's fallen over to the other
IP address, or....)  Perhaps this only merits a SHOULD rather than a SHALL?

> I do not like this text, as it is against the idea of accepting one/few RCPT
> per transaction, and telling all the other recipients “421 … retry=0”.

Agreed on that - it means that if there are multiple mail items queued for
a delivery to a host that has been unreachable for some amount of time,
the client can't complete running the queue - even if the remaining messages
would not have been greylisted.

And if you were planning to greylist all the remaining messages, why aren't
we using a simple '250-GREYLIST retry=00:10:00' in the EHLO response?

> The PRDR-after-DATA proposal will make progress, if providers are willing to
> switch the work on dealing with spam folders to the sender.  As user I want not
> to deal with Spam folder, and this switch makes it possible.

And as a user, if the sender is doing the spam management, you *can't* deal
with your spam folder anymore.  Which will totally suck if the sender decides that
since the mail got greylisted, it must be spam - and it's a false positive on
an actually important mail like a bank notification.  Remember - spam folders
exist precisely because spam detection isn't perfect.

---

From: Дилян Палаузов <[email-masked]>
Date: Sat, 09 Feb 2019 23:38:15 +0100

Hello Valdis,

if on 250-GREYLIST on retries the IP address of the client is changed, then the Greylist-421-retry=00:00:10 puzzle is not solved and the time begins again from the beginning on each IP-client-address change.  This is not in the interest of the sender.

The Junk folder is:
- place for the mail provider to store real spam for the recipient; and
- place for the mail provider to store false positives (no spam) for the recipient; and at the same time
- UI for the user to train the spam filters by moving emails in and out.

When I wrote that as user I do not want to have a Spam mailbox, I meant that I do not want to have a place, where the provider mixes real spam with false positives.  I am willing to have means to train the spam filter and undo the training.

What will a sender do, once she sees that her email is spam?  Use the phone, fax or snail mail.  No software training in first place.

If a message is important and is false positive:
- if the recipient has to deal with it, the recipient has the special right to oversee it, forever;
- if the sender has to deal with it, there is no such special right.

In your example, if the bank cannot reach you/me, it just partially locks the account, forcing you/me to contact it and does in the meantime what is reasonable.

Pfui, new radical smtp reply code 600 Mail Was Delivered as Junk? The purpose is to keep both sender and recipient responsible for handling false positives.

Regards
  Дилян

On February 9, 2019 8:11:15 PM GMT+01:00, [email-masked] wrote:
>On Sat, 09 Feb 2019 12:53:09 +0000, ���������� ���������������� said:
>
>> I propose inserting in the draft under section ���3.  SMTP Service
>Keyword��� the text:
>
>> ���If a SMTP client is rejected by the Greylisting Server during the
>>   to end the session. ���
>> per transaction, and telling all the other recipients ���421 ���
>retry=0���.
>

---

From: [email-masked]
Date: Sat, 09 Feb 2019 19:39:34 -0500

On Sat, 09 Feb 2019 23:38:15 +0100, Дилян Палаузов said:
> if on 250-GREYLIST on retries the IP address of the client is changed, then
> the Greylist-421-retry=00:00:10 puzzle is not solved and the time begins again
> from the beginning on each IP-client-address change.  This is not in the
> interest of the sender.

As I mentioned, I don't think it rises to the level of a SHALL - that's usually
reserved for things where failure to do so causes a high risk of the protocol
failing to work properly.  Retrying the transaction from a different IP address
merely means that the greylist timer will be reset - that's well into SHOULD
territory.

> When I wrote that as user I do not want to have a Spam mailbox, I meant that
> I do not want to have a place, where the provider mixes real spam with false
> positives.  I am willing to have means to train the spam filter and undo the
> training.

So instead you mix delayed-delivery reports with spam in the sender's mailbox.
I'm not sure that's much of a win.

Also, how does the provider distinguish between real spam and false positives,
so that the user doesn't have to? (Hint - it's easy to get it right 95 to 98%
of the time.  100% is probably impossible to do, especially with the incomplete
knowledge the provider has. I suspect, but cannot prove, that 100% reliable
spam detection is isomorphic to the Turing Halting Problem, similar to how Fred
Cohens PhD thesis showed that malware detection is isomorphic to it)

As a data point, the only time I *almost* bit on a phish mail was from a
Latvian-language website.  That I actually had an account on.  In better
Latvian than mine - and I'm a native speaker.

> What will a sender do, once she sees that her email is spam?  Use the phone, 
> fax or snail mail.  No software training in first place.

You're making the assumption that the same users who aren't able to understand
a non-delivery report from the remote end  *will* be able to understand the
message-delayed report from their provider's end. 

> In your example, if the bank cannot reach you/me, it just partially locks the
> account, forcing you/me to contact it and does in the meantime what is reasonable.

And what exactly do you define as "*partially* locks"?

> Pfui, new radical smtp reply code 600 Mail Was Delivered as Junk? The purpose
> is to keep both sender and recipient responsible for handling false positives.

That's another case of the "say the message was delayed but deliver it anyhow"
mess we had earlier in this discussion....

---

From: Дилян Палаузов <[email-masked]>
Date: Sun, 10 Feb 2019 12:07:45 +0000

Hello Valdis,

I described a case, where an ordinary meeting invitation was delivered in the Junk folder, read in turn two weeks later
and because of this the meeting has not happened.  Please propose what can be done in order to avoid such things in the
future.

On Sat, 2019-02-09 at 19:39 -0500, [email-masked] wrote:
> On Sat, 09 Feb 2019 23:38:15 +0100,   said:

The provider cannot distinguish between false positives and spam. The user has to do this work.  The question is: which
user - the one who writes, or the one who reads.

Compared to DMARC, with p=quarantine the domain owner states, that the recipient has to deal with suspicious emails,
while with p=reject the domain owner states, that it wishes, that the senders deal with suspicious emails.  The
recipient is not bound to the preferences on the domain owner and can decide in all or in none case do the extra the
work.

This applies for spam, too.  What is wrong, if each recipient (mailbox owner) can decide, whether the recipient or the
sender has to deal with emails, evaluated by the provider as likely spam?

Currently this is not possible (done), and one of the reasons is that users are stupid.  Yes, if the users are stupid
and cannot agree on a meeting using emails, because design decisions made by the providers based on stupid users, there
is a vicious circe.

> message-delayed report from their provider's end. 

I speak about users who can understand both messages.  If a user cannot understand a message, ... you propose that this
message shall not be delivered in first place.

> And what exactly do you define as "*partially* locks"?

E.g. lock only the online banking interface, or lock your card for online payments, or do investigations in address
registers about new physical addresses.  This happened actually to me, when mailmen messed things and returned to the
bank physical envelopes as non deliverable.  This goes now out of SMTP domain.

> mess we had earlier in this discussion....

The purpose of the discussion is to avoid overseeing emails, that the provider classified as very likely spam, but which
messages are in fact ham.  I described problems in the way SMTP currently works (is applied) and proposed how these
problems can be approached.  My suggestions replace one problem with another problem.  Criticizing me, that I propose
new, problematic approaches, without seeing what gets better, leads nowhere.

Let’s see now how shifting the work on dealing with false positives from the recipients to the senders works in
practice, and if this can be constructive:

When I click on emails on this mailing list on Reply-All, my answers go sometimes directly to [email-masked] .  From
there I got on 7 Feb 2019 07:19:01 GMT, 6 Feb 2019 21:48:22 GMT, 5 Feb 2019 21:03:41 GMT non-delivery reports stating

  The original message was received at Thu, 7 Feb 2019 07:19:01 GMT
  from x2f3b179.dyn.telefonica.de [2.243.177.121]

     ----- The following addresses had permanent fatal errors -----
  <[email-masked]>
      (reason: 554 IP 144.76.142.78 Rejected By System Policy Filter)

     ----- Transcript of session follows -----
  ... while talking to mail.isdg.net.:
  <<< 554 IP 144.76.142.78 Rejected By System Policy Filter
  554 5.0.0 Service unavailable

Likewise, my answers are supposed to go sometimes to [email-masked], and these were returned on 7 Feb 2019 22:20:32
GMT, 6 Feb 2019 22:11:56 GMT, 5 Feb 2019 08:06:47 GMT  with:

The original message was received at Wed, 6 Feb 2019 22:11:56 GMT
from x5d877ef8.dyn.telefonica.de [93.135.126.248]

   ----- The following addresses had permanent fatal errors -----
<[email-masked]>
    (reason: 553 Blocked due to excessive spam. Contact [email-masked] from another network if it is fixed.
Requests without the specific IP address will be ignored.)

   ----- Transcript of session follows -----
... while talking to mx1.taugh.com.:
>>> RCPT To:<[email-masked]>
<<< 553 Blocked due to excessive spam. Contact [email-masked] from another network if it is fixed. Requests without
the specific IP address will be ignored.
550 5.1.1 <[email-masked]>... User unknown
>>> DATA
<<< 553 Blocked due to excessive spam. Contact [email-masked] from another network if it is fixed. Requests without
the specific IP address will be ignored.

What is the problem with the IP 144.76.142.78 or the emails as whole?  Is it better, if I am aware that the mails were
not delivered, rather than delivering the emails as junk and not reading them?

Obviously the answers of those quesions, the actions taken, consequences and alike, do not depend on having the
possibililty to apply different delivery policy per recipient.

Regards
  Дилян

---

From: Paul Smith <[email-masked]>
Date: Sun, 10 Feb 2019 14:25:14 +0000

> and because of this the meeting has not happened.  Please propose what can be done in order to avoid such things in the

If the meeting is important, then you need confirmation. Either ask the 
recipient for confirmation, or use another medium (eg telephone) where 
confirmation is implicit. If you send an email asking for confirmation 
and you don't receive confirmation, then you need to follow up the email 
in some way.

WHATEVER you do with email (just as with post, SMS, carrier pigeon, 
etc), if you don't ask for confirmation, then you don't know if the 
recipient has received the message (and agreed to the meeting) or not.

--

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sun, 10 Feb 2019 17:48:34 +0100

On 2019-02-09 19:39:34 -0500, [email-masked] wrote:
> On Sat, 09 Feb 2019 23:38:15 +0100, ?????????? ???????????????? said:
> failing to work properly.

I also think a SHALL is inappropriate here on purely formal grounds: An
extension is by definition optional. A client may or may not implement
it. Putting a mandatory requirement on a client which hasn't yet
indicated that it supports the extension is therefore impossible.

Just stating that keeping the values in the triple constant (or at least
restricting their range) will improve the chances of a speedy delivery
is IMHO preferrable to stating a specific requirement.

        hp

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sun, 10 Feb 2019 18:05:50 +0100

On 2019-02-10 12:07:45 +0000, Дилян Палаузов wrote:
> I described a case, where an ordinary meeting invitation was delivered
> in the Junk folder, read in turn two weeks later and because of this
> the meeting has not happened.  Please propose what can be done in
> order to avoid such things in the future.

Yes. And as far as I can see you haven't demonstrated how anything you
have proposed in this thread reduces the probability of this happening.

This is possibly because you have proposed for at least three or four
different mechanisms in this thread (not all of them related to the
subject) and I at least have a hard time keeping track of what you are
currently talking about and what it is supposed to accomplish.

It might help if you concentrate on this single scenario and describe in
detail what the sending person, the sender MTA, the recpient MTA and the
receiving person should do in you opinion and how it differs from what
is possible (or common) now.

> The provider cannot distinguish between false positives and spam. The
> user has to do this work.  The question is: which user - the one who
> writes, or the one who reads.

The one who writes will always claim that their message isn't spam.

> This applies for spam, too.  What is wrong, if each recipient (mailbox
> owner) can decide, whether the recipient or the sender has to deal
> with emails, evaluated by the provider as likely spam?

Nothing.

> Currently this is not possible (done),

Of course this is possible, and it is also done (we did until recently).
It just isn't very common.

> and one of the reasons is that users are stupid.

Not necessarily stupid, but generally not interested in the technical
details. Other reasons are that giving users more knobs to turn results
in more support calls (and hence higher cost), and that other
organization's software (which you have no control over) may not deal
with it gracefully.

        hp

---

From: Ned Freed <[email-masked]>
Date: Sun, 10 Feb 2019 09:48:09 -0800 (PST)

> extension is by definition optional.

That fact that an extension is optional doesn't mean that no requirements
have to be met in order to implement the extension properly. 

> A client may or may not implement

It's assumed that requirements given in an extension specification apply
to software that implements the extension in a compliant fashion.

				Ned

---

From: Hector Santos <[email-masked]>
Date: Sun, 10 Feb 2019 12:59:29 -0500

> would not have been greylisted.

I have not observed this in over 15+ years of Greylisting operations. 
The specific paragraph in SMTP-GREY, in whole, is:

    If a SMTP client is rejected by the Greylisting Server during the
    session, the client SHOULD NOT attempt to start a new transaction
    during the same session and SHOULD immediately issue a QUIT command
    to end the session.  Its been observed that some mail senders will
    hold the connection for 1-5 minutes and retry the same mail
    transaction or a new transaction.  The SMTP server rejecting the
    initial transaction MAY stop accepting any new transactions attempts
    during the same session.

This is based on real scenarios where unauthenticated, anonymous 
(unknown) senders try to bypass SMTP 5yz/4yz filter responses by 
restarting a new transaction, for example:

   C: MAIL FROM: <[email-masked]>
   S: 250 OK
   C: RCPT TO: <[email-masked]>
   S: 250 Local User and/or Local Host Ok
   C: DATA
   S: 354 Start mail input; end with <CRLF>.<CRLF>
   ...payload transferred, local filters spawned ...
   S: 451 Greylisted. Please try again later (TempFail) retry=00:00:55
   C: RSET
   S: 250 Reset State #1
   C: MAIL FROM:  <[email-masked])
   S: 451 Transactions disabled due to policy.

This was learned not from Greylisting but, in fact, from SPF and other 
SMTP rejection filters where the unknown client attempted a new 
transaction to bypass an earlier rejection.  In addition, there were 
clients (determined to be from mail services using a specific SMTP 
package) that allowed for holding a socket connection open while it 
waited for any new outbound mail to arrive and to be sent.  Without a 
client/server negotiated authorized contract, this was consdered to be 
CPU and load expensive and a high scaling overhead on the receiver 
side.  In fact, because of this, our SMTP package offers local policy 
options to change the SMTP wait time from 5 minutes to 1 min or less 
to help minimize these client hold times which normally do not send 
another transaction.

I agree, overall, this thread is about dealing with multiple 
recipients and also multiple transactions per session. So there should 
be information about this, but I think the protocol doesn't prohibit 
these design considerations. The above protocol note implies the 
Compliant SMTP-GREY Client should be aware a negative reply code COULD 
restrict a client from starting a new transaction and also apply other 
restrictions. i.e. lower wait times is a possibility.

Should I add this protocol note?

    Compliant Clients SHOULD be aware Greylisting Server MAY restrict
    multiple transactions, multiple recipients and wait times after
    a 4yz Greylist response has been issued.

It's all possible today, even without greylisting.  In today's era of 
advancing filtering methods, it could even be a general SMTP state 
machine note to consider, if not already implied in RFC5321.

> exist precisely because spam detection isn't perfect.

If I follow, I am not sure I agree, this is about driving the SMTP 
client retry scheduler by providing a wait time delta to the existing 
equation. Server controls using 4yz have long existed. What is 
different is the "reply=" hint.

-- 
HLS

---

From: Claus Assmann <[email-masked]>
Date: Sun, 10 Feb 2019 11:13:02 -0800

>   C: MAIL FROM: <[email-masked]>
                 ^

PLEASE use the correct syntax!

>   C: RCPT TO: <[email-masked]>
               ^

RFC 5321
      MAIL FROM:<reverse-path> [SP <mail-parameters> ] <CRLF>
...
      RCPT TO:<forward-path> [ SP <rcpt-parameters> ] <CRLF>
...

   Since it has been a common source of errors, it is worth noting that
   spaces are not permitted on either side of the colon following FROM
   in the MAIL command or TO in the RCPT command.  The syntax is exactly
   as given above.

---

From: Hector Santos <[email-masked]>
Date: Sun, 10 Feb 2019 14:38:16 -0500

> is possible (or common) now.

+1

It sounds to me he is looking for UI-controls here:

    "Hey {USER.NAME},

     You got a message from sender {RETURN-PATH} which is temporarily
     being blocked due to a system-wide GREYLIST policy.

     If this is a legit sender, you can do nothing and receive
     the message shortly when it is resent.

     If the mail is not resent, it was most likely, a spammer.

     If you wish to white list this sender now, you can click this
     link to avoid greylisting this sender in the future.  Note,
     it will not white list the sender for all users, just to you.

     If you know this is a Bad Guy, you can black list the sender
     right now. Note, it will not black list the sender for
     all users, just to you."

and all the variations of this including confusing IP data, etc, etc.

SMTP-GREY can apply before DATA and at DATA. If at DATA, it was always 
possible to show the user the message before it is "officially" SMTP 
accepted (during the block time).  In fact, when the radical idea of 
intentional greyListing clients was first implemented in 2003, there 
was an unsureness about this radical idea. This was especially true 
with many IETF folks -- no way were going to endorse this 
"Greylisting" SMTP concept.   That changed by 2012 when I wrote the 
draft.  But this was the main reason for capturing the payload at DATA 
in order to have gain operator confidence just in case legit mail not 
being retried and they could manually import it into the mail 
database. Of course, the proof of concept was shown: Good SMTP systems 
do retry per SMTP requirement, Bad SMTP systems do not and ince the 
confident was high with the concept, the only thing left was to make 
sure there was purging of a now high accumulation of payload files on 
a regular basis!

Over the years, there were times when I had a new service sign up and 
I know I will get be getting a confirmation email. So I will do one of 
the following:

   - Just wait until the retry is done and the mail is accepted.
     Not concern about time here.

   - I can't wait, a greylist occurs and I read the stored payload,
     get the URL confirmation link or  code to entry at some GUI.
     I don't worry about the official reception of the email
     eventually arriving.  Fortunately, systems have not connected the
     two ideas yet -- using an official email SMTP acceptance with
     a GUI confirmation process.

   - Anticipating the new service email address will be a common email
     and domain to be received, I'll add the domain to a white list.
     I don't like doing this white listing when it is not IP protected.

None of this is within the "scope" of this SMTP-GREY draft, in my opinion.

But of course, developers could and should keep those user vs 
system-wide policy ideas in mind when implementing or enhancing their 
existing greylist system. I don't think the draft prohibits these 
advanced SMTP-GREY implementations.

Advanced systems can do many things with a captured payload, including 
adding SPF/DKIM/DMARC into the Greylist equation and I suppose they 
can do things like the above notification at the UI level.

-- 
HLS

---

From: Hector Santos <[email-masked]>
Date: Sun, 10 Feb 2019 14:51:00 -0500

+1

I'm sure most folks here do follow the general idea of the "example." 
  But wow, my bad, you're technically 100% right. I hope not to repeat it.

It should also be noted that not all SMTP servers will be strict here 
since there is a huge history of clients using a space, especially 
with legacy clients.  Our SMTP server is relaxed here with such 
clients. If I recall, I think OE was one of them.  Was not going to 
reject OE clients on this basis.  I will be interested to know how 
your server behaved here? Strict or relax?

Thanks

-- 
HLS

---

From: Dave Crocker <[email-masked]>
Date: Sun, 10 Feb 2019 12:10:54 -0800

> to software that implements the extension in a compliant fashion.

This touches a profoundly simple and essential point:

      A specification defines a sandbox (or 4 walls, or any other for a 
realm that is distinguish from beyond that realm).

      There are rules or choices for deciding when to play in the 
sandbox.  The rules can be social, legal, personal, whatever.

      Separate from them are the rules for playing /in/ the sandbox.  A 
specification is for the latter.  Once you decide to play in the 
sandbox, the rules of the specification apply.

      Hence the semantic for normative language within a spec is 
predicated upon a decision to conform to the spec -- ie, to play in its 
sandbox.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Дилян Палаузов <[email-masked]>
Date: Sun, 10 Feb 2019 21:06:49 +0000

Hello,

> recipient has received the message (and agreed to the meeting) or not.

Irrespective of the importance of the meeting, if you ask for confirmation, the confirmation can be again false
positive.  Shall be there a confiramation, of the confirmation?  Where does the loop end?

If the recipients servers have rejected the message, I know it was not received.  Otherwise I want to assume the message
was properly delivered.

> currently talking about and what it is supposed to accomplish.

Argeed.  Let’s do things one by one:

In the simple case where a sender writes an email to a single recipient, and the server decides the email is spam, but
the sender and recipient (possibly) think that the mail is ham (false positive), the options are:
-- deliver the email in the Junk folder, after DATA say "250 It is my honour to accept your message"
-- reject the email at SMTP level after DATA, telling the sender in the DATA-dot reply that this email is not going to
be accepted, possibly ellaborating more on this.
-- communicate to the client:
  599-Your message was delivered in the spam folder of the recipient.  Tell the recipient 
  599-offline to mark your email as ham, so that future emails from you are delivered 
  599 directly in recipient's Inbox...

What is valid, what is not acceptable for a single recipient case?

Is it acceptable if the recipient can tell its provider, in case of doubt, for an email only to that recipient, that the
provider has to:
  - reject at SMTP level the message, possilby inserting in the reply text provided by the user (e.g. PO box, fax and
phone number);
  - accept the email in the spam folder;
  - accept the email in the spam folder and communicate over 599-after-DATA reply, that the the email was put in the
spam folder, including also text provided by the user?

Once this is clarified, the discussion will go for the cases, where the mail has more recipients.

Regards
  Дилян

---

From: "John Levine" <[email-masked]>
Date: 10 Feb 2019 16:31:09 -0500

>positive.  Shall be there a confiramation, of the confirmation?  Where does the loop end?

If I write to you asking if you're coming to the meeting, and you
write back saying you are, or write back saying you aren't, the loop
is done.  This is not complicated.

>If the recipients servers have rejected the message, I know it was not received.  Otherwise I want to assume the message was properly delivered.

As has been noted before, you can't always have what you want.

R's,
John

---

From: [email-masked]
Date: Sun, 10 Feb 2019 18:25:47 -0500

> is done.  This is not complicated.

I'm not sure if he's talking about the case where the reply gets stuck in a
spam/greylist. The TCP equivalent would be a lost ACK on a packet.

Or if he's talking about the case where you've invited Fred to the meeting,
Fred accepts/declines, but then *Fred* is unsure about whether his response
was received by your meeting software...

---

From: Claus Assmann <[email-masked]>
Date: Mon, 11 Feb 2019 04:26:54 -0800

On Sun, Feb 10, 2019, Hector Santos wrote:

> I will be interested to know how your server behaved here? Strict or relax?

We had this discussion back in 2005, e.g.,
Message-ID: <[email-masked]>

My server still rejects it as Syntax error.

---

From: Hector Santos <[email-masked]>
Date: Mon, 11 Feb 2019 09:11:01 -0500

Really?

It is sort of ridiculous to bring this again 13 years later knowing 
full well there are servers that are relaxed in this regard due the 
hard fact there are there were clients that did issue the space.

But I am happy it wasn't a problem for you.  I don't need the headache 
of dealing with that silly support issue. I prefer to be more 
pragmatic, democratic and relaxed in this specific regard.  No need to 
remove this relaxation.

Lets move on, Thanks

-- 
HLS