Subject: Re: [ietf-smtp] [Emailcore] Status of Greylisting (i'd wish MessageID were part of SMTP prologue)

From: "John Levine" <[email-masked]>
Date: 6 Jan 2022 17:06:59 -0500

It appears that Steffen Nurpmeso  <[email-masked]> said:
>These rotations surely have an impact on RFC 6647, 5.,
>
>   1.  Implement greylisting based on a tuple consisting of (IP address,
>       RFC5321.MailFrom, and the first RFC5321.RcptTo).
>
>that was not forseeable to this extent in 2012?

Item 5 in section 5 says:

       To accommodate those senders that have clusters of outgoing mail
       servers, greylisting servers MAY track CIDR blocks of a size of
       its own choosing, such as /24, rather than the full IPv4 address.
       (Note, however, that this heuristic will not work for clusters
       having machines on different networks.)  A similar grouping
       capability MAY be established based on the domain name of the
       mail server if one can be determined.

Is this the problem you are encountering or something else?

In my experience, allowing matches within a /24 in IPv4 or a /64
in IPv6 largely addresses this problem.

>How useful is greylisting on
>overall and/or on this scale today, and tomorrow?

My small system recently greylisted 21238 sending hosts of which 12745
retried and 8770 didn't. Once a host retries, it isn't greylisted
again unless it hasn't sent any mail for over a month. Spot checks
show that the 40% of hosts that don't retry are almost all spambots,
so it's useful, but not so much we'd change the protocol.

Apropos jck's question, while we might consider revising 6447, this has nothing
to do with 5321 so replies are directed to the ietf-smtp list.

R's,
John

---

From: Steffen Nurpmeso <[email-masked]>
Date: Fri, 07 Jan 2022 00:52:49 +0100

John Levine wrote in
 <[email-masked]>:
 |It appears that Steffen Nurpmeso  <[email-masked]> said:
 |>These rotations surely have an impact on RFC 6647, 5.,
 |>
 |>   1.  Implement greylisting based on a tuple consisting of (IP address,
 |>       RFC5321.MailFrom, and the first RFC5321.RcptTo).
 |>
 |>that was not forseeable to this extent in 2012?
 |
 |Item 5 in section 5 says:
 |
 |       To accommodate those senders that have clusters of outgoing mail
 |       servers, greylisting servers MAY track CIDR blocks of a size of
 |       its own choosing, such as /24, rather than the full IPv4 address.
 |       (Note, however, that this heuristic will not work for clusters
 |       having machines on different networks.)  A similar grouping
 |       capability MAY be established based on the domain name of the
 |       mail server if one can be determined.
 |
 |Is this the problem you are encountering or something else?

Yes that section from the RFC.

 |In my experience, allowing matches within a /24 in IPv4 or a /64
 |in IPv6 largely addresses this problem.
 |
 |>How useful is greylisting on
 |>overall and/or on this scale today, and tomorrow?
 |
 |My small system recently greylisted 21238 sending hosts of which 12745
 |retried and 8770 didn't. Once a host retries, it isn't greylisted
 |again unless it hasn't sent any mail for over a month. Spot checks
 |show that the 40% of hosts that don't retry are almost all spambots,
 |so it's useful, but not so much we'd change the protocol.

Below.

 |Apropos jck's question, while we might consider revising 6447, this \
 |has nothing
 |to do with 5321 so replies are directed to the ietf-smtp list.

I was not subscribed there.  Now i am (shall this thread
continue).

This is interesting, for example the Firefox browser i use only
can manage one password for all the IETF mailing-lists i am
subscribed too (iirc, many many months, but it tried to auto-fill
a false one now, and i definetely recall having problems with
password auto-fill for mailing-list subscriptions), so
auto-filling the password just does not work.

These are interesting numbers far beyond mine, thank you!  (It
surely will increase now that i post on @ietf.org, as always.)
Yes /24, not /8.  Really very interesting that /24 is of so much
use even today.  Many sites use multiple "deferred" until a retry
is accepted, and your "one month" white listing is also a number
quite large i think.
I find it interesting that such simple greylisting that cannot
even correctly identify a specific message seems so useful still.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

---

From: "John Levine" <[email-masked]>
Date: 6 Jan 2022 19:23:34 -0500

>auto-filling the password just does not work.

I believe you, but that has nothing whatsoever to do with SMTP or
the way that mail works.

>is accepted, and your "one month" white listing is also a number

It might as well be forever.  The only point of greylisting is
to see if an MTA follows the spec well enough to retry.  Once
you've seen a retry, greylisting is of no further use for that
source.  The only reason I time out after a month is to keep
the local whitelist database from getting too big.

>even correctly identify a specific message seems so useful still.

But identifying the specific message is not important. How often does
a spambot send two different messages with the same envelope in a
time gap that makes it look like a retry? Basically, never.

I have tried a version of greylisting that did the retry after DATA
and kept a hash of the message. I found that a certain number of
systems regenerate the retried message in a form that is not quite the
same, e.g., different timestamps. The hashes don't match and the
retries always failed.

R's,
John

---

From: Steffen Nurpmeso <[email-masked]>
Date: Fri, 07 Jan 2022 18:51:51 +0100

John Levine wrote in
 <[email-masked]>:
 |It appears that Steffen Nurpmeso  <[email-masked]> said:
 |>This is interesting, for example the Firefox browser i use only
 |>can manage one password for all the IETF mailing-lists i am
 |>subscribed too (iirc, many many months, but it tried to auto-fill
 |>a false one now, and i definetely recall having problems with
 |>password auto-fill for mailing-list subscriptions), so
 |>auto-filling the password just does not work.
 |
 |I believe you, but that has nothing whatsoever to do with SMTP or
 |the way that mail works.

It was only a ready to hand example of the importance of
identifiability (of course).

 |>These are interesting numbers far beyond mine, thank you!  (It
 |>surely will increase now that i post on @ietf.org, as always.)
 |>Yes /24, not /8.  Really very interesting that /24 is of so much
 |>use even today.  Many sites use multiple "deferred" until a retry
 |>is accepted, and your "one month" white listing is also a number
 |>quite large i think.
 |
 |It might as well be forever.  The only point of greylisting is
 |to see if an MTA follows the spec well enough to retry.  Once
 |you've seen a retry, greylisting is of no further use for that
 |source.  The only reason I time out after a month is to keep
 |the local whitelist database from getting too big.

Ok i mean with all the checks that the SMTP server performs before
the greylist policy even comes into play (most importantly DNS domain
name and address/mapping verification) this may indeed be
doable, but i do see very much different behaviour, for example by
NetBSD.org, with multiple deferrals and short-time whitelisting.
(Without knowing the actual configuration details, and happily
going down a colour name route btw.)

 |>I find it interesting that such simple greylisting that cannot
 |>even correctly identify a specific message seems so useful still.
 |
 |But identifying the specific message is not important. How often does
 |a spambot send two different messages with the same envelope in a
 |time gap that makes it look like a retry? Basically, never.

You know and that is what is so hard to believe.  Given that the
concept is twenty years old and the standard becomes ten this
year, wouldn't it make sense for a bot to simply try an address
a second time after X minutes, if it has the time and space?

Never done that myself, but in my logs i have seen SSH attacks
which really figured out firewall delays and then tried logins in
an interval that was not causing blacklisting, over months!  (I
actually cheered the attacker via a public list he/she might have
read, and doing this now and here, again!  (The login as such was
not successful, thus.))

What i also have seen is, i tried out OpenSMTPD once for a day, it
might have been in 2017, it was to my shame a misconfiguration
(falsely read the manual) that turned it into an open relay, and,
i really should have kept the logs because it was so fascinating,
one IP connected, and did nothing for several minutes, then
another IP connected, and then they started sending mails
simultaneously (how did they know??), to my shame.  (All against
Yahoo Taiwan if i recall correctly, then there was a friendly
email from Yahoo Taiwan iirc that my account produces suspicious
activity, massively understated.)  It was a nice interval (for
nothing), IPs came, send massively for some time, then went again.
(Luckily i looked once again before i went, reinstalled postfix
and with its *error_limit= directive that storm subsided within
minutes.)

What i mean is, there are smart attacks doable and out there, and
whereas the small attack surface of greylisting may not seem worth
the effort, and maybe only today, improving this situation by
strengthening the concept of greylisting standard-wise may bring
benefits in the future.

 |I have tried a version of greylisting that did the retry after DATA
 |and kept a hash of the message. I found that a certain number of
 |systems regenerate the retried message in a form that is not quite the
 |same, e.g., different timestamps. The hashes don't match and the
 |retries always failed.

I personally did not think about greylisting at such a late time,
quite a lot of processing has happened already by that time.
I thought with say a regular SMTP extension that brings the unique
Message-ID into the SMTP prologue greylisting could be
strengthened by allowing exact identification of messages, and so
greylisting and its delays could be improved by allowing the
greylist daemon to let exact matches pass through after one retry,
whereas software which does not use the extension could continue
to be penaltized by even (many) hours, as can be seen in practice.

I mean, of course grey listing affects only a small subset of
traffic at all, and many gigantic providers like gmail etc., as
well as a lot of well-known domains, they all are or have to be
whitelisted anyway, best at pre-greylisting stages, of course
(though greylist server whitelisting may even be faster than
explicit whitelisting dependent on the used database format and
the file system's locking speed).

--steffen
|

---

From: "John Levine" <[email-masked]>
Date: 7 Jan 2022 18:07:52 -0500

>NetBSD.org, with multiple deferrals and short-time whitelisting.

I have seen strange implementations of greylisting like this. When
I've asked people what the point of all of the extra delay is, the
most coherent answer I've gotten is that if they delay the mail and
it's spam, the IP might have gotten added to a DNSBL by the time they
retry. Of course, if that is what they really want, they should put
the incoming mail into a queue, wait a half hour, and then recheck the
DNSBLs before delivering it. It seems like they believe that making
greylisting stricter will make mail more secure, for ill defined
definitions of "more" and "secure."

I also think some of the thinking is stuck in the distant past
when consumer ISPs didn't block outgoing port 25 and it was
more common for mail to come from behind NATs.

>a second time after X minutes, if it has the time and space?

That's not how bot spamware works.  It's just about volume,
blasting mail out and not caring what happened to it.  To
retry you have to remember what you've sent and have some
sort of retry queue.  Naah, they have plenty of addresses,
they'll just send more spam.

>simultaneously (how did they know??)

I see lots of botnets doing open relay scanning, with results
acted on quite fast.

R's,
John

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sat, 08 Jan 2022 22:11:48 +0100

John Levine wrote in
 <[email-masked]>:
 |It appears that Steffen Nurpmeso  <[email-masked]> said:
 |>doable, but i do see very much different behaviour, for example by
 |>NetBSD.org, with multiple deferrals and short-time whitelisting.
 |
 |I have seen strange implementations of greylisting like this. When
 |I've asked people what the point of all of the extra delay is, the
 |most coherent answer I've gotten is that if they delay the mail and
 |it's spam, the IP might have gotten added to a DNSBL by the time they
 |retry. Of course, if that is what they really want, they should put
 |the incoming mail into a queue, wait a half hour, and then recheck the
 |DNSBLs before delivering it. It seems like they believe that making
 |greylisting stricter will make mail more secure, for ill defined
 |definitions of "more" and "secure."

I have to say i am not really satisfied with this statement of
yours, for one the software side says "what queue", and "how to
drive this in practice with out-of-the-box software" (not to
mention "easily"), and then NetBSD people never appeared
half-assed to me, especially regarding network technology.

For example they patch postfix to connect it with their blocklist
daemon (that is now also in FreeBSD) in order to cheaply announce
failed login attempts to the firewall.  That spirit does not seem
to be so brutely simple and asocial ("simply lay down
responsibility to the network") as you imply here.

So i am not convinced, but of course, the mechanisms you mention
may also reduce spam load, and it may well be that this plays
a role.

 |I also think some of the thinking is stuck in the distant past
 |when consumer ISPs didn't block outgoing port 25 and it was
 |more common for mail to come from behind NATs.

So you are in favourite to obsolete Greylisting as such?
I am just wondering, given how rigorously you limit access to your
own mail account, in a world where more and more organizations and
public bodies like Universities etc. outsource their entire mail
infrastructure to the giant providers like GMail, to drop their
administrative effort, most likely that will be the reason.

 |>You know and that is what is so hard to believe.  Given that the
 |>concept is twenty years old and the standard becomes ten this
 |>year, wouldn't it make sense for a bot to simply try an address
 |>a second time after X minutes, if it has the time and space?
 |
 |That's not how bot spamware works.  It's just about volume,
 |blasting mail out and not caring what happened to it.  To
 |retry you have to remember what you've sent and have some
 |sort of retry queue.  Naah, they have plenty of addresses,
 |they'll just send more spam.

You know, i am not selling something here.  You are surely right
for most of the bots around.  But there are more intelligent
things around.  And there are experienced, considerable players
around which configure accordingly, in practice.

I am only implementing a greylisting daemon, a standardized
anti-spam aka anti-abuse policy, then further hopefully next week,
and i was struggling with the impossibility to exactly identify
emails, and that a bot that simply sends almost whatever email
anew after x+1 minutes can only be whitelisted when looking for
the triple client_address (mutilated!) / sender / recipient.
And if players stop being nice or for whatever reason the IPv4/24
aka IPv6/64 mutilation may even render Greylisting impossible.

And so my thinking was that the possibility to announce the one
UUID that the Internet Message Format SHOULDs via a SMTP extension
could be an idea how to improve this situation.  Like i said, then
your "single retry" could be sufficient if senders play nice, and
the multi-hour delays that exist as of today would be left to the
rest.

 |>(falsely read the manual) that turned it into an open relay, and,
 |>i really should have kept the logs because it was so fascinating,
 |>one IP connected, and did nothing for several minutes, then
 |>another IP connected, and then they started sending mails
 |>simultaneously (how did they know??)
 |
 |I see lots of botnets doing open relay scanning, with results
 |acted on quite fast.

Yaaah, but note they did not log in, the first one only connected
and staid.  Well maybe they acted according to the greeting
string, it is likely that more than one read the manual as it was
by then falsely.  You know it had that robotic-AI i-see!
appearance, as it appeared in the logs, followed by the
aggregation of troops that came thereafter in intervals.  My gut
feeling at least.  Whereas that SSH attack i have seen, it was
like a spider walking, very cautious, it must have taken months to
get there, it was only one, and she/he did it.  Whatever.

A nice Sunday i wish everybody.

--steffen
|

---

From: "John Levine" <[email-masked]>
Date: 8 Jan 2022 16:27:08 -0500

It appears that Steffen Nurpmeso  <[email-masked]> said:
>anti-spam aka anti-abuse policy, then further hopefully next week, ...

Everyone I know who does greylisting with Postfix uses Postgrey,
available here:

https://postgrey.schweikert.ch/

Are you sure you need to reinvent this particular wheel?

R's,
John

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sat, 08 Jan 2022 23:37:43 +0100

John Levine wrote in
 <[email-masked]>:
 |It appears that Steffen Nurpmeso  <[email-masked]> said:
 |>I am only implementing a greylisting daemon, a standardized
 |>anti-spam aka anti-abuse policy, then further hopefully next week, ...
 |
 |Everyone I know who does greylisting with Postfix uses Postgrey,
 |available here:
 |
 |https://postgrey.schweikert.ch/
 |
 |Are you sure you need to reinvent this particular wheel?

Well, the answer is a bit lengthy, i know this software, and
i have perl on my server at the moment.  (Ever since lighttpd
suddenly stopped creating cgit output, and on the cgit list then
was no support at all even though maintainer of lighttpd correctly
pointed out that failing a sendfile(2) should be covered as per
"Applications may wish to fall back to read(2)/write(2) in the
case where sendfile() fails with EINVAL or ENOSYS", but the
provided patch did not even get a response ... the maintainer
as a kernel guru in some sense of guru seems to have known it is
a kernel bug, but no report was given to the kernel people until
someone finally stepped up .. anyhow it took months before the
very combination Linux / lighttpd / cgit resumed normal operation
without local effort, and one day i was no longer willing to and
fell back to gitweb that ships with git anyhow, .. which requires
perl.)

So that postgrey uses perl would be ok even though it uses quite
a bit of modules.  It uses BerkeleyDB, which is no longer free
software, version 5.3.28 is, aka
  Berkeley DB 11g Release 2, library version 11.2.5.3.28:
  (September 9, 2013)
as via [1], which is quite old.  But ok.  It however seems to use
what i call "[Chris ]Torek Hash", which faded a bit to grey when
facing possible real world attacks me thinks.  It is also very big
and slow, and imposes expensive locking.

  [1] http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/

As an effort to restrict all my (easy non-SQL, and only those here
right now) DB use cases to only LMDB of the OpenLDAP project for
example i wrote an accepted patch for bogofilter(1) in 2018, and
by then these were the numbers

  | |> it is very small (the raw AlpineLinux code package is 90KB,
  | |> whereas DB is 1.6MB; the cloned repo is 1.2MB, whereas the 5.3.28
  | |> DB tar ball unpacked in git is 31MB), and the code is also open
  | |> and openly maintained.  And Postfix supports LMDB as a replacement
  | |> for DB out of the box, too.  All this is very desirable to me.
  | |
  | |Repo size of a support library isn't normally a relevant metric, but
  | |this is a valid point, as is its license:
  | |
  | |   text          data     bss     dec     hex filename
  | |  80510          1504       8   82022   14066 /usr/lib64/liblmdb.so
  |
  |Runtime is much smaller here, too:
  |
  |  #?0[steffen@essex nail.git]$ size /usr/lib/liblmdb.so
  |     text    data     bss     dec     hex filename
  |    69680    1344      80   71104   115c0 /usr/lib/liblmdb.so
  |  #?0[steffen@essex nail.git]$ size /usr/lib/libdb.so
  |     text    data     bss     dec     hex filename
  |  1549515   38744      64 1588323  183c63 /usr/lib/libdb.so

(It does not support VACUUM or any such, therefore i dump-out and
recreate these DBs in such heavy use cases once a month
.. automatically.)  Very, very fast and minimal overhead.

But in fact all this is part of an effort to replace the
end-of-life Python2 Mailman2 with something homegrown.
I initially planned to use the policy server to whitelist
subscribers fast, for example.  That it also should do regular
greylisting was not the first use case.  However, in the meantime
i added regular use cases for a postfix configuration

   check_sender_access lmdb:/etc/postfix-lmdb/sender_access,

before the envisioned check_policy_service, which made this less
desirable, because LMDB uses fast read/write locks (or what i do
not like, "robust" mutexes) if available, and then even very fast
unix(7) communication will likely be outperformed.
However, maybe that will be replaced by whitelist on the policy
server side (again).
It will use SipHash for its all-in-memory dictionaries.

--steffen
|