Subject: [ietf-smtp] Levels of proposals

From: Brandon Long <[email-masked]>
Date: Thu, 3 Dec 2015 16:36:16 -0800

The WG proposal seems to imply taking all IPs out.  The discussion has
mostly been about submission.

It seems to me that there are at least three different IPs used, and some
of these are going to be visible regardless of intent.

Ie, there is the submission IPs, there are "internal" IPs, and external
ones.

Submission IPs seem like the largest level of risk, and from my gross
understanding of anti-spam, pretty minor.  I'm not sure what the current
source levels are, but submission IPs would be most useful in the case of
hijacked account spam or abusive account spam.  Presumably, if spam reports
about such are forwarded to the MSP, then the MSP can easily store the
information somewhere other than the easily forged headers and take the
appropriate action.  Only if the answer is "they don't take action" would
you need more.

Also, if the previous thread's list of large MSPs inclusion of submission
IPs is correct, then >2 out of the top 3 have already removed them (ie,
only a fraction of Gmail mail has them at this point).

Internal IPs, this hardly seems controversial.  If any mail system did
that, not sure if anyone would bat an eye.

External IPs, ie server to server... I guess one may learn "something" if
you can tell which submission server they talked to, we certainly have
servers across the world... but even with 20 odd locations, I doubt that
would be that specific.

So, I would recommend concentrating on submission IPs.  I might also
include a recommendation for submission servers to store the IPs for some
length of time to allow for abuse handling, or even to include an encrypted
version in the message.

Brandon

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 4 Dec 2015 01:19:50 +0000

This seems like another sensible opinion that doesn't make
overbroad claims about the goodness/badness of what's proposed
to be examined. More of us limiting our posting like this
would be good IMO.

S.

On 04/12/15 00:36, Brandon Long wrote:
> _______________________________________________
> Shutup mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/shutup
>

---

From: Chris Lewis <[email-masked]>
Date: Thu, 3 Dec 2015 21:00:01 -0500

> mostly been about submission.

The draft has been mostly about submission, but there have been some 
conversations about internals.  I've seen auditors bring up points about 
obscuring internals (corporate context, it's one of their checkoff 10 
commandments), but, we managed to convince them that ease of diagnosing 
problems outweighed the downsides.

Before getting too far into concrete proposals, we still need to decide 
whether it should be done, and what (if anything) should be done in 
mitigation.

That said:

> understanding of anti-spam, pretty minor.

Yes, and no...  It depends on how you use/acquire them, this is a fairly 
complex/esoteric end of spam filtering, and just doing it justice would 
take up pages of text.  I'll try to short cut - purely from a filtering 
but not-ML perspective:

Using them:

If you have a list of IPs known to be infected with AUTH-cracking 
spambots, it's of immediate/valuable use to both the MSAs themselves in 
detecting malicious injects, as well as the recipient's filtering, and 
header forgery is not an issue (certainly not to MSAs, and headers that 
forge collisions with the list you don't want anyway).

The potential yields are huge - ISPs (using a list we create of 2.2M IPs 
demonstrably capable of AUTH-cracking) can see botnet suppression rates 
at the MSA of 80% (of all submissions) or even higher, depending on the 
spam/botnet de-jour.  In just one case it's in the 1000+/second range. 
Some of these botnets are very very prolific per IP.

Which boils down to the MSAs being able to stop it at source, or the 
receivers being able to stop it at destination.  If a significant 
fraction of MSAs stopped it at source, then, maybe, the MSA Received 
line isn't important.  But where they don't (the majority), losing the 
MSA Received line is potentially a big hit.

Obtaining them:

Obviously, someone trying to authenticate to you and is demonstrably 
infected isn't spoofable.  That yields the vast majority of our list.

On a case-by-case basis with specialized knowledge it is possible to 
derive trustworthy infected submission addresses from received lines. 
Yes, actually, it really is.  Useful, but not terribly significant (or 
at least how far we've followed that rathole).

By eliminating MSA received clauses, it means that only the MSA can do 
blocking based on this technique, the receivers can't.

On one server, in a 40 minute interval, 774123 out of 1126763 
connections were AUTH spoofing from just one particular botnet.  That's 
775,000 spams that ordinary receivers couldn't filter via IP if the from 
clauses disappeared.

> Also, if the previous thread's list of large MSPs inclusion of
> submission IPs is correct, then >2 out of the top 3 have already removed
> them (ie, only a fraction of Gmail mail has them at this point).

The top 3 aren't generally large scale botnet spam sources, and botnets 
generally don't use real mail servers (except via AUTH crackers), and 
the top 3 are amply well instrumented with rate limiters etc.  So they 
really aren't relevant to this.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 02:11:47 +0000

Thursday, Dec 3, 2015 9:00 PM Chris Lewis wrote:
> If you have a list of IPs known to be infected with AUTH-cracking spambots, it's of immediate/valuable use to both the MSAs themselves in detecting malicious injects, as well as the recipient's filtering, and header forgery is not an issue (certainly not to MSAs, and headers that forge collisions with the list you don't want anyway).

Can you unpack "AUTH-cracking spambots" for the greenhorns?   I have no idea what this means, and google unfortunately was unable to help.

--
Sent from Whiteout Mail - https://whiteout.io

My PGP key: https://keys.whiteout.io/[email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Thu, 03 Dec 2015 18:31:53 -0800

> Can you unpack "AUTH-cracking spambots" for the greenhorns?  I have no
> idea what this means, and google unfortunately was unable to help.

Standard practice for attackers these days is to automate attacks on any
sort of password-protected system, whether that be web pages,
authentication providers, or anything else that takes a password.  Usually
this is done by taking some list of common passwords and some list of
account names and just brute-forcing combinations, although some attackers
do more sophisticated things.

Obviously, that sort of brute force approach is easy to detect and
throttle, so the next step in the arms race was for attackers to use large
networks of compromised machines, usually home machines behind DSL and
cable modem links, each of which tries a small number of passwords against
a variety of targets to stay below the radar.  Those machines were
generally compromised via malware of some kind and are part of a botnet,
without the knowledge of the user of the machine.

This is used against SMTP AUTH just like it is against anything else on
the Internet that takes a password.  The usual goal is to send out spam
using other people's valid credentials to bypass spam filtering, or to
send phishing or stock pump and dump schemes, or what have you.

One useful tool in fighting this sort of attack is to be able to collect
and share information about currently compromised client IP addresses so
that you can detect them as being part of a bot net and use much more
aggressive rate limiting on these sorts of attempts, or block any email
that they successfully sent after cracking someone's SMTP AUTH password.

-- 
Russ Allbery ([email-masked])              <http://www.eyrie.org/~eagle/>

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 02:39:35 +0000

> without the knowledge of the user of the machine.

Thanks for explaining!

I am still a bit puzzled: how does increasing the number of attackers help to bypass the throttling mechanism?   Why isn't the throttle per id/password pair, rather than per ip-address/password/id triple?

Secondarily, if distributed processing makes throttling per id/password pair difficult, why is it hard to do the botnet IP address matching at the authentication point?   This seems like it would avoid a _lot_ of extra processing.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 03 Dec 2015 19:48:52 -0800

Ted Lemon <[email-masked]> writes:

> I am still a bit puzzled: how does increasing the number of attackers
> help to bypass the throttling mechanism?  Why isn't the throttle per
> id/password pair, rather than per ip-address/password/id triple?

Why would you throttle per id/password pair?  The attacker doesn't try the
same pair more than once.  That would be pointless.

Let me try explaining this from a different direction.  Suppose you are an
attacker.  You have a list of 10,000 common account names, and a list of
10,000 common passwords.  You want to try each of those passwords with
each of those account names and try to find a combination that works.

If you connect from your compromised host and start plowing through the
100 million combinations, after the first few thousand that you try from
that host any decent authentication system will go "hey, there are tons of
failed authentications from this one IP address" and block all further
attempts, successful or not, from that IP address.  That's pretty easy to
implement.

So what the attacker does instead is use their botnet of a million
compromised personal computers (that's sadly not really an exaggeration),
and has each one of those hosts try 100 combinations and then disconnect.
This is now below (or at least near) the threshold for an actual customer
typoing their address or password or something, and rate limiting becomes
fairly useless as a defense.

> Secondarily, if distributed processing makes throttling per id/password
> pair difficult, why is it hard to do the botnet IP address matching at
> the authentication point?  This seems like it would avoid a _lot_ of
> extra processing.

Chris addressed this quite well in his message.  I don't really have much
to add to what he already said.

The TLDR in case something about that message was confusing is that only
the authentication point can block the IP addresses at the authentication
point, but you can analyze Received headers to do a bunch of other things,
such as determine compromised botnet IP addresses that someone else
*didn't* block but that you *do* want to block for *your* service.  It
improves the scale and flexibility of what you can do by basically giving
you more threat intelligence.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 04:22:46 +0000

> same pair more than once.  That would be pointless.

I think you missed the distinction I was making.   You are describing throttling per ip-address, irrespective of the password/id pair.   I am asking why you don't simply say "if more than 10 attempts are made on this username per hour, lock it out for a while."   This is a common practice on some web sites.   If you require the user to have a non-dictionary password, then 10 attempts per hour is few enough to prevent a botnet of any number of hosts from guessing that password.

> you more threat intelligence.

Yes, I have heard that before.  It makes sense.   However, it does make me wonder why you don't just stop accepting mail from sites that are this badly run until they shape up.   You seem to be suggesting that it's because you value the intelligence that you glean from their incompetence.   Did I misunderstand?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 03 Dec 2015 20:53:32 -0800

>> Why would you throttle per id/password pair?  The attacker doesn't try
>> the same pair more than once.  That would be pointless.

> I think you missed the distinction I was making.  You are describing
> throttling per ip-address, irrespective of the password/id pair.  I am
> asking why you don't simply say "if more than 10 attempts are made on
> this username per hour, lock it out for a while."

Oh, you didn't mean throttle by id/password pair.  You meant throttle
purely by user ID.

There are two reasons (well, at least -- maybe more) why this doesn't help
as much as it sounds like it would, particularly in the case of SMTP AUTH.

One is that the attacker just changes the pattern of their brute force
search to distribute it across more known accounts or across more time, so
they just probe more accounts with fewer passwords.  It's all the same to
them; they're just doing a combinatoric search, and varying parameters
that way doesn't have too negative of an impact on their ability to
compromise accounts.

Another reason is more specific to SMTP AUTH: dozens of incorrect login
attempts for the same username is a common *legitimate* pattern for users
who have a single misconfigured device (a typo in their password, for
instance.  (And thanks to cell phones, those failed logins often come from
a huge variety of IP addresses.)  SMTP UAs often retry authentication
pretty aggressively and will happily look like an attacker.  If you lock
an account for that pattern, you can lock out that user's other legitimate
devices.  Web sites usually don't have to worry as much about automated
incorrect legitimate login attempts.

Now, you can do more complex things like checking whether the failures are
all the same password or are different passwords; I'm not saying there
aren't multiple approaches here.  My point, rather, is that figuring out
what IP addresses are probably part of botnets is really useful and not
something to be casually discarded as additional information for your
abuse prevention.

This example is obviously simplified in an attempt to answer your original
question, since you said you were wholly ignorant of this whole area of
abuse handling.  Real-world scenarios get more complicated and involve
more complex parameters to rate limiting and ways of figuring out what's
going on.  The point is that threat intelligence about compromised IPs is
really useful input into those systems, and that can be garnered from spam
email Received headers in some fairly effective ways, as touched on by
Chris in his original message.

Note that this sort of thing (login abuse detection) is something for
which many companies employ an entire team of full-time people devoted to
nothing but this.  Those people use all the relevant tools that they can
get their hands on because this is an arms race against quite
sophisticated attackers.

I was (mostly successfully) staying out of this thread, but one other
comment I'll make while I'm here: I suspect a lot of people who work on
email abuse detection and mail filtering would make the argument that they
care just as greatly about user privacy as you do, but consider phishing
attacks a much more serious threat to the privacy of the average user than
the information that leaks out in email headers.  It's one thing to get a
user's home address; it's quite another to get all of their bank
information and access to their personal email.  Obviously none of us want
those two goals to be in conflict, but please remember that spam filtering
is also *phishing* filtering, and therefore is *also* privacy defense for
users, and quite important privacy defense at that.

> Yes, I have heard that before.  It makes sense.  However, it does make
> me wonder why you don't just stop accepting mail from sites that are
> this badly run until they shape up.  You seem to be suggesting that it's
> because you value the intelligence that you glean from their
> incompetence.  Did I misunderstand?

Yes, in a couple of ways.

For one, just deciding to ban whole sending sites from your email system
and saying tough luck if someone at that site wants to mail you is the
sort of thing that you can do if you're running a hobbyist or personal
mail service.  When you have a large-scale email system for paying
customers, it turns out that those customers are unimpressed by your
decision to not let specific people talk to them because you don't like
how they run their mail system.  Or, similarly, if you're running a
large-scale mail system for a company, deciding that your sales people
don't get to hear from certain prospects, or that your business people
simply don't get to talk to certain domains, based on your opinions about
their email policy doesn't go over well.  You have to be pretty damn sure
that the sending site is 100% garbage before you can do things like that,
as opposed to something more sophisticated.

For another, I think you're missing that can garner useful information
from the Received headers of messages that you *reject*.  You can accept
the whole message body first and then reject it or throw it away.  If
you're pretty sure for other reasons that this collection of a few million
messages are all spam or phishing, analyzing the headers of those messages
to pull out patterns like, say, a list of submission IP addresses and then
using that IP addresses as an additional data point in analyzing more
borderline messages, or as a set of IP addresses for which to do
additional login throttling for your own authentication systems, can be
very helpful.  Chris had some concrete numbers in his original message.

---

From: Rich Kulawiec <[email-masked]>
Date: Fri, 4 Dec 2015 06:34:32 -0500

> as much as it sounds like it would, particularly in the case of SMTP AUTH.

Here's one of those "more".

Some sites construct usernames by taking a user's real name and turning
it into "first-initial lastname" or "lastname first-initial", e.g., John
Smith becomes jsmith or smithj.  Sometimes this is truncated to N characters,
with 8 being fairly common.

A quick online search will reveal the 1000 most common surnames in the US.
Prepending/appending the 26 letters of the alphabet and truncating yields
52K candidate login names.

Some of these same sites try to implement account login throttling
policies that are based on F failed tries in M minutes, e.g., if
F > 10 and M < 3, then the account is locked out for ten or fifteen minutes.
They might do this with with SMTP AUTH or IMAP or SSH or any other service
which requires a username/password pair.

An attacker in control of a botnet can thus conduct an effective DDoS attack
against at least some of the users of such sites by hitting those 52K
accounts (whether they exist or not) repeatedly with random passwords
and tripping the throttling mechanism.  Depending on how the throttling
mechanism works, they may be able to fine-tune the attack to match it;
or they can simply fire blindly, setting the attempt rate high enough
that it exceeds any plausible F under any plausible M.  And of course
by randomizing the source IP of the attempts they can make attempts by
defenders to block this tedious or impractical.

Salt to taste for other countries with different surname popularity or
for other operations with different algorithmic means of constructing login
names from real names, etc.  Augment with more login names constructed
by scraping the target's web site.  And so on.

This is not a hypothetical, incidentally.

---rsk

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 08:10:58 -0500

> Can you unpack "AUTH-cracking spambots" for the greenhorns?   I have no idea what this means, and google unfortunately was unable to help.

Primarily botnets that connect to MSAs and use stolen (or potentially 
brute-forced) userid/password credentials in order to use the MSA/MTA 
combo as an open relay.

There's at least one major windows executable botnet that does this in 
very high volumes.  This is a particularly long-standing botnet that has 
many other nasty tricks at its disposal (DDOS, keystroke harvesting, 
account stealing etc).

There are several web server compromises that do the same thing, 
potentially using the same sources for compromised userid/passwords, 
tho, these aren't nearly the volume as the aforementioned botnet.

It's really quite surprising how successful that some spam campaigns are 
in doing this, and I can only imagine that at least part of it is  the 
botnet dredging out an infected user's MSA/userid/password triple from 
their mail reader and propagating it to the rest of the botnet (along 
with harvesting address books), and hence have enormous numbers of 
compromised accounts to exploit.  They may also be using more generic 
userid/password dumps from online site leaks.

We've seen ISPs subject to massive attacks where hundreds or thousands 
(or more) of different IPs do authenticated MSA submissions of 10s or 
100s of thousands of spams in fairly short (hours) intervals, resulting 
in traffic flows 100s of times above normal to the point of overloading 
the ISP's MSAs.  This is even with quite aggressive per-user and per-IP 
rate limiting.

AUTH-cracking to this extent is a relatively recent phenomena, and is 
clearly being used as an attempt to bypass normal direct-2-MX botnet 
blocking and hijack the reputation of the MTA instead of some random 
cracked PC.

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 08:26:57 -0500

> fairly useless as a defense.

Hi Russ, it's been a long time since we last were in the same place 
(physically or otherwise ;-)

One of the most depressing indicators I've seen is watching thousands of 
different IPs only attempt one or two submissions, and not trying again 
EVEN THO both submissions succeeded and none failed.

Secondly, we have full instrumentation, and the passwords they're trying 
are pretty obviously not "common password" dictionaries or brute-forcing 
attempts.

Clearly they're not lacking in compromised accounts.

So, the problem is much larger than "brute force or common password 
dictionaries"

> to add to what he already said.

Yeah, it's obvious that the authentication point is the best place to do 
it, but the reality is that most of them don't (we're working on that), 
so...

> you more threat intelligence.

you have to do it for them so that _you_ can block what they should have 
and didn't.

Furthermore, intelligent botnet operators will aim their cannons away 
from the sites successful at MSA-blocking to the ones who aren't.

---

From: Paul Smith <[email-masked]>
Date: Fri, 4 Dec 2015 13:44:03 +0000

>>> the authentication point?  This seems like it would avoid a _lot_ of
That made me think - is there a place for a distributed approach to 
dealing with the problem?

If there was a "central" repository of IP addresses which have generated 
failed login attempts to ANY server, then it seems to me to make sense 
that you could catch them quicker - if someone tries one failed login to 
5 different servers in the past hour, the chances are they are doing 
something naughty, but 5 failed logins to one server may not even 
trigger a warning in many cases. So, every time a server gets 
'suspicious' about an IP address it can tell the repository about it, 
and other servers can use something like DNS to query that repository 
and act as it wishes.

I know that isn't really related to SMTP, but since it came up, it made 
me think.

Probably a stupid idea, what with scaling, potential for abuse, etc, but 
may be worth considering for a second or two.

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 08:46:51 -0500

> as much as it sounds like it would, particularly in the case of SMTP AUTH.

Third and most telling reason: few people know how many valid 
id/password pairs they already have, and coupled with the large numbers 
of IPs they also have, and the fact that userid/password pairs often 
work under multiple services, you can spread the spam across so many IPs 
and userids that throttling is only useful in some marginal edge cases 
(even when the throttle levers are in the hands of a certified BOFH).

Case in point: less than 10% of our list of auth-cracking IPs is derived 
from methods useable by MSAs, and those that are derived from MSAs 
utilize very specialized fingerprinting that has nothing to do with 
throttles.

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 09:06:30 -0500

> may be worth considering for a second or two.

It's not an altogether stupid idea at all if more than one person has 
thought of it ;-)

But when starting out with it, we found it a lot safer to go with said 
repository only including IPs of demonstrably-by-infection auth-crackers 
until you have enough data sources to correlate on failures.  It's 
beginning to look as if the "demonstrably-by-infection" is sufficiently 
complete that increasing the risk/complexity by correlating "suspicious" 
for more listings isn't necessarily worth it.

---

From: Claus Assmann <[email-masked]>
Date: Fri, 4 Dec 2015 06:12:35 -0800

On Fri, Dec 04, 2015, Paul Smith wrote:

> That made me think - is there a place for a distributed approach to dealing
> with the problem?

Maybe another DNS block/reputation list (compare spamhaus?).

---

From: Ned Freed <[email-masked]>
Date: Fri, 04 Dec 2015 07:17:55 -0800 (PST)

> Oh, you didn't mean throttle by id/password pair.  You meant throttle

An important reason not to have a pure user ID block is it provides an easy way
to block someone's access to their email: Simply set up a system to constantly
bang on their account with the wrong password, and they're effectively locked
out.

An alternative is a pure user ID slowdown (as opposed to a complete block),
but there are limits to how effective this can be. Bad guys can afford to be
patient whereas legitimate clients time out pretty quickly, turning a 
slowdown back into a block.

> incorrect legitimate login attempts.

I note that this also applies to IMAP and POP3. In fact the most common
case is probably going to be legitimate IMAP clients banging on the door.

Sometimes a single client even does it from multiple threads.

			Ned

---

From: Hector Santos <[email-masked]>
Date: Fri, 04 Dec 2015 10:59:00 -0500

On 12/4/2015 8:10 AM, Chris Lewis wrote:
>> Can you unpack "AUTH-cracking spambots" for the greenhorns?   I have
>> no idea what this means, and google unfortunately was unable to help.
>
> ...

Hi, I'm surprise to read you say this is "relatively recent."  Are you 
mean in months, years or one to several decades?

I have seen tons of "auth-cracking/brute force" auth attempts done at 
all the IP protocol servers. Generally, they do it all in one 
connection session where a "Maximum Failed Login Attempt" count may is 
used.  I always knew it was possible, but I suppose there are now 
increased distributed "botnet" forms of this where the Failed Login 
Attempt Count would be reset at the connection.  I have not seen the 
need to do much because I am not seeing damage. Everything (robust 
high scale servers) have been in place for many years, you know "Set 
it and forget it."

Overall, having robust 24x7x365 load balancing/control IP 
tracking/blocking server(s) is par for the course, so this is more 
about relaxing the STD10/RFC2821/RFC5321.Received SMTP receiver 
protocol requirement to help avoid mitigate negative correlation 
methods with ESMTP AUTH, right?

Incidentally, what I am seeing more of late, are single or "botnet" IP 
connections that do nothing, i.e. they sit there idle and they don't 
attempt to login. What is interesting is that if you (force) 
close/drop it, they immediately reconnect with the same IP.

-- 
HLS

---

From: Paul Smith <[email-masked]>
Date: Fri, 4 Dec 2015 16:16:44 +0000

> high scale servers) have been in place for many years, you know "Set 
We used to see things connect and try lots of logins until they got 
blocked (they'd actually carry on for a bit after that because we don't 
tell them they've been blocked)

Now, we see lots of IP addresses connecting and making one or two login 
attempts then giving up. It's obviously a concerted 'attack' because 
lots of IP addresses do it at the same time, but each individual IP 
address doesn't try much. This does tend to defeat our software's 
blocking system, and it's a bit tricky to get around given that our 
software is on individual small business servers which might only see a 
few hundred auth attempts every few days, rather than huge mail 
providers which I guess see a lot more.

What I'm thinking of is having the option of the software reporting 
failed login IPs 'home' to a central server here, which other 
installations can query through a DNS lookup, but I'm still pondering 
whether that would help or hinder (the extra load of reporting & 
querying may override the load of dealing with the failed logins). If 
two or more of our software installations receive failed login attempts 
from the same IP address within a day or so of each other, the chances 
are it's a bot.

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 11:31:36 -0500

> mean in months, years or one to several decades?

I should say that "back in the day", SMTP-auth from BOTs was 
sufficiently rare that it could safely be ignored.

SMTP-auth from bot started in a noticable fashion about 2-3 years ago 
and continuing to rise to extreme levels in the past 6-12 months.  To 
some MSAs, the impacts were obvious before that.

To me, this is "relatively recent".  Sorry, should have clarified.

As a MUCH more recent development, remember "open relay"?  That was 
obsolete 10 years ago, and except for a couple of low volume Chinese 
spammers, not seen at all.  Well, guess what?  One extremely prolific 
spambot started doing it in very high volumes less than a month ago. 
That's right, spambots attempting to open relay through MTAs.  Shipping 
almost exclusively malware at that.

Fun eh?

---

From: Steve Atkins <[email-masked]>
Date: Fri, 4 Dec 2015 08:32:05 -0800

> On Dec 4, 2015, at 8:16 AM, Paul Smith <[email-masked]> wrote:
> 
>> I have seen tons of "auth-cracking/brute force" auth attempts done at all the IP protocol servers. Generally, they do it all in one connection session where a "Maximum Failed Login Attempt" count may is used.  I always knew it was possible, but I suppose there are now increased distributed "botnet" forms of this where the Failed Login Attempt Count would be reset at the connection.  I have not seen the need to do much because I am not seeing damage. Everything (robust high scale servers) have been in place for many years, you know "Set it and forget it."
> We used to see things connect and try lots of logins until they got blocked (they'd actually carry on for a bit after that because we don't tell them they've been blocked)
> 
> Now, we see lots of IP addresses connecting and making one or two login attempts then giving up. It's obviously a concerted 'attack' because lots of IP addresses do it at the same time, but each individual IP address doesn't try much. This does tend to defeat our software's blocking system, and it's a bit tricky to get around given that our software is on individual small business servers which might only see a few hundred auth attempts every few days, rather than huge mail providers which I guess see a lot more.
> 
> What I'm thinking of is having the option of the software reporting failed login IPs 'home' to a central server here, which other installations can query through a DNS lookup, but I'm still pondering whether that would help or hinder (the extra load of reporting & querying may override the load of dealing with the failed logins). If two or more of our software installations receive failed login attempts from the same IP address within a day or so of each other, the chances are it's a bot.

If it's a bot then 95%+ of the traffic it emits is malicious, so neither detecting it nor using the data are limited to solely email login attempts - meaning the effort can be shared and the benefits multiplied. It'd be interesting to compare those addresses against the XBL, as one example.

Cheers,
  Steve

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 12:03:24 -0500

> If it's a bot then 95%+ of the traffic it emits is malicious, so neither detecting it nor using the data are limited to solely email login attempts - meaning the effort can be shared and the benefi…

Without actually seeing the data in question, I'm pretty sure that there 
will be a high correlation with the XBL.  There has with previous analysis.

However, as a caution, the XBL (botnets) is a superset of auth-capable 
botnets, so blind use of the XBL on the MSA will potentially lead to a 
sizable number of FPs.

To explain that, consider that most botnets are (still) only capable of 
direct-to-MX.  Hence, it's often the case that someone that is infected 
with a direct-to-MX botnet spewing crap can still be trusted 
(relatively! ;-) when submitting through their MSA.  An 
auth-capable-botnet list can therefore be much more safe to use on a MSA 
than the XBL is, unless the MSA uses the XBL in conjunction with other 
information (eg: integration with flow or auth fail statistics).

---

From: Paul Smith <[email-masked]>
Date: Fri, 4 Dec 2015 17:06:21 +0000

> If it's a bot then 95%+ of the traffic it emits is malicious, so neither detecting it nor using the data are limited to solely email login attempts - meaning the effort can be shared and the benefi…
Just for fun, I've looked at the logs of one of our customer's servers 
(based in the UK).

The last few days have seen some login attempts for unknown users. Note 
that these were login attempts for SMTP, IMAP4, LDAP and POP3... There 
were even some login attempts to our own custom service. It has a 
similar initial banner to POP3 (but is on a totally different port, and 
is most certainly not POP3), so I guess the crackers thought it was POP3 
on a different port, which is interesting, as that must mean they're 
port scanning as well as it's not a standard port.

The first few source IP addresses were:
46.183.217.136 (datacentre in Latvia somewhere?) - listed in XBL (CBL)
85.255.235.110 (Vodafone, UK "strategic"?) - listed in PBL and XBL (CBL)
85.255.232.62 (Vodafone, UK "strategic"?) - listed in PBL
94.117.180.221 ("The Cloud Networks", UK) - not listed
193.189.117.150 (Delorian Internet Services, Poland) - listed in SBL and 
XBL (CBL)
94.3.210.188 (Sky Internet, UK) - listed in PBL
and many, many more

Obviously we can't block logins from the PBL, but it looks like it may 
help if we have the option of checking login attempts against the XBL. 
It won't block everything, but it shouldn't harm. We do use the XBL for 
spam filtering, but I must admit that I hadn't thought about using it 
for checking logins.

The CBL docs say it shouldn't be used for blocking logins, but it may be 
useful to instantly block IPs which are on the CBL and also make a 
failed attempt to login, rather than giving them a few attempts as we 
normally do. But, given that our software is used by businesses for 
themselves, it may be OK to let them just use the CBL(XBL) outright to 
block logins, as they can easily change it if they need to, unlike if 
it's a huge MSP.

---

From: Steve Atkins <[email-masked]>
Date: Fri, 4 Dec 2015 09:10:31 -0800

> On Dec 4, 2015, at 9:06 AM, Paul Smith <[email-masked]> wrote:
> 
> Just for fun, I've looked at the logs of one of our customer's servers (based in the UK).
> 
> The last few days have seen some login attempts for unknown users. Note that these were login attempts for SMTP, IMAP4, LDAP and POP3... There were even some login attempts to our own custom service. It has a similar initial banner to POP3 (but is on a totally different port, and is most certainly not POP3), so I guess the crackers thought it was POP3 on a different port, which is interesting, as that must mean they're port scanning as well as it's not a standard port.
> 
> 193.189.117.150 (Delorian Internet Services, Poland) - listed in SBL and XBL (CBL)
> Obviously we can't block logins from the PBL, but it looks like it may help if we have the option of checking login attempts against the XBL. It won't block everything, but it shouldn't harm. We do use the XBL for spam filtering, but I must admit that I hadn't thought about using it for checking logins.
> 
> The CBL docs say it shouldn't be used for blocking logins, but it may be useful to instantly block IPs which are on the CBL and also make a failed attempt to login, rather than giving them a few attempts as we normally do. But, given that our software is used by businesses for themselves, it may be OK to let them just use the CBL(XBL) outright to block logins, as they can easily change it if they need to, unlike if it's a huge MSP.

It'd be an additional data source to consider, for sure. But pay close attention to Chris's comments about what the CBL/XBL lists and the behaviour to expect from those nodes, and be very aware of corporate farms of windows boxes behind a single NAT IP.

(I think we're drifting quite a long way from SMTP, let alone where this thread started, though :) ).

Cheers,
  Steve

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 13:15:09 -0500

> (I think we're drifting quite a long way from SMTP, let alone where this thread started, though :) ).

Heh, yah.  Which I've been replying only to ietf-smtp.

Corporate NAT farms tend not to be as big an issue as you may think, 
because such things tend to use mail infrastructures of their own, 
isolated from the NAT, and, having B2B access outside of that (including 
unrestrained visitors SMTP'ing to random places) tends to be frowned 
upon.  We simply did not permit outbound port 25 from anything but the 
mail servers even tho we didn't NAT.

Section 7 of the CBL Terms and Conditions gives him an out, his mileage 
(direct usage or in combination with other information) will vary 
depending on his environment.

Corporate/organizational (rather than provider) may do quite well 
without running afoul of the FP caveats.  He can but experiment, measure 
and decide for himself.

---

From: "John Levine" <[email-masked]>
Date: 4 Dec 2015 18:20:52 -0000

>understanding of anti-spam, pretty minor.  ...

If every mail provider had infrastructure and instrumentation as
sophisticated as Google's, that would be more or less true.  But it's
going to be a long time until we get there.

Also, as other people have noted, you can learn a lot from cross
correlating what you see, e.g., I see a fair amount of spam from AUTH
attackers and it's useful to know where the bot is and see whether the
same bots are attacking multiple sites or categories of sites.

We certainly have stuff to talk about, but the tradeoffs are a vastly more
difficult and subtle than the WG's proponents appear to understand.  Chris
Newman's counterproposal would be a good place to start.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Fri, 04 Dec 2015 11:18:21 -0800 (PST)

> some MSAs, the impacts were obvious before that.

FWIW, our customers are seeing this as well. It used to be that AUTH-cracking
on SUBMIT was a nonissue, now it's something you can't safely ignore.

I even see it on my home system. It kinds of amazes me that my little box is
seen as a target worth spending time banging on, but my logs show ~12,000
password guessing attempts in the last 12 hours. (It's all coming from Hong
Kong, the IPs doing it are in the SBL, and it seems to be driven on a generic
list of likely account names, not anything more targeted.)

> almost exclusively malware at that.

We're also getting reports of activities that look like attempts to trick
MTAs into relay through the use of oddball address formats, some legal,
some not. Not sure if this is what you're seeing or not.

				Ned

---

From: Chris Lewis <[email-masked]>
Date: Fri, 4 Dec 2015 17:19:07 -0500

> some not. Not sure if this is what you're seeing or not.

What you're describing sounds like old-school open relay trickery. 
Indeed, what you could be seeing is decade or older open relay testers 
being used to scan for "buggy" open relay prevention.

What I'm seeing isn't that at all.  What I see is more like "for 
everybody I want to spam (address A), pick another address (address B), 
connect to address B's MX, forge the email to be From: address B, and 
attempt to get B's MX to relay to address A".

However, it's possible that what I'm seeing would shift gears to more 
obvious trickery if the first attempt failed.  It you see the same IP 
hitting more normal relay rejects before trying the oddball addresses, 
it could be.

If you sent me a log record or received string[s] (off-forum please) I 
can probably tell for sure.

---

From: Brandon Long <[email-masked]>
Date: Fri, 4 Dec 2015 14:51:19 -0800

On Thu, Dec 3, 2015 at 6:00 PM, Chris Lewis <[email-masked]> wrote:

> take up pages of text.  I'll try to short cut - purely from a filtering but
> not-ML perspective:
>
> Using them:
>
> demonstrably capable of AUTH-cracking) can see botnet suppression rates at
> the MSA of 80% (of all submissions) or even higher, depending on the
> spam/botnet de-jour.  In just one case it's in the 1000+/second range. Some
> of these botnets are very very prolific per IP.
>
> receivers being able to stop it at destination.  If a significant fraction
> of MSAs stopped it at source, then, maybe, the MSA Received line isn't
> important.  But where they don't (the majority), losing the MSA Received
> line is potentially a big hit.
>
> Obtaining them:
>
> derive trustworthy infected submission addresses from received lines. Yes,
> actually, it really is.  Useful, but not terribly significant (or at least
> how far we've followed that rathole).
>
> On one server, in a 40 minute interval, 774123 out of 1126763 connections
> were AUTH spoofing from just one particular botnet.  That's 775,000 spams
> that ordinary receivers couldn't filter via IP if the from clauses
> disappeared.
>
> generally don't use real mail servers (except via AUTH crackers), and the
> top 3 are amply well instrumented with rate limiters etc.  So they really
> aren't relevant to this.
>

We've had to deal with hijacked account spam for years at this point, and
have invested heavily against it, and why we are doing our best to sunset
"programmatic" password based auth, ie:

https://googleonlinesecurity.blogspot.com/2014/04/new-security-measures-will-affect-older.html

https://support.google.com/accounts/answer/6010255

Of course, oauth isn't quite ready for widespread adoption, due to issues
with discoverability of end-points and software registration.
And a botnet may be able to lift the oauth tokens out of the local
software, but it has some benefits.

Brandon

---

From: Rich Kulawiec <[email-masked]>
Date: Sun, 6 Dec 2015 17:53:34 -0500

> list of likely account names, not anything more targeted.)

Do you expect to ever see a valid authentication from Hong Kong?

If not, then why not firewall it out?

The overwhelming majority of operations see locality of authentication:
that is, they see legitimate/successful attempts from the country they
operate in, or from perhaps a few countries.  (Obviously this is not
true of huge operations or of multinationals or anything like that.
But while those are prominent, they're also only a small fraction of
"all operations".  Joe's Donuts in Dubuque will never see a valid SMTP
AUTH request from Dubai.)

For such operations, it's useful to block everything and then just
allow traffic from the country (or handful of countries) that are known
a priori to originate valid attempts.  ipdeny.com has the appropriate
ranges and is updated regularly.

This is NOT a panacea.  However, *if* applicable, and it often is, it
does cut down on the noise considerably.  This in turn makes various
approaches to dealing with the remainder of the issue more tractable.

---rsk

---

From: Rich Kulawiec <[email-masked]>
Date: Sun, 6 Dec 2015 17:57:31 -0500

On Fri, Dec 04, 2015 at 01:44:03PM +0000, Paul Smith wrote:
> If there was a "central" repository of IP addresses which have
> generated failed login attempts to ANY server, then it seems to me
> to make sense that you could catch them quicker [snip]
> 
> Probably a stupid idea, what with scaling, potential for abuse, etc,
> but may be worth considering for a second or two.

It's not a stupid idea, but it does have serious operational, scalability,
privacy, and security problems.  For example (and this is not exhaustive):

- how do we decide who gets to submit data?

- how do we know what they're submitting is valid?

- how do we anonymize the data (which is a much more difficult
problem that it might first appear)?

- what happens when botnet operators decide to have some fun with this
and generate failed 6B login attempts to 50M servers from 400M addresses?

- how do we query this repository?

-how and when do we remove entries from it?

And so on.

This also presumes that your attackers are my attackers for many
values of {you, me}.  And sometimes that's true.  But quite often
it's not.  I think an operation-specific approach which combines
(a) a priori knowledge of where legitimate login attempts will
originate and (b) logs of failed logins to all servers/services
with the operation is probably more useful.  I've built many
variations of such things, and one of their useful properties
is that they can be deployed "untuned", i.e., as a rough draft,
and then gradually refined over time as experience accumulates.

---rsk

---

From: Chris Lewis <[email-masked]>
Date: Sun, 6 Dec 2015 20:16:23 -0500

> -how and when do we remove entries from it?

Heh, I never intending on doing this.  But...

What makes you think it has to be just one?

We're running one, specifically oriented towards blocking auth-cracking 
spambots.  I would assume that most of us are all interested in blocking 
spambots, so our "attack" will be other's "attack".

We've been running one with a modest number of large scale providers 
using it in production for quite a while.  It is already preloaded with 
auth-crack-capable bots (which is a byproduct of other systems), but the 
wider intel we can get on plain failures, the better we can do with 
extended heuristics based on AUTH-time behaviour - that was always our 
intent.

Normal scale usage would involve rsync transfers.  But small-scale 
experiments can be done by DNSBL query if there's a need.

Usage is, and will remain, free based upon the ability to usefully 
contribute data to it, or to other of our other projects.  You don't 
need to be able to feed data back right away, but that is the expectation.

This is primarily of interest to people running MSA farms.  If you're 
interested in trying it out, please drop me a note off-forum with an 
idea of how you want to try/use it, and I'll fill you in on the details.

---

From: Ned Freed <[email-masked]>
Date: Sun, 06 Dec 2015 18:23:25 -0800 (PST)

> If not, then why not firewall it out?

As long as it's not affecting network or system performance - and it isn't -
it's not worth the bother. In fact I'd rather know what the attackers are up
to. Of course this is just me; YMMV.

I do occasionally get hammered, but that's usually just unauthenticated
SMTP traffic. In those case I put in a block.

> AUTH request from Dubai.)

Sure.

> ranges and is updated regularly.

Good to know.

				Ned

---

From: Rich Kulawiec <[email-masked]>
Date: Fri, 11 Dec 2015 14:35:02 -0500

> appropriate action.  Only if the answer is "they don't take action" would

I'm going to try to be kind here, but I'm probably not going to succeed.

Because the sad answer, in 2015, is almost always "they don't take action".

If all email providers were behaving in a professional, ethical,
responsible fashion -- which includes, among other things, individually
answering and acting on every single message sent to their "abuse"
address -- then we could give this serious consideration.

But they're not.  They haven't been for years (or in some cases, decades).
Oh, they're very concerned with not letting spam in -- to the point
where I see FPs or reports of FPs from them on a near-daily basis --
but they're apparently far less concerned with the spam they let out.

Which means that most of us trying to defend ourselves from the spam
cannons operated by Amazon (for example) don't even bother sending abuse
reports any more.  After years without so much as an acknowledgement
(let alone a polite thank-you for our assistance and a meaningful apology
for the abuse we've suffered at their hands) what would be the point?

	[ Note that, for example, Amazon AWS doesn't even accept abuse
	reports except through a byzantine process using a horribly-designed
	web form.   And then, as far as I can tell, it discards them. ]

Instead we have come to rely on numerous techniques (such as those
Chris Lewis and others have described) that in turn rely on scraps of
information -- including submission IP addresses.  Or we just block
them outright, which is something I've done with AWS.

This is clearly suboptimal.  I really don't want to do it.  But the
refusal of so many large operations to run themselves properly has
made it necessary.

---rsk

---

From: Ted Lemon <[email-masked]>
Date: Fri, 11 Dec 2015 20:19:31 +0000

> address -- then we could give this serious consideration.

This practice has been abandoned because it is impossible to do, not because people are bad.   The system has a massive scaling problem.   DOSing an admin problem report mailbox is trivially easy, and it's automatic, because spammers always send spam to known addresses like this.   So how on earth _could_ any operator of mail service behave in what you are describe as a "professional, ethical, responsible fashion?" 

If the mail architecture is set up in such a way that people cannot operate it according to the specifications, is that a problem with the people, or with the architecture?

---

From: Brandon Long <[email-masked]>
Date: Sun, 13 Dec 2015 20:43:51 -0800

On Dec 11, 2015 12:19 PM, "Ted Lemon" <[email-masked]> wrote:
> This practice has been abandoned because it is impossible to do, not
because people are bad.   The system has a massive scaling problem.
 DOSing an admin problem report mailbox is trivially easy, and it's
automatic, because spammers always send spam to known addresses like this.
 So how on earth _could_ any operator of mail service behave in what you
are describe as a "professional, ethical, responsible fashion?"
>
> If the mail architecture is set up in such a way that people cannot
operate it according to the specifications, is that a problem with the
people, or with the architecture?

Manual abuse complaints and handling have been mostly replaced with
automated handling involving specific report formats (ARF) from
authenticated trusted parties (FBLs).   It is true these systems aren't
designed to say thanks.

I'm unclear how any of this is relevant to this discussion.  Our automated
systems can obviously pull the relevant IPs from encrypted headers.  We can
also make them available to law enforcement following legal procedures.

Brandon

---

From: Rich Kulawiec <[email-masked]>
Date: Wed, 16 Dec 2015 07:17:47 -0500

> are describe as a "professional, ethical, responsible fashion?"

It's really not that hard: in fact, the bigger the scale, the easier
it is -- because there are more computing and personnel resources
available to deal with it.  And the bigger the scale, the more important
it is that this be done: a high-powered distributed mail system can
do a lot of damage very quickly, particularly to smaller operations.

With great power, comes great responsibility.

Solutions for the meta-abuse problems associated with role accounts
(not just "postmaster", but "abuse", "webmaster", etc.) are well-known
and can be adjusted to fit any environment.  I've deployed some of
them, singly and in combination, and in practice with some gradual
fine-tuning over time they work.  Not perfectly, of course -- hence
the need for ongoing fine-tuning -- but they do work.

[ If you'd like me to explain some of those in some detail,
contact me off-list.  We're already wandering somewhat and
this message is long enough. ]

So that's not the problem.  The problem is grasping that it's an obligation
and that you [rhetorical you, I don't mean to pick on you personally]
don't get to just blow it off just because it's nontrivial.  If it's
too hard to run your service properly, then do the Internet a favor and
*shut it down*.  You won't be missed, and you'll easily be replaced.
Nobody is so precious, nobody is so important, nobody is so special,
that they get a pass on this: not an operation with 5 users, not one
with 500 million.  RFCs are not written with exceptions for prominence
or size: the rules apply to everyone.  If you *choose* to participate,
then you have also *chosen* to play by the rules -- and to properly
discharge your fundamental responsibilities to the entire rest of
the Internet.

> people, or with the architecture?

Oh, people can operate it that way: they're just choosing not to
because it's convenient and cheap -- the costs are borne by the
entire rest of the Internet, in the same way that pollution-spewing
industries impose their costs of operation on everyone else.

That's why, for example, some of the largest sources of spam that
makes it past my defenses are MAGY.  They boast constantly about
how much they do to defend their users, and that's all fine and good,
but what are they doing to defend the rest of the Internet?  And how
good a job can they be doing if they're sticking their fingers in
their ears in order to avoid hearing reports of abuse?

Here's a quick-and-dirty breakdown of spam received (e.g., not blocked)
at my primary addresses thus far in 2015:

	count	source
	-----	------
	157	outlook.com/hotmail.com
	19	aol.com
	104	google.com
	1021	yahoo.com
	1301	total over MAGY
	-----	------
	2245	1495 other assorted sources, mostly snowshoe
	-----	------
	3546	total

So, roughly 1/3 of the 2015 spam that was addressed to me and made it past
my filters came from MAGY -- none of whom have answered an abuse complaint
this decade.

---rsk

---

From: Ted Lemon <[email-masked]>
Date: Wed, 16 Dec 2015 17:40:16 +0000

> at my primary addresses thus far in 2015:

I typically get about 200-300 per day pre-filter (I don't have any blocks in place, for science), and very little of that comes through MAGY.   So the only conclusion I think you can realistically draw from this analysis is that MAGY are a spam problem for you because of the positive reputation that they have: mail from them is so much more likely not to be spam than to be spam that the occasional spam message that comes through them gets a higher score than it should.

This would seem to be an indication that they are spending their admin money wisely, not foolishly.   Yahoo, the worse offender, is delivering about three spam messages a day to you.   That's _minuscule_.

---

From: Chris Lewis <[email-masked]>
Date: Wed, 16 Dec 2015 18:57:58 -0500

> Wednesday, Dec 16, 2015 7:17 AM Rich Kulawiec wrote:

Compared their inboxes in terms of MAGY.

Every inbox and every receiving infrastructure is different in terms of 
spamload.

Comparing individual maiboxes is an exercise in futility.

Even I, who have 150 million spams per day across dozens of spamtraps 
representing 10s of thousands of domains won't generalize to this 
extent.  Because they're all different.

I do a daily tuning run on these traps.  The daily tuning run is to find 
out which spams evaded the "SMTP client delivering to me is infected 
with <botname X>" heuristics so I can tune them.  It does not trigger on 
real MTAs for obvious reasons (including MAGY's).

The "SMTP client delivering to me is infected" catches, right now, 
better than 95% of all inbound.  So I'm only looking at 5% of the flood. 
  Of that 5%, at most about 20% is MAGY.  Usually <10%.

So at least on one trap cluster (~35M/day) MAGY isn't a big deal.  At 
most about 1-2% of the total flow.  Does that generalize everywhere? 
No.  But I think it's a better measure than _one_ mailbox.

In the meantime, my personal account is unfiltered, similarly "for 
science".  More than 50% of all spam that I receive in total is spamming 
"[email-masked]" - an account that was created JUST for being on this 
mailing list less than a month ago.

IOW: the IETF is responsible for greater than half of my spam.

[Back in the days when I ran the spam filters for a large corporate, 
every user who received more spam than I (~200/day) were frequent fliers 
on IETF mailing lists.  Including the poor sod getting 1600/day.  IETF 
mailing lists considered harmful.]

---

From: Ted Lemon <[email-masked]>
Date: Thu, 17 Dec 2015 04:15:11 +0000

> Comparing individual maiboxes is an exercise in futility.

Yes, exactly.   Sorry if that wasn't clear from my response.  I by no means think my mailbox is a representative sample.

> I do a daily tuning run on these traps.  The daily tuning run is to find out which spams evaded the "SMTP client delivering to me is infected with <botname X>" heuristics so I can tune them.  It does not trigger on real MTAs for obvious reasons (including MAGY's).

Do you have a botnet farm for harvesting fresh templates, or is there a place where this information can be had?

> In the meantime, my personal account is unfiltered, similarly "for science".  More than 50% of all spam that I receive in total is spamming "[email-masked]" - an account that was created JUST for being on this mailing list less than a month ago.
> 

Hm, so I think what you are saying is that either spammers subscribe to the IETF mailing lists and deliberately target us for spam, which seems unlikely, or they get addresses to try by scraping web sites, and since IETF publishes unredacted email addresses on our mailing list archives, these can be scraped by spammers.

I assume that you are exploiting this!