Subject: [Asrg] Some statistics on SPF and spam

From: Martijn Grooten <[email-masked]>
Date: Tue, 12 Feb 2013 11:01:16 +0000

I had promised to produce some stats on SPF and spam.

Over the Christmas holidays, I sent over 60k spam messages through 21 spam filters in the spam-filter test I run regularly. I checked the SPF status of the messages and measured how many filters failed to block each message.

Here are the results:
SPF fail: 3171 emails, on average missed by 0.24 filters (out of 21) with a standard deviation of 0.04.
SPF pass: 8106 emails, avg 0.93, stddev 0.23
SPF softfail: 8672 emails, avg 0.45, stddev 0.09
SPF neutral: 13466 emails, avg 0.34, stddev 0.04
SPF none: 26938 emails, avg 0.43, stddev 0.06

A neater table and a graph can be found here: http://www.virusbtn.com/news/2013/02_04.xml

Now correlation doesn't imply causation and there are good reasons why the relationship here may not causal, but let's for a moment we assume it is.

This means that if you're a spammer, failing SPF isn't a good idea, while making sure your emails pass SPF means you're more likely to see your messages delivered, but you by no means get a free ride to users' inboxes.

If you find a 'clever' way to avoid failing SPF by using a domain with no SPF record, there is only a small improvement in your delivery rates.

Martijn.

________________________________

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

---

From: [email-masked]
Date: Tue, 12 Feb 2013 14:55:49 -0500

I didn't see the discussion where you promised to produce this, but I think
the problem is how much non-spam also fails SPF.  

>From ruleqa.spamassassin.org/?daterev=20130211-r1444680-n&rule=%2Fspf :

  MSECS    SPAM%     HAM%     S/O    RANK   SCORE  NAME   WHO/AGE
      0   0.0236   0.9635   0.024    0.15    0.00  SPF_FAIL  
      0   0.0383   0.3059   0.111    0.27    0.00  SPF_SOFTFAIL  

Way more non-spam is failing than spam.

Catching spam is easy.  Doing so without excessive false positives is
what's hard.

On 02/12, Martijn Grooten wrote:
> _______________________________________________
> Asrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/asrg
> 

-- 
"Begin at the beginning and go on till you come to the end; then stop."
- Lewis Carrol, Alice in Wonderland
http://www.ChaosReigns.com

---

From: Dave Warren <[email-masked]>
Date: Tue, 12 Feb 2013 13:41:41 -0800

> Way more non-spam is failing than spam.

If you just use SPF for positive scoring and never for negative scoring 
or blocking then that's okay.

I'm also note suggesting that SPF or DKIM or similar alone is sufficient 
for positive scoring, but when combined with a local whitelist, I can 
aggressively whitelist companies that we do business with without having 
to worry about a spammer spoofing a whitelisted major corporation.

When the company starts sending mail from a non-listed IP, they don't 
get the benefit of whitelisting, but nothing else "breaks", so there's 
no harm done.

> Catching spam is easy.  Doing so without excessive false positives is

Amen.

I guarantee you that I can block every single spam, 100% of the time, no 
questions asked, as long as one of the unasked questions is the false 
positive percentage.

-- 
Dave Warren
http://www.hireahit.com/
http://ca.linkedin.com/in/davejwarren