Subject: RE: [Asrg] REDIRECt to ASRG RE: Reputation systems

From: "Ryan Malayter" <[email-masked]>
Date: Tue, 17 Aug 2004 11:51:59 -0500

[Pete McNeil]
> Here is a link to the analysis:
> 
> http://www2.spamchk.com/public.html
> 
> Hope this helps,
> _M

This is very useful, and most of what I was looking for, but I'd still
like to see an analysis done with hand-filtered email.

It does appear that "SBL" (which I assume is SpamHaus's SBL) has a very
low false positive rate. However, it also blocked only about 10% of
incoming spam. The XBL caught a lot more, but still less than 65% of the
spam. But this better performance cost nearly 30 false positives (each
of which could lead to a very irate customer for a business or
government organization).

The numbers seem to indicate that IP-based blacklisting is essentially
useless without further content filtering. None even came close to
filtering 75% of spam, and some caused hundreds or thousands of false
positives.

I wonder if the same poor performance will be the output of
domain-name-based reputation systems after MARID's work is widely
deployed.

Do you have a key that explains what each of the test abbreviations are
on this page? Some are obvious, others are not.

Thanks,
	Ryan

---

From: "John R Levine" <[email-masked]>
Date: 17 Aug 2004 13:25:05 -0400

> government organization).

30 out of how many?  People I know who use the XBL and the CBL (the main
source for the XBL) tell me that the fp rate they see is very low.  This
includes people at big companies who depend heavily on e-mail.

> filtering 75% of spam, and some caused hundreds or thousands of false

I hope that doesn't come as a surprise to anyone.  The advantage of DNSBLs
is that they are cheap to use and can reject mail very early in the SMTP
process.  If you can knock out 2/3 of the spam with DNSBLs, you can avoid
2/3 of the expensive content based scoring filters.

> domain-name-based reputation systems after MARID's work is widely

The only honest answer is that nobody knows.  In the forseeable future,
MARID verification will only be useful to whitelist domains known to be
friendly.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Information Superhighwayman wanna-be, http://iecc.com/johnl, Mayor
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: wayne <[email-masked]>
Date: Tue, 17 Aug 2004 23:04:03 -0500

> like to see an analysis done with hand-filtered email.

I agree, this is very useful information.

I found it interesting that, despite several comments to this list,
the SpamCop DNSBL came out looking very good.  It blocked more spam
than any other DNSBL and yet its false positive rate isn't a huge
amount worse than the highly regarded CBL DNSBL.

The SpamCop DNSBL has been around for a long time and was one of the
first automated DNSBLs.  Its algorithm has changed a lot and the
size of the reporting base for SpamCop has increased greatly.  I
wonder how much of the bad reputation is due to old data/experiences.

-wayne