Subject: RE: [Asrg] 7. Best Practices - DNSBLs - Article

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 12 Aug 2003 15:50:11 -0400

At 12:48 PM 8/12/2003, Jason Steiner wrote:
>"Yakov Shafranovich" <[email-masked]>
> >
> > Any kind of auditing or control over the inner procedures of a black
> > list would improve the situation.
>
>I dispute this. Blocklists have reputations, even if that reputation is
>no reputation at all, and anyone who does a minimal amount of
>research can find out what that reputation is. If you don't like a
>blocklist's reputation, don't use it.

The problem is that mail administrators are so fed up with spam, they 
choose blacklists based on their effectiveness, not reputation. The more 
IPs a blacklist blocks, the more effective it is but it is hard to do an 
evaluation for an ISP on how much innocent email is being blocked. It would 
be much easier if ISPs would flag messages like SpamAssasin does with input 
from DNSRBLs and then let the users decide. Within the consent framework 
relying on a single source of information is not a good thing. The best 
thing is using multiple sources of information and then letting the consent 
system decide what do with it.

Mail administrators should be aware of the need to investigate blocklists - 
this is something for the BCPs. Also, shedding some light on the inner 
working of a blocklist such as SPEWS.org, by a third party might make it 
easier for mail administrators to evaluate it.

>Do we really need an auditing organization to tell us that SPEWS
>has no contact information and lists more than just the spammer?
>Of course not. So what is the real goal of proposals for auditing
>organizations?
>
>To make sure that every blocklist follows the same standards, which
>just so happen to be the preferred standards of the person making
>the proposal! But that defeats the whole purpose of having multiple
>blocklists. And it would be far easier to start a new blocklist that
>actually embodied those standards than it would be to start a
>organization intended to enforce those standards on other blocklists
>that might well disagree with them.

The purpose of auditing is to make sure that the blocklist procedures 
listed on their website are actually being followed. That's all - all we 
need to know is whether SPEWS.org or some other DNSRBL is actually listing 
and de-listing IPs based on the criteria that is mentioned on their 
website. And if there is no such criteria, then the DNSRBL in question can 
write one based on whatever philosophy they want. Currently many people do 
not trust a closed no-contact DNSRBL to actually follow their own 
procedures, but rather believe that a lot of listing/delisting procedures 
are being done on a whim.

We do not need to make standards for blocklists, however a small BCP might 
be useful with things like a need to state clearly what your 
listing/delisting policy is, etc. Most of the BCPs for DNSRBLs would center 
around the need to state what is actually going on in inside one, not 
actually trying to enforce specific policies. Additionally, support for 
consent systems and associated protocols might be useful as well.

> > A rating system would not be sufficient since it has potential for
> > abuse. We would need someone to audit the blacklists,
> > something like the Truste seal program for privacy. Or perhaps
> > just a reliable auditor like a major accounting or security company,
> > maybe even ISO 9002 (although with Enron and Arthur Andersen
> > story that might not be reliable enough).
>
>Exactly. And Truste's reliability isn't all that great either.

I know, it was just a thought.

> > All we need is that a third impartial party has examined the procedures
> > of  the blacklist. Unfortunatly that is not being done today.
>
>Where are you going to find a truly impartial third party, and who's
>going to make sure that they stay impartial? Where will we find these
>angels to rule us?
>
>I submit that any anti-spam proposal that requires some centralized,
>incorruptible, and totally impartial authority is fundamentally unworkable.
>They do not exist, and even if they did, a centralized organization would
>be too big a target to escape eventual corruption.

I agree with you - I am not seeking a centralized organization with "one 
ring to rule them all". Nor are there impartial third parties. I guess I 
went a bit overboard with this - but my main point is:

If you operate a DNSRBL, then disclose your inner procedures and actually 
follow them.

IF for whatever reasons you do not want to do so, then have someone else 
audit your procedures and let them vouch for you.

Yakov

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked] (Justin Mason)
Date: Tue, 12 Aug 2003 13:12:17 -0700

> consent systems and associated protocols might be useful as well.

BTW is there a BCP for measuring spam filter effectiveness -- e.g. stating
that effectiveness should be measured in terms of portable numbers like

  FP% = (false positives as a percentage of nonspam mail)
  FN% = (false negatives as a percentage of spam mail)

?  This would be valuable.

I've found that often, DNSBL effectiveness is stated as "it blocked this
much spam" with no discussion of false positive or negative rates.  Once
the FP% and FN% are measured, the picture becomes a lot clearer ;)

--j.

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 12 Aug 2003 16:20:56 -0400

>the FP% and FN% are measured, the picture becomes a lot clearer ;)

This can be expended to measuring the effectiveness of any anti-spam tool 
(within the consent framework this would apply to source tracking 
components and consent systems themselves).

If you or anyone else is willing to work on this, go right ahead, just let 
the chair know.

Yakov

---

From: [email-masked] (Justin Mason)
Date: Tue, 12 Aug 2003 13:41:12 -0700

> If you or anyone else is willing to work on this, go right ahead, just let 

Ah, time's a problem. ;)  However I'd be happy to help out if anyone is
working on it, at the review/comment/helpful tips stage.

--j.

---

From: [email-masked] (Justin Mason)
Date: Tue, 12 Aug 2003 17:42:43 -0700

We (SpamAssassin) measure FP/FN rates for DNSBLs as part of our rescoring
process.   However the results aren't prettified for the web.  Maybe
sometime soon.

Scroll down on this page:

  http://useast.spamassassin.org/dist/rules/STATISTICS-set1.txt

and take a look at the effectiveness ratings there. e.g.

OVERALL%   SPAM%     HAM%     S/O    RANK   SCORE  NAME
 39.058  56.1503   0.2042    0.996   0.98    1.10  RCVD_IN_DSBL

that means the dsbl.org BL hit 56% of spam, 0.2% of nonspam, which is an
effectiveness rate of 99.6% accuracy.  (RANK and SCORE are irrelevant
for non-users of SpamAssassin.)

in terms of figuring out which
DNSBL is in question, if it's RCVD_IN_FOO, it's generally the DNSBL
called "foo" ;)  Take a look in
http://useast.spamassassin.org/dist/rules/20_dnsbl_tests.cf for more
details.

RAZOR2_CF_RANGE_51_100 means Razor 2, with a "confidence rating" from
teh razor server of between 51 and 100.

We measure these at a point up to 6 months after the mail was received,
so that *may* mean that the BL data is no longer accurate for that mail
(e.g. if a relay is closed.)   But in practice that does not seem to be
a big issue...

What the hell, I'll just paste the ratings here:

OVERALL%   SPAM%     HAM%     S/O    RANK   SCORE  NAME
 495260   343948   151312    0.694   0.00    0.00  (all messages)
 47.918  68.8645   0.3033    0.996   1.00    1.55  RAZOR2_CF_RANGE_51_100
 54.243  77.7344   0.8459    0.989   0.99    0.90  RAZOR2_CHECK
 39.058  56.1503   0.2042    0.996   0.98    1.10  RCVD_IN_DSBL
 31.969  45.9802   0.1209    0.997   0.97    1.10  RCVD_IN_SORBS_HTTP
 27.132  39.0053   0.1441    0.996   0.96    1.10  RCVD_IN_NJABL_PROXY
 23.752  34.1549   0.1057    0.997   0.96    1.10  RCVD_IN_SORBS_MISC
 19.465  28.0124   0.0363    0.999   0.95    1.10  RCVD_IN_OSIRU_PROXY
 15.868  22.8473   0.0040    1.000   0.95    4.30  RCVD_IN_OPM
 12.937  18.6272   0.0026    1.000   0.94    4.30  RCVD_IN_OPM_HTTP
 10.065  14.4926   0.0007    1.000   0.94    4.30  RCVD_IN_OPM_HTTP_POST
  8.070  11.6201   0.0013    1.000   0.93    4.30  RCVD_IN_OPM_SOCKS
  6.901   9.9242   0.0304    0.997   0.92    1.81  DCC_CHECK
  1.104   1.5904   0.0000    1.000   0.92    4.30  RCVD_IN_OPM_WINGATE
  3.903   5.6145   0.0132    0.998   0.92    1.10  RCVD_IN_SORBS_SMTP
  0.029   0.0424   0.0000    1.000   0.92    2.80  RCVD_IN_SORBS_WEB
  0.015   0.0212   0.0000    1.000   0.92    2.80  RCVD_IN_OPM_ROUTER
  0.001   0.0015   0.0000    1.000   0.92    0.00  RCVD_IN_NJABL_MULTI
 12.520  17.9202   0.2439    0.987   0.91    1.27  RCVD_IN_SBL
  1.783   2.5591   0.0192    0.993   0.90    0.53  RCVD_IN_NJABL_DIALUP
  1.022   1.4665   0.0112    0.992   0.90    0.85  RCVD_IN_OSIRU_DIALUP
  0.359   0.0087   1.1546    0.007   0.90   -4.30  RCVD_IN_BSP_TRUSTED
  9.706  13.8724   0.2353    0.983   0.89    0.93  RCVD_IN_OSIRU_SPAMWARE
  0.082   0.0026   0.2630    0.010   0.89   -0.10  RCVD_IN_BSP_OTHER
  9.036  12.9098   0.2320    0.982   0.89    0.64  RCVD_IN_NJABL_SPAM
 41.526  58.5996   2.7169    0.956   0.88    0.10  RCVD_IN_SORBS
  3.435   4.8891   0.1302    0.974   0.85    0.00  RCVD_IN_OSIRU_RELAY
  3.143   4.4719   0.1209    0.974   0.85    2.55  RCVD_IN_DYNABLOCK
 41.161  57.6715   3.6322    0.941   0.84    0.10  RCVD_IN_NJABL
  0.361   0.5108   0.0192    0.964   0.82    2.60  RCVD_IN_SORBS_ZOMBIE
 11.074  15.4265   1.1810    0.929   0.76    0.10  RCVD_IN_RFCI
  3.185   4.4132   0.3945    0.918   0.72    1.31  RCVD_IN_NJABL_RELAY
 38.389  51.8980   7.6802    0.871   0.68    0.00  RCVD_IN_OSIRU
  3.802   5.2075   0.6060    0.896   0.67    0.00  RCVD_IN_SORBS_SPAM
  0.056   0.0718   0.0185    0.795   0.46    1.52  RCVD_IN_SORBS_SOCKS
 11.116  13.7768   5.0683    0.731   0.38    0.00  RCVD_IN_OSIRU_SPAM_SRC
  0.016   0.0180   0.0106    0.630   0.23    0.00  RCVD_IN_NJABL_CGI
  0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_SORBS_BLOCK

--j.

---

From: Brad Knowles <[email-masked]>
Date: Wed, 13 Aug 2003 03:53:28 +0200

>    http://useast.spamassassin.org/dist/rules/STATISTICS-set1.txt

	Interesting.  Taking this data and sorting it by the accuracy, we get:

   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_RSS
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_RBL
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_NML
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_DUL
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_BL_SPAMCOP_NET

	And sorting by the "ham" percentage:

   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_BL_SPAMCOP_NET
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_DUL
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_NML
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_RBL
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_RSS

	And then by score:

   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_RSS
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_RBL
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_NML
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_DUL
   0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_BL_SPAMCOP_NET

	I note that several of the black lists have zero hits on both the 
"spam" and "ham" categories.  Is this because you do not have access 
to them on this machine, and therefore cannot measure their 
effectiveness and accuracy?  Of course, the lists with a negative 
score should be white listed.

	What about measures of unique hits?  I.e., a particular IP is 
found only in a single black list?  Is there some other way to 
measure hit coverage?  It would be interesting to know what the 
minimal set of black lists would be to get the maximum coverage....

	Thanks!

-- 
Brad Knowles, <[email-masked]>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

---

From: [email-masked] (Justin Mason)
Date: Tue, 12 Aug 2003 19:06:32 -0700

Brad Knowles writes:
> effectiveness and accuracy?  

No -- they are commercial.  (the bl.spamcop.net test is semi-commercial,
in that donations are requested).  Since these figures are used in the
SpamAssassin rescore run, we can't use commercial tests since most users
would not be able to enable them.

> Of course, the lists with a negative 

Yes, I should have noted that, they are whitelists.  The "Bonded Sender"
list if I recall correctly.

> minimal set of black lists would be to get the maximum coverage....

It's pretty tricky to figure that out from this table, unfortunately; one
good guideline is, if a score is higher than the accuracy would suggest,
the GA has found it to provide useful "information" (in the Shannon
sense).  If the score has been zeroed, it overlapped too much with other
tests or was massively inaccurate (see RCVD_IN_OSIRU,
RCVD_IN_OSIRU_SPAM_SRC).

We do occasionally run overlap tests, but they're quite heavy tests to
run, since you have to run each rule against every other rule as far as I
can see.

--j.

---

From: Brad Knowles <[email-masked]>
Date: Wed, 13 Aug 2003 11:05:18 +0200

>    0.000   0.0000   0.0000    0.500   0.11    0.00  RCVD_IN_MAPS_DUL

	Thinking about this some more, I notice that the MAPS black lists 
do not appear to be tested at all.  For the sake of comparison, I 
believe that they should be included in the tests and ranked against 
the other black lists, or they should be omitted from the list of 
black lists altogether.

	Moreover, this is less than forty black lists.  My understanding 
is that there are well over a hundred in existence.  This list would 
need to be significantly expanded, in order to cover all known black 
lists and be a more fair comparison.

	I wonder -- have you run this comparison with other spam/ham 
corpii?  Do you continue to expand your spam/ham corpus as time goes 
on?

---

From: Brad Knowles <[email-masked]>
Date: Wed, 13 Aug 2003 11:08:21 +0200

>  would not be able to enable them.

	For your comparisons, I can understand.  However, for the sake of 
the kind of data we're looking for, I believe that they should be 
included in the tests.

>  tests or was massively inaccurate (see RCVD_IN_OSIRU,

	Interesting.  Good to know.

>  run, since you have to run each rule against every other rule as far as I

	Fair enough.  Thanks again!

---

From: Daniel Feenberg <[email-masked]>
Date: Wed, 13 Aug 2003 08:04:52 -0400 (EDT)

> run, since you have to run each rule against every other rule as far as I

That's roughly right. The results given are informative for the selection
of a single DNSBL. If you can use two blacklists, then it the best
combination doesn't necessarily include the best single DNSBL. With 20
lists, you would need to test 20*19/2 possible combinations to pick the
best pair. But that is what computers are for.

Also note that with a pair of lists, you have a choice to block based
on a single hit, or two hits, and if a single hit, it may depend on which
list. So multiply the possibilities by 3.

Daniel Feenberg
feenberg isat nber dotte org

---

From: Brad Knowles <[email-masked]>
Date: Wed, 13 Aug 2003 16:52:49 +0200

>   495260   343948   151312    0.694   0.00    0.00  (all messages)

	Hmm.  This is the total number of messages, and not the total 
number of IP addresses to look up, right?  Do you have any idea how 
many IP addresses there are per message that you have to look up?

	My own spam archive is pretty short.  It only goes back to Sat 
Mar 02 00:16:31 2002, comprises 130,338,812 bytes, contains 49,848 
messages, and has 20,411 unique IP addresses that I've been able to 
find.

	My "ham" archive is quite a bit larger.  Some folders go back as 
far as 1995, it comprises 1,041,157,733 bytes, 506,115 messages, and 
supposedly 64,164 unique IP addresses (I suspect that my command line 
that I used to find the unique IP addresses in the spam archive was 
not able to handle the amount of input from the ham archive).

	I am curious -- is there a reason why you tested with a much 
larger spam archive than your ham archive?

---

From: [email-masked] (Justin Mason)
Date: Wed, 13 Aug 2003 09:56:43 -0700

> many IP addresses there are per message that you have to look up?

The total number of messages -- and I'm afraid I don't have a clue on how
many IPs there are there.  There's about 15-20 individual contributors
behind all that mail, they do not send us the entire mail message itself
during the process, and grepping out that data would be tricky :(

> larger spam archive than your ham archive?

I don't know -- Theo ran the corpus selection ;)   But as far as I know,
no reason -- probably simply that more people submitted results for
spam over ham.   Since the main reason for this was to run the genetic
algorithm, and in our experience the GA is resilient to that kind of
proportion, it wasn't an issue.

> the other black lists, or they should be omitted from the list of 

This is true.   (They are only included in that list because of my
oversight. oops.)

> lists and be a more fair comparison.

Good point...  note that it's actually *under* 40 as many of those
"rules" indicate that a multi-value DNSBL -- like SORBS -- returned
one of its possible values -- like 127.0.0.6 .

Those are just the DNSBLs that we found to provide useful results.  Many
of the others either (a) have unusual listing/delisting criteria, (b) were
not totally free to query, (c) did not want to have thousands of
SpamAssassin users hitting their servers (which is pretty reasonable!), or
(d) didn't make the grade in terms of QA.

In particular, referring back to the "overlap" question -- if I recall
correctly Dan dropped a couple of BLs due to large overlap with the ones
in that list.

> corpii?  Do you continue to expand your spam/ham corpus as time goes 

This comparison is made using ham/spam corpora from about 15 to 20 people.

> 	In terms of analyzing black list performance, all we need is the 
> IP address(es) found in the headers of the message.  Everything else 
> is superfluous, and indeed gets in our way.

FWIW, SpamAssassin 2.60 will include a method for the user/admin to tag
all messages with a header containing the DNSBL lookup results at the
moment the message was scanned -- like so:

  X-spam-rbl: <dns:244.130.118.163.opm.blitzed.org> [127.1.0.4]

Once that's in the field for a while, it'd be easy enough to grep out
those headers and come up with "live" DNSBL performance data, and the
list of IPs.

--j.