Subject: RE: [Asrg] 7. Best Practices - DNSBLs - Article

From: Yakov Shafranovich <[email-masked]>
Date: Mon, 11 Aug 2003 15:14:14 -0400

This is something that should be reflected in the BCPs for mail 
administrators - investigate the blacklist before you use it.

At 08:30 AM 8/11/2003, Alan Monaghan wrote:
>So don't use their list. All of the block lists have problems. Some worse
>than others.
>If Spews did give any contact information, they would be out of business in
>a week as people try to take them to court for any number of things just to
>make them spend all their money and time.
>The Register Article did not present any balance in the information nor did
>it show that the ISP in question is a problem for everyone.
>Spews is working the only way it can right now. Again, if it doesn't work
>for you, don't use it.
>
>
>  -----Original Message-----
>From:   Yakov Shafranovich [mailto:[email-masked]]
>Sent:   Sunday, August 10, 2003 6:00 PM
>To:     [email-masked]
>Subject:        [Asrg] 7. Best Practices - DNSBLs - Article
>
>Here is a Register article describing a site being blacklisted by SPEWS.org
>and how they are dealing with it:
>
>http://www.theregister.co.uk/content/6/32231.html
>
>Also see this Politech Bot post:
>
>http://www.politechbot.com/p-05013.html
>
>The site in question was blacklisted for using an ISP that has been known
>for hosting spammers in the past. SPEWS.org does not provide contact
>information nor are their operations audited by anyone.
>
>The problem with SPEWS.org specifically and other similar initiatives, is
>that they do not provide contact information and there is no way to gauge
>whether their internal operations are run properly. This is a problem with
>all anti-spam tools and is something that should be addressed in the BCP
>area. Additionally, mail administrators should be cautious when selecting
>or relying on a specific tool (this also goes into BCPs). Within the
>consent framework, these are SOURCE TRACKING COMPONENTs which should be
>selected carefully.
>
>Yakov

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Brad Knowles <[email-masked]>
Date: Mon, 11 Aug 2003 22:32:58 +0200

>  administrators - investigate the blacklist before you use it.

	Indeed, but we need better investigative tools.  Tools that can 
give us a more authoritative view on whether or not a particular 
black list actually adheres to the rules they espouse, or if maybe 
they are revenge-based, or whatever.  Tools that tell us if certain 
black lists import data wholesale from other sources, without 
checking that first.

	I'm not proposing any solutions per se, but it would be nice if 
we could have some sort of a rating system that we could apply to 
various aspects of all black lists, and on which trusted members of 
the community could then vote.

	Maybe an Advogato-like solution?  Or perhaps one that is more 
broad-community based and less small-trusted-group based, such as the 
Amazon or imdb.com rating systems?

-- 
Brad Knowles, <[email-masked]>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

---

From: Markus Stumpf <[email-masked]>
Date: Mon, 11 Aug 2003 22:44:08 +0200

> administrators - investigate the blacklist before you use it.

Don't know if this qualifies for a BCP ...
We use our own DNSBL. We do also use other DNSBLs but the messages are
tagged only. This tagging is used as a decision aid:
  if we get spam and the IP is listed in other DNSBLs as well, we add
  it to your own DNSBL right away.
this saves some work.

(I have no concrete proof for that, maybe others can check some data and
see if they notice something similar)
I have the impression DNSBLs get abused for finding open relays.
Motivation: checking IP  ranges for relay open mailserver or open
   proxies is a time consuming task and you need email addresses to
   receive the relayed messages at.
Strategy: Do port scans for hosts listening on port 25. Make a list of
   all those IP Addresses.
   Now use a tool that abuses 0wned hosts and do distributed sumbissions
   of these IP's to the various DNSBLs for testing.
Result: After some hours/days do DNS lookups and see if the IP got
   blocked. If it is blocked the IP is vulnerable, add it to your list
   of vulnerable host and qickly start abusing it.
Downside: they use a blacklisted IP for spamming. However from my
   personal observation most DNSBLs are used for informational purposes
   and not for hard blocks, so the amount of spam they get through is
   large enough and pays off the gain of having others do the dirty work
   (with a white hat on).

I had this idea, when I installed a new machine in an old IP block on a
IP that hasn't been in use for at least 6 years. After a few minutes I
saw the first port scan on port 25 and about an hour later there was a
second one. Only 4-5 hours later the host was scanned by three DNSBLs.
IIRC the first scan was from a ATT dialin and the second one was from a
RoadRunner IP.

	\Maex

-- 
SpaceNet AG            | Joseph-Dollinger-Bogen 14 | Fon: +49 (89) 32356-0
Research & Development |       D-80807 Muenchen    | Fax: +49 (89) 32356-299
"The security, stability and reliability of a computer system is reciprocally
 proportional to the amount of vacuity between the ears of the admin"

---

From: Brad Knowles <[email-masked]>
Date: Tue, 12 Aug 2003 00:32:08 +0200

>    it to your own DNSBL right away.

	How do you determine if you get a "spam" from a particular IP 
address?  Is this done in any automated way, or just any time a user 
complains?

>  I have the impression DNSBLs get abused for finding open relays.

	Some do.  However, I suspect that it's mostly the ones that are 
oriented towards open relays anyway, and for them would it really be 
"abuse" in that sense?

>     of these IP's to the various DNSBLs for testing.

	You mean like <http://www.ordb.org/faq/#test_no_list> or 
<http://dsbl.org/programs>?

>     of vulnerable host and qickly start abusing it.

	Already being done, I'm sure.  I figure there are also a lot of 
people out there who are querying the various black lists with 
relatively random selections of IP address ranges (and using various 
open caching/recursive nameservers), to fill in their list of 
potential servers to abuse.

>     large enough and pays off the gain of having others do the dirty work

	Here's another idea.  Take the list of the sort you mention 
above, then front that with a DNS-based load-balancing program.  When 
a spammer looks up the address to spam through, they get re-directed 
to one of millions of vulnerable machines, then drop that connection 
after sending a small number of messages, and start the process all 
over again.

---

From: Markus Stumpf <[email-masked]>
Date: Tue, 12 Aug 2003 17:53:27 +0200

> address?  Is this done in any automated way, or just any time a user 

It's mainly done on spam email we receive ourselves. It's done
semiautomatic from logfiles catching addresses like
    [email-masked]
    [email-masked]
    [email-masked]
    [email-masked]
    [ ... ]

> <http://dsbl.org/programs>?

Not exactly. I think most DNSBLs check submissions and if there are
some 100 submissions for the same subnet from the same IP they might
block it. So use a proxy on 0wned hosts and distribute the submissions
over a wider range of hosts.

> after sending a small number of messages, and start the process all 

I have the impression if they start to abuse a host they stick to it
as long as possible, because after they start to abuse the host chances are
high that the server might get fixed and they have to go for a new one.

	\Maex

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 12 Aug 2003 15:57:29 -0400

Brad Knowles wrote:

>     Indeed, but we need better investigative tools.  Tools that can give 
> us a more authoritative view on whether or not a particular black list 
> actually adheres to the rules they espouse, or if maybe they are 
> revenge-based, or whatever.  Tools that tell us if certain black lists 
> import data wholesale from other sources, without checking that first.

Given the variety of ways that DNSBLs are built, there's really no tools 
that can do it.  Nor is it necessary.

A properly built filtering system will tell you right away if something 
is awry.  For example, our filtering system rejects (not bounces) back 
to the sender.  The sender can contact our false positive handling 
process, and we can investigate.

We know _exactly_ how each of the DNSBLs we're using work - their 
effectiveness ratings, FP ratings, because we see and measure all of it.

>     I'm not proposing any solutions per se, but it would be nice if we 
> could have some sort of a rating system that we could apply to various 
> aspects of all black lists, and on which trusted members of the 
> community could then vote.

There are a number of "ratings" services already that can be help with 
the initial choice.  There's no excuse, however, in operating any 
meaningful size filtering system without being able to determine the 
pros and cons on your own.  Caveat emptor.

I should also point out that each site will have its own requirements. 
Forcing some sort of "ratings" system on them will disenfranchise 
someone.  If the site wants to use a blacklist that blocks 200/8, they 
should be able to.

[We have a DNSBL BCP which I should publish soon.]

---

From: Brad Knowles <[email-masked]>
Date: Wed, 13 Aug 2003 01:17:20 +0200

>  effectiveness ratings, FP ratings, because we see and measure all of it.

	Not true.  You know how the first one in your list is operating, 
at least to the level of rejections that result from it, and then 
complaints that come when when those rejections appear to be 
inaccurate.

	You don't know how the next one would have done, if the first one 
hadn't blocked that message.  You don't know how many false negatives 
made it through.  And of the positives, you don't really know how 
many are true and how many are false.

	You would have to look up each address in all black lists (noting 
both hits and misses, as well as the time of the lookup) before 
making your decision, and you would have to use other means to 
investigate the negatives that make it through the system and re-look 
them up again at various recurring periods in time (to see if they 
were added to the list after the message got through your system).

	Then you would have to track which black lists result in the most 
positive hits with the lowest false negative and false positive 
ratios.

	In other words, you'd have to look up every single address that 
ever contacts your mail server (including ones that may get rejected 
for other reasons before they would normally hit the black list 
checks), in all black lists, re-query on a periodic basis, and track 
all hits and all misses for all IP addresses, ad infinitum.

	I'm pretty sure you don't do that.  I've made an attempt at doing 
that for ntp.org, at least in terms of ordering the black lists that 
we do use.

>  There are a number of "ratings" services already that can be help
>  with the initial choice.

	Really?  Where are they?  I'd like to make use of them....

>  [We have a DNSBL BCP which I should publish soon.]

	I look forward to seeing it.

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 12 Aug 2003 20:49:38 -0400

At 07:17 PM 8/12/2003, Brad Knowles wrote:
> complaints that come when when those rejections appear to be inaccurate.
> made it through.  And of the positives, you don't really know how many 
> are true and how many are false.
> both hits and misses, as well as the time of the lookup) before making 
> your decision, and you would have to use other means to investigate the 
> negatives that make it through the system and re-look them up again at 
> various recurring periods in time (to see if they were added to the list 
> after the message got through your system).
> positive hits with the lowest false negative and false positive ratios.
> ever contacts your mail server (including ones that may get rejected for 
> other reasons before they would normally hit the black list checks), in 
> all black lists, re-query on a periodic basis, and track all hits and all 
> misses for all IP addresses, ad infinitum.
> that for ntp.org, at least in terms of ordering the black lists that we do use.

Chris, can you provide some information on DNSBLs and how your system 
evaluates their effectiveness?

>         Really?  Where are they?  I'd like to make use of them....

Some information in this area would be useful as well.

Yakov

---

From: "Chris Lewis" <[email-masked]>
Date: Wed, 13 Aug 2003 16:34:19 -0400

> evaluates their effectiveness?

Taking the second part first:

1) Our false positive process involves having the sender contact a 
specific address, and that will contain a "block ID" from the message 
rejection.  Using the block identifier, we can retrieve the message, and 
identify _why_ it was blocked (may not be DNSBL).  Then, if it's a 
DNSBL, we can locally whitelist the IP.

2) Our metrics scan the logs and identify block/unblock counts for all 
messages by IP.  During metrics processing, we generate tables of how 
much email a given blacklist blocked, _plus_ how many email messages 
_would_ have been blocked by that blacklist, but weren't because they 
were whitelisted.

Thus, for each blacklist, we generate:

a) how many emails were blocked, and the percentage of total inbound it 
represents.
b) how many emails had to be whitelisted for that blacklist, and what 
percentage of the emails blocked by that blacklist it represents.

Note that the latter (b) isn't _really_ a "lost email" metric (to us). 
Those messages weren't lost, they got through because we whitelist.  But 
it's a good FP metric.

Basically coming up with a "blocking percentage" and a "whitelisted" 
percentage for each blacklist.

Here's our overall effectiveness metrics - last two week period - 
percentages are of _total_ email to our inbound mail servers.

This is meant merely as a demonstration of what significant site 
_should_ be doing with metrics from their own filtering, not an 
exhaustive list of all DNSBLs, or what others see.

Due to the variety of DNSBLs, it's very difficult to generate overall 
metrics for all.  Without zone transfering the DNSBL for example, it's 
too slow to measure it with reasonable sample sizes.

The counts are _all_ total recipients affected.  So, a single email with 
5 recipients counts as 5.

Rule                        count     %       FPs  % of blocked
CBL                        352153  16.86       7   0.0020
MONKEYPROXY                261436  12.52      83   0.0317
CONTENT                    260391  12.47     N/A
BOPM                       167277   8.01       0   0
OSsocks                    152168   7.29       0   0
SPEWS (not used)            94285   4.51     N/A
NTthresh                    54233   2.60      27   0.0498
SBL                         47614   2.28     156   0.3276
NTmanual                    45366   2.17    2955   6.5137
NTauto                      25094   1.20      12   0.0478
OBproxies                   21068   1.01      18   0.0854
PDL                         16547   0.79     N/A
NTnordns                     5814   0.28       0
Flonetwork                    229   0.01       0

[The below numbers are whitelisted recipients per associated BL, as a 
percentage of the total inbound, not of blocks per-se.  "OK" is _total_ 
whitelisting entries, whether or not the whitelisting is still necessary]:

OK                          36770   1.76
OK SBL                        156   0.01
OK OBproxies                   18   0.00
OK NTthresh                    27   0.00
OK NTmanual                  2955   0.14
OK CBL                          7   0.00
OK NTauto                      12   0.00
OK MONKEYPROXY                 83   0.00

TOTAL                     2088400 100.00
TOTAL BLOCK                571850  27.38

Ie: out of 2088400 attempted deliveries, 352153 (16.86%) were blocked by 
the CBL.  Only 7 messages (0.0020% of 352153) would have been blocked if 
we weren't whitelisting.

Some notes:

"CONTENT" is the union of all of our non-DNSBL based.  Our system 
doesn't let us compute FPs on that directly.

BOPM, Monkeys, OSSocks, OBproxies are third party open proxy/socks 
blacklists.  The first three are public.

CBL is more-or-less an open proxy/socks blacklist (see cbl.abuseat.org)

SBL is Spamhaus.

NTmanual is local manual black/whitelist (the high FP rate is perfectly 
okay because those are whitelisted, and it's under our control)

NTnordns is "spam over threshold, no rDNS" (AOL does this now)

NTthresh is "spam over threshold, listed in ORDB or SpamCop"
NTauto is local open relay/proxy detector.

Our system doesn't let us compute FPs on BLs we don't use, so we don't 
have the number for SPEWs.

PDL (DHCP pools) does block occasional legitimate email, but, by policy 
we don't want email from DHCP pools.  So we don't whitelist it - we get 
the sender to either use their proper mail servers, or get PDL to update 
the entry if it's erroneous.  Whitelisting would be silly if the entry 
was accurate, because the IP can essentially be random.

Flonetwork: zone listing doubleclick/dartmail/flonetwork.  No, we 
_really_ don't want email from them, period.

If someone wants subjective commentary on the public DNSBLs (ie: 
proposed BCP compliance), let me know.

> Some information in this area would be useful as well.

there's been a few of those posted.