Subject: [Asrg] Collecting IP reputation data from many people

From: [email-masked]
Date: Wed, 20 Oct 2010 21:45:02 -0400

I'd like your thoughts on collecting reputation data (% spam vs. non-spam
originating at every IP) from everyone willing to submit it.

My goal would be to be able to tell anyone how much spam and non-spam any
IP address has sent recently.  And that they could use that information to
aid spam filtration.  

I realize it wouldn't be easy to make this useful, given the interest of
spammers to corrupt the data.  The example of Mizhen vs. Microsoft posted
here a few months ago was excellent.  Also, the availability of captcha
solving for $2 per 1000.  

I realize that the vast majority of "people" submitting data could be
spammer controlled zombies, repeating back to the system that they've
received spam and non-spam from IPs others have reported, and once in a
while many of them could claim an IP is sending lots of non-spam, and no
spam, in the hopes that this system would identify that IP as a
non-spammer, allowing spam from that IP to get through filters.

I hope I'll be able to identify the quality of the data coming from each
person well enough that this would not be a problem.  Maybe a large number
of zombies could cause some spams to get through for one day, and then
be identified as malicious.  I guess that's my biggest question though.
How many times can the spammers use "a large number of zombies" to get
spam through for a day before they run out?  Obviously I'd take advantage
of things like Spamhaus's XBL (list of zombies).  It's also an interesting
problem of lacking known truth - how do you know which people are
reporting good data when everything you have to compare it to might be
maliciously bad data from spammers?

I am not interested in trying to block easily identifiable spam.  I'll
continue to run spamassassin and use DNSRBLs at my MTA, and only report
spam that makes it into my inbox into this system.

I have been involved with dnswl.org (a dns based whitelist used by
spamassassin).  My motivation for trying to collect data from many people
is largely the fact that DNSWL appears to only cover a third of non-spam
(according to http://rulequa.spamassassin.org ).

For implementation details, I'm thinking something like:

  User clicks junk / not-junk button on their MUA, MUA sends user ID
  (randomly generated by MUA once) and sender IP, along with "spam" or
  "non-spam" to my server over an encrypted tcp connection on some port which
  is not currently used.
  
  Mail filters download full list of IPs including number of spams and
  non-spams transmitted by that IP recently, once a day.  Every time an email
  is received, the filter increases or decreases the score of each email
  based on the data in that list.  I'd prefer not to deal with the network
  load of serving this data via DNS, but I haven't attempted to calculate
  the break-even point.

I think basic implementation wouldn't be a lot of work for me.  Set up the
server to accept data, aggregate it, and serve it (http?  rsync?).  Create
modules for spamassassin to report the sending IP via "spamassassin
--report" to report spam (which I did for dnswl), and "spamassassin
--revoke" to report non-spam.  This would take advantage of
internal_networks in the SA configuration to figure out which IP is
actually the sender and not relays internal to the persons email provider,
and trusted_networks to avoid reporting trusted proxies or other external
relays.

So do you think it's worth my effort?  

How would you improve this?

Do you think it could be useful enough that you'd be willing to click a
button to send me data occasionally?

-- 
"Government is not reason, it is not eloquence, it is force; like fire,
a troublesome servant and a fearful master. Never for a moment should
it be left to irresponsible action." - George Washington
http://www.ChaosReigns.com

---

From: der Mouse <[email-masked]>
Date: Thu, 21 Oct 2010 01:05:53 -0400 (EDT)

> I'd like your thoughts on collecting reputation data (% spam vs.
> non-spam originating at every IP) from everyone willing to submit it.
> [...]

You know, I really hate to be a parade-rainer.  I also don't like to
tell anyone to not try experiments.  But I see this as a basically
hopeless task in view of the botnet problem: as soon as it becomes
widespread enough to be noticed, the signal in its input data will be
swamped by gaming attempts.

You are clearly aware of that basic problem, though, so you may be able
to come up with something.  Indeed, I hope you do; anything capable of
resisting such gaming while still doing any kind of crowdsourcing is
interesting for that alone.

I would warn you against inferring from successful preliminary tests
that it would be equally successful if widespread; there are zillions
of techniques that work fine provided few enough people do them that
spammers don't notice or don't think they're worth bothering to
circumvent.  (I depend on a few myself.)

So, sure, go ahead and try it.  Even if it's a failure in the form you
now envision, something useful may end up coming out of it.  Research
is like that.

> So do you think it's worth my effort?  

Dunno.  I don't think I'd put effort into it myself, but that doesn't
mean much in view of how large my to-do list of things I'd enjoy much
more is.  You sound fired up, though, and I suspect something valuable
may well come out of it even if it's not what you intended when you
started.  So I'll go with "probably" here.

> How would you improve this?

I have basically no idea, and wouldn't until the experiment had
progressed significantly.

> Do you think it could be useful enough that you'd be willing to click
> a button to send me data occasionally?

Yes, except that (a) my MUA doesn't involve "clicking" "buttons", (b) I
doubt human-generated data from me would be of much use to you (most
spam aimed at me never makes it past my SMTP listener), and (c)
automated data from me would probably be even worse because I block a
nontrivial amount of ham because it comes from actors whose level of
abuse I am not willing to tolerate even to the point of accepting their
ham (eg, Google, Yahoo) or it is constructed in sufficiently broken
ways that I consider rejecting it a public service even if it's not
spam (eg, marked as 8859-1 but containing octets in the 0x80-0x9f
range, or with a Message-Id: that does not conform to the syntax).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 21 Oct 2010 10:53:22 -0400

> circumvent.  (I depend on a few myself.)

I think it's worth pointing out that _everybody_ does to one extent or 
another.  Anti-spam is like security in general.  No protection is 
absolute.  You're always relying on setting the bar high enough to not 
being worth the ROI.  Part of that equation is trying to avoid being too 
much alike someone else who it _is_ worth bothering with.

IOW: don't make yourself too much similar to Hotmail, Gmail or AOL, 
because the incentives to break them are far higher than "little ol' 
me", and if they get broke, you're toast.

I'm with der Mouse.  I'm not sure you're going to be able to build a 
prototype with broad enough reach to derive useful conclusions, and if 
you do manage that and it starts to become more broadly used, you'll 
become a target with entirely new massive problems to solve.  I know 
there are things that can help, but I think you'd get swamped so badly 
that even, say, 99% suppression of malicious (and just dumb) input won't 
be enough to yield a useful signal.

Also like der Mouse, I think it might a useful instructive exercise. But 
be prepared for the exercise proving itself to you that it won't work 
and why.  And please publish the result, at least here.

---

From: Rich Kulawiec <[email-masked]>
Date: Fri, 22 Oct 2010 12:53:03 -0400

Others have largely covered what I'd say already, so I'll add this:
you may find it useful to read aboute Credence:

	http://www.cs.cornell.edu/People/egs/credence/

and to consider how a system like that deployed in an IP reputation
project might (or might not) be able to resist accidental and deliberate
manipulation.

---rsk

---

From: Dave CROCKER <[email-masked]>
Date: Fri, 22 Oct 2010 10:02:16 -0700

On 10/21/2010 7:53 AM, Chris Lewis wrote:
> I think it's worth pointing out that _everybody_ does to one extent or another.

I used to think that, too.  But lots of other folk told me it was not true, so I 
stopped believing it...

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: [email-masked]
Date: Fri, 22 Oct 2010 15:19:47 -0400

> 	http://www.cs.cornell.edu/People/egs/credence/

That's interesting, but hard to apply.  I can't really give everyone a
custom list of reputation data only using data from other people with whom
theirs highly correlates.  Well, maybe I could try.  Probably insane
though.

Yes, finding groups with highly correlating reports will be important, but
I don't expect it to be obvious which groups are the spammers and which
groups are legit.  Or which groups are just incompetent.

Something I'm surprised didn't come up on this list, that was mentioned on
the spamassassin users list:
http://www.mimedefang.org/reputation
An IETF draft of a Reputation Reporting Protocol.

I'm not a huge fan.  It uses UDP which, as it mentions, allows forging of
the sending IP.  Other good stuff in it though.

The spamassassin users list also mentioned Google's paper from 2006 on
their internal reputation system:  http://www.ceas.cc/2006/19.pdf
I'm curious how much the usefulness of authenticated domain based
reputations has changed in the years since.

-- 
"I'd rather be happy than right any day."
- Slartiblartfast, The Hitchhiker's Guide to the Galaxy
http://www.ChaosReigns.com

---

From: Rich Kulawiec <[email-masked]>
Date: Fri, 22 Oct 2010 18:14:04 -0400

On Fri, Oct 22, 2010 at 03:19:47PM -0400, [email-masked] wrote:
(about Credence)
> theirs highly correlates.  Well, maybe I could try.  Probably insane

Agreed, it *is* hard to apply.  Maybe even insane.  One of the daunting
parts of a task like this is the scope of the problem: the universes of
(IPv4 addresses), (IPv6 addresses), (observations), and (reports)
are all pretty big; and (reporters) might need to be fairly large
in order to be effective.  That's a lot of data of questionable
provenance to juggle, and while some of what's been learned via
Credence may help, I don't think it's a full solution.  I just thought
it might be useful to consider their approach and see if any parts
of it lend themselves to this.

---Rsk

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 22 Oct 2010 18:29:40 -0400

> I used to think that, too.  But lots of other folk told me it was not true, so I

That everybody's spam filters don't have techniques that are subject to 
attack and subversion if the stakes are high enough?

I don't believe that for a second.  I don't think you should either.

That will remain as long as email has the current paradigms.

This is why diversity in filter solutions continues to be so important. 
  I don't want there to be a single methodology that everybody uses, 
because once that's broken, and the stakes are high enough that it will 
be (Hotmail being a case in point), everybody's screwed.

---

From: "Daryl C. W. O'Shea" <[email-masked]>
Date: Fri, 22 Oct 2010 21:57:03 -0400

> An IETF draft of a Reputation Reporting Protocol.

I actually wrote a plugin for SA to query one such service, April 
Lorenzen's outboundindex.net.  I'm not sure how she ever made out with 
it, or if she's still at it.  Yours, hers, and others, ideas are quite 
similar.  I myself only ever got around to a small scale test of my own 
and registering a handful of domain names.

http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/dos/SIQ.pm

> the sending IP.  Other good stuff in it though.

You could avoid bulk amounts of forgery by passing sequence numbers back 
and forth or what not.  I'm thinking we might have been passing a secret 
token back and forth at one point.  UDP for speed can't easily be beat, 
though.  When you're dealing with millions of messages, and thus 
queries, from each clients' mail server each day you need to handle 
massive scaling.

Daryl