Subject: Re: [Asrg] "Uncaught spam" research project

From: Martijn Grooten <[email-masked]>
Date: Tue, 4 May 2010 13:06:50 +0100

Aaron Wolfe wrote:
> In my experience, you will find more variation in performance between
> a properly configured / maintained spam filter and a system left at
> defaults and forgotten about than you will find between different
> vendors.  Filters will use a variety of tactics to detect spam but
> most are common and unless a vendor has implemented them incorrectly
> they should perform identically.  For instance RBLs, smtp syntax
> checks, dns checks, IP connection characteristics are all going to be
> common. There is variation in how the results of these things are
> used, but this is often configurable and needs to be tweaked for a
> particular type of site for best performance anyway.
>
> Are you planning to compare these systems in their default
> configurations?  If so your results may be more an indicator of which
> vendor's defaults work best for your system than anything else.

The setup is part of a comparative anti-spam test I'm running and products have been set up by their developers to run ideally in that context. I've been running these tests for some time and before I started running them I was a bit worried that all products would catch the same spam. However, while products easily catch 98% of (spam trap) spam, it's not the always same spam that is caught: 10-15% of spam is missed by at least one filter; barely no message isn't blocked by at least one filter. This is one of my motivations for wanting to do this project.

Martijn.

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

---

From: Martijn Grooten <[email-masked]>
Date: Tue, 4 May 2010 13:15:06 +0100

Bill Cole wrote:
> That skews your sample quite a bit. A significant fraction of the hard
> cases
> these days are CAN-SPAM compliant campaigns sent by well-meaning
> originators
> using mostly-whitehat ESP's. The vast majority of spam is nothing like
> that,
> but the vast majority is also fairly easy to shun for all but the
> largest
> and cheapest providers. The spam that ends up in a real user's normal
> Inbox
> (where it is most annoying to them) is unlikely to ever hit a trap
> address.
> On the other side, traps vary in provenance so there is a lot of
> variation
> in what they get but as a direct corollary of the fact that you can
> call an
> address a trap, the mail it is sent will be coming from less careful
> and
> clueful spammers. That tilt won't be the same for every sort of trap,
> but it
> will be there for all traps to some degree.
>
> That does not make your research pointless, but it is important to
> understand the skew and the subjective "cost" of different sorts of
> uncaught
> spam in applying whatever you find.

Oh, I absolutely agree. There are many reasons why I want to concentrate on spam-trap spam (perhaps the most important one being that for these messages it's so much easier to decide whether they are actually both spam and unwanted -- they are both more or less by definition), but I am well aware that it only covers part of the spam. And a part that's already easy to filter (but, arguably, also a part where not filtering, especially in the case of phishing, can be more dangerous). I might run the same project on a different spam corpus at some point in the future.

> Again, that is an issue that speaks primarily to interpretation and
> application of your results, rather than to the whole plan. If you
> don't
> have a skilled SA wrangler handy you cannot test the results of a
> customized
> and tuned SA deployment, and that is one of the strong arguments
> against
> using straight SA in the real world.

Good point. For this very reason I'm not testing SA as part of the comparative test; that just wouldn't be fair on it.

> > [3] I would love to include DKIM, but I can only distinguish between
> does
> > have and does not have a DKIM-signature; the redacting of emails to
> hide
> > the original recipient makes me unable to decide whether a present
> > signature was actually valid.
>
> Probably not a big loss in itself, as DKIM correlation with spamminess
> is
> very sensitive to the sort of mailstream one has, and in complex ways.

Actually, (almost) no spam trap spam I receive has a DKIM signature, valid or not.

> HOWEVER, this raises a very serious design pitfall. You need to make
> sure if
> you plan to feed redacted messages to filters that they can be made to
> ignore whatever redaction-spoor is in those messages. The simplest
> example
> is the one you give: broken signatures. A valid signature may correlate
> very
> poorly to validity of the mail while a bad signature may correlate
> quite
> well to invalidity. As a general rule, redaction for the purpose of
> hiding
> individual identity naturally tends to make all of the redacted
> messages a
> little bit more like each other in ways that may be obvious or may be
> subtle, and many filters are designed to look for patterns of
> similarity as
> evidence of spam.
>
> Ultimately I think it is so hard to be sure that you are avoiding
> significant effects from redaction that researching filters with
> redacted
> inputs is a total waste of time. I don't really understand the point of
> redaction for this anyway, since the addresses are traps. The case for
> recipient redaction may be plausible when spam is being reported in
> public
> or to untrusted parties, i.e. to protect individual privacy or the
> effects
> of disclosing a trap address. For filter testing with trap spam, the
> only
> risk of disclosure would be if a filter works in some collaborative
> fashion
> akin to DCC or Razor/Pyzor. If you are afraid of that resulting in
> harmful
> disclosure of your trap addresses, then you have an intractable
> problem. The
> GIGO principle applies, and any modification of your inputs from what a
> filter would see in the real world makes your inputs garbage.

I agree with you in theory. And I know one should be wary assuming things about spammers' behaviour.

However, all that happens when redacting the messages is something along the lines of

s/spamtrapdomain/mydomain/g

and

s/spamtrap-local-part/my-local-part/g

Apart from signatures -- which, in practise, turn out to be hardly there -- I don't see how this affects the setup.

Martijn.