Subject: busrt of spam to WG lists?

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 12 Feb 2008 10:31:13 -0800

Hi -

Starting at 10:24 this morning, all sorts of spam has been showing
up for moderator approval...  Are others experiencing this?

Randy

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 12 Feb 2008 11:50:27 -0800

> up for moderator approval...  Are others experiencing this?

I have been getting some spam to my list since Sunday at 9am Pacific Time, and
it has continued since (roughly 1-2 spams/hour Monday and Tuesday). 

You might try adding the spam filtering that James described previously (see
attached).  Myself, I have a separate system run
http://www.waider.ie/hacks/workshop/perl/mailman.pl for me every hour, which
provides me a log of messages it deletes.

-d

Subject: SPAM filtering on IETF lists
From: "James M Galvin" <[email-masked]>
To: <[email-masked]>
Date: Mon, 4 Feb 2008 11:52:58 -0800

The announcement describing the email changes said that we "updated 
the SPAM filtering".  A better choice of description would have 
been that we are experimenting with different SPAM filtering.

What I want to do is explain what we did and why we did it.  You 
should know that none of this is "cast in stone."  I'm sorry we 
could not do this before the cut-over but now we would like to ask 
for your help.  We would like your opinion on how things are going 
and to discuss what is the best way to handle SPAM filtering on 
mailing lists hosted at "ietf.org".

This is a long message and I'm sorry for that.  I figured it was 
best to just get it all out there at once.

Please understand that there was a significant time pressure 
associated with this transition.  All things considered it has 
worked out pretty well but we had to make some choices to ensure 
the success we did have.  One of those choices is that we did not 
include TMDA in front of all the mailing lists.  There were two 
principal reasons for this.

First, SpamAssassin had never been setup properly at NSS.  Thus, 
its contribution to SPAM abatement was minimal, if at all.  We did 
not have any good statistics on just how much SPAM there was nor 
how much SPAM we could control with SpamAssassin.  SpamAssassin is 
straightforward to setup and easy to use in front of all email 
delivery and we wanted to continue to use it as part of the SPAM 
abatement process.

Second, TMDA, from an implementation point of view, is impractical 
for mailing lists in the best case.  To set it up each mailing list 
has to become a user on the server.  Email for the list is then 
delivered to that user.  As part of the delivery process the email 
gets run through TMDA and, if authorized, then gets forwarded to 
Mailman.  Mailman then goes through its authorization process, 
which in fact duplicates some of the work of TMDA.

Further, each TMDA user gets a queue database of challenge-response 
messages that needs to be managed.  This was not being handled 
properly.

Finally, the master white list that was used for TMDA had grown to 
over 39,000 email addresses.  Clearly that needed to be cleaned up 
because it was contributing to delivery load.

Putting all that together we decided to do two things.

1. Setup SpamAssassin correctly and tag all messages.  We could 
then take action on messages with scores that were too high.

2. For public mailing lists, which is all IETF working group 
mailing lists hosted at ietf.org, we would drop all messages with a 
SPAM score of 5.0 or greater.  Recall that the announcement pointed 
out that dropping messages was acceptable when there was a public 
archive of distributed messages.  These messages are dropped by 
adjusting a mailman configuration element that each of you as a 
mailing list administrator controls.

A consequence of this is that obviously some SPAM messages are 
getting through to the moderation queue.  One thing that would be 
useful to know is the volume of messages that you are getting.  We 
fully understand that SpamAssassin is not perfect and will not get 
all messages.

Also, you have control over the score for messages that are 
dropped.  Currently, each list is setup to drop messages with a 
score of 5 or greater.  One thing you might consider doing is 
dropping messages with a score of 4 or 3 or lower.  I suspect it 
would not be a good idea to go lower than that in general, but then 
again your list may exhibit different behavior.

When you go to your moderation queue you should check the score of 
the moderator messages.  To do this you need to actually look at 
the message and then scroll through the headers shown.  The 
X-Spam-* headers should be readily visible.  Look for the 
"X-Spam-Level" header which has only a set of "*" characters for 
its value.  That is the one that is used in the mailman 
configuration to decide which messages to drop.

To control the dropping, you should look at the "privacy options" 
for your mailing list.  Inside that look at the "spam filters".  On 
that page notice "Spam Filter Rule 1".  There you will see a 
regular expression that looks like this:

    X\-Spam\-Level: \*{5,}

This means that if that header has at least 5 "*"'s in it, drop the 
message.  You can lower or raise the drop score by changing the "5" 
to whatever you think would work best for your list.

That completes the explanation.  The truth is we would rather not 
implement TMDA for all mailing lists.  It really is impractical. 
While it is doable and not unreasonable for a few mailing lists, it 
really is not a suitable technology for putting in front of almost 
200 mailing lists.

Are there any other options that can be put to use?  Any 
suggestions would be very helpful.

How bad is it to not have TMDA?  This is also an important 
consideration.

Thanks,

Jim