Subject: SPAM filtering on IETF lists

From: James M Galvin <[email-masked]>
Date: Mon, 04 Feb 2008 14:52:58 -0500

The announcement describing the email changes said that we "updated 
the SPAM filtering".  A better choice of description would have 
been that we are experimenting with different SPAM filtering.

What I want to do is explain what we did and why we did it.  You 
should know that none of this is "cast in stone."  I'm sorry we 
could not do this before the cut-over but now we would like to ask 
for your help.  We would like your opinion on how things are going 
and to discuss what is the best way to handle SPAM filtering on 
mailing lists hosted at "ietf.org".

This is a long message and I'm sorry for that.  I figured it was 
best to just get it all out there at once.

Please understand that there was a significant time pressure 
associated with this transition.  All things considered it has 
worked out pretty well but we had to make some choices to ensure 
the success we did have.  One of those choices is that we did not 
include TMDA in front of all the mailing lists.  There were two 
principal reasons for this.

First, SpamAssassin had never been setup properly at NSS.  Thus, 
its contribution to SPAM abatement was minimal, if at all.  We did 
not have any good statistics on just how much SPAM there was nor 
how much SPAM we could control with SpamAssassin.  SpamAssassin is 
straightforward to setup and easy to use in front of all email 
delivery and we wanted to continue to use it as part of the SPAM 
abatement process.

Second, TMDA, from an implementation point of view, is impractical 
for mailing lists in the best case.  To set it up each mailing list 
has to become a user on the server.  Email for the list is then 
delivered to that user.  As part of the delivery process the email 
gets run through TMDA and, if authorized, then gets forwarded to 
Mailman.  Mailman then goes through its authorization process, 
which in fact duplicates some of the work of TMDA.

Further, each TMDA user gets a queue database of challenge-response 
messages that needs to be managed.  This was not being handled 
properly.

Finally, the master white list that was used for TMDA had grown to 
over 39,000 email addresses.  Clearly that needed to be cleaned up 
because it was contributing to delivery load.

Putting all that together we decided to do two things.

1. Setup SpamAssassin correctly and tag all messages.  We could 
then take action on messages with scores that were too high.

2. For public mailing lists, which is all IETF working group 
mailing lists hosted at ietf.org, we would drop all messages with a 
SPAM score of 5.0 or greater.  Recall that the announcement pointed 
out that dropping messages was acceptable when there was a public 
archive of distributed messages.  These messages are dropped by 
adjusting a mailman configuration element that each of you as a 
mailing list administrator controls.

A consequence of this is that obviously some SPAM messages are 
getting through to the moderation queue.  One thing that would be 
useful to know is the volume of messages that you are getting.  We 
fully understand that SpamAssassin is not perfect and will not get 
all messages.

Also, you have control over the score for messages that are 
dropped.  Currently, each list is setup to drop messages with a 
score of 5 or greater.  One thing you might consider doing is 
dropping messages with a score of 4 or 3 or lower.  I suspect it 
would not be a good idea to go lower than that in general, but then 
again your list may exhibit different behavior.

When you go to your moderation queue you should check the score of 
the moderator messages.  To do this you need to actually look at 
the message and then scroll through the headers shown.  The 
X-Spam-* headers should be readily visible.  Look for the 
"X-Spam-Level" header which has only a set of "*" characters for 
its value.  That is the one that is used in the mailman 
configuration to decide which messages to drop.

To control the dropping, you should look at the "privacy options" 
for your mailing list.  Inside that look at the "spam filters".  On 
that page notice "Spam Filter Rule 1".  There you will see a 
regular expression that looks like this:

    X\-Spam\-Level: \*{5,}

This means that if that header has at least 5 "*"'s in it, drop the 
message.  You can lower or raise the drop score by changing the "5" 
to whatever you think would work best for your list.

That completes the explanation.  The truth is we would rather not 
implement TMDA for all mailing lists.  It really is impractical. 
While it is doable and not unreasonable for a few mailing lists, it 
really is not a suitable technology for putting in front of almost 
200 mailing lists.

Are there any other options that can be put to use?  Any 
suggestions would be very helpful.

How bad is it to not have TMDA?  This is also an important 
consideration.

Thanks,

Jim

---

From: "Mary Barnes" <[email-masked]>
Date: Mon, 4 Feb 2008 14:03:18 -0600

There is another unfortunate side effect of no longer having TMDA on the
IETF mailing lists. The amount of SPAM being sent to some lists has been
great enough that our (Nortel) corporate filters started bouncing all
messages from some of the WG mailings lists and we have been
unsubscribed.  This was a periodic problem in the past that we have not
had since TMDA was put on the IETF mailing lists. 

I'm not an expert in this area and so I can't suggest an alternative - I
just know that TMDA worked quite well both in ensuring that our
corporate SPAM filter wasn't rejecting  legitimate traffic due to the
SPAM and in my job of managing multiple IETF mailings lists.  Has the
problem with gmail been fixed yet?  In the past, that had been my backup
when the Nortel filters became over-zealous.

Note, that I did send an email to ietf-action with the appropriate
contact at Nortel to help resolve the Nortel specific issue, but I would
imagine we're not the only company that has this issue.

Regards,
Mary Barnes
[email-masked]

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of James M Galvin
Sent: Monday, February 04, 2008 1:53 PM
To: [email-masked]
Subject: SPAM filtering on IETF lists

The announcement describing the email changes said that we "updated the
SPAM filtering".  A better choice of description would have been that we
are experimenting with different SPAM filtering.

What I want to do is explain what we did and why we did it.  You should
know that none of this is "cast in stone."  I'm sorry we could not do
this before the cut-over but now we would like to ask for your help.  We
would like your opinion on how things are going and to discuss what is
the best way to handle SPAM filtering on mailing lists hosted at
"ietf.org".

This is a long message and I'm sorry for that.  I figured it was best to
just get it all out there at once.

    X\-Spam\-Level: \*{5,}

Thanks,

Jim

---

From: James M Galvin <[email-masked]>
Date: Mon, 04 Feb 2008 15:19:02 -0500

This should be an entirely different problem.

There should never be SPAM that actually gets distributed to a 
mailing list.  If that is a problem it is a different problem than 
SPAM in the moderation queues.  This discussion is about SPAM in 
the moderation queues, which affects only administrators/moderators.

Please do let me know what list is getting SPAM distributed so we 
can investigate what is happening.

With respect to the issue with GMail, that has been fixed. 
However, it brings up the point that other sites may have similar 
issues.  Given the change in IP address of servers delivering IETF 
email, it is possible that other sites need to update their local 
whitelists.  I do believe that the IETF should have an "information 
page" with this kind of information on it, and that is on the list 
of things to do for a post transition activity.  It has never 
existed before so this would be new.

Jim

-- On Monday, February 04, 2008 2:03 PM -0600 Mary Barnes 
<[email-masked]> wrote regarding RE: SPAM filtering on IETF 
lists --

> There is another unfortunate side effect of no longer having TMDA
> on the IETF mailing lists. The amount of SPAM being sent to some
> lists has been great enough that our (Nortel) corporate filters
> started bouncing all messages from some of the WG mailings lists
> and we have been unsubscribed.  This was a periodic problem in
> the past that we have not had since TMDA was put on the IETF
> mailing lists.
>
> I'm not an expert in this area and so I can't suggest an
> alternative - I just know that TMDA worked quite well both in
> ensuring that our corporate SPAM filter wasn't rejecting
> legitimate traffic due to the SPAM and in my job of managing
> multiple IETF mailings lists.  Has the problem with gmail been
> fixed yet?  In the past, that had been my backup when the Nortel
> filters became over-zealous.
>
> contact at Nortel to help resolve the Nortel specific issue, but
> I would imagine we're not the only company that has this issue.
>
> Regards,
> Mary Barnes
> [email-masked]
>
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of James M Galvin
> The announcement describing the email changes said that we
> "updated the SPAM filtering".  A better choice of description
> would have been that we are experimenting with different SPAM
> filtering.
>
> could not do this before the cut-over but now we would like to
> ask for your help.  We would like your opinion on how things are
> going and to discuss what is the best way to handle SPAM
> filtering on mailing lists hosted at "ietf.org".
>
> its contribution to SPAM abatement was minimal, if at all.  We
> did  not have any good statistics on just how much SPAM there was
> nor  how much SPAM we could control with SpamAssassin.
> SpamAssassin is  straightforward to setup and easy to use in
> front of all email  delivery and we wanted to continue to use it
> as part of the SPAM  abatement process.
>
> Second, TMDA, from an implementation point of view, is
> impractical  for mailing lists in the best case.  To set it up
> each mailing list  has to become a user on the server.  Email for
> the list is then  delivered to that user.  As part of the
> delivery process the email  gets run through TMDA and, if
> authorized, then gets forwarded to  Mailman.  Mailman then goes
> through its authorization process,  which in fact duplicates some
> of the work of TMDA.
>
> Further, each TMDA user gets a queue database of
> challenge-response  messages that needs to be managed.  This was
> not being handled  properly.
>
> Finally, the master white list that was used for TMDA had grown
> to  over 39,000 email addresses.  Clearly that needed to be
> cleaned up  because it was contributing to delivery load.
>
>
> mailing lists hosted at ietf.org, we would drop all messages with
> a  SPAM score of 5.0 or greater.  Recall that the announcement
> pointed  out that dropping messages was acceptable when there was
> a public  archive of distributed messages.  These messages are
> dropped by  adjusting a mailman configuration element that each
> of you as a  mailing list administrator controls.
>
>
> useful to know is the volume of messages that you are getting.
> We  fully understand that SpamAssassin is not perfect and will
> not get  all messages.
>
> would not be a good idea to go lower than that in general, but
> then  again your list may exhibit different behavior.
>
> When you go to your moderation queue you should check the score
> of  the moderator messages.  To do this you need to actually look
> at  the message and then scroll through the headers shown.  The
> for your mailing list.  Inside that look at the "spam filters".
> On  that page notice "Spam Filter Rule 1".  There you will see a
> This means that if that header has at least 5 "*"'s in it, drop
> the  message.  You can lower or raise the drop score by changing
> the "5"  to whatever you think would work best for your list.
>
>
> While it is doable and not unreasonable for a few mailing lists,
> it  really is not a suitable technology for putting in front of
> almost  200 mailing lists.
>

---

From: "Mary Barnes" <[email-masked]>
Date: Mon, 4 Feb 2008 15:04:40 -0600

There has been SPAM making it to the IETF mailing lists, including the
MMUSIC WG mailing list per the archives:
http://www.ietf.org/mail-archive/web/mmusic/current/mail2.html (on Feb
1st)
Also, I can see some in the GEOPRIV and XCON archives. 
So, I imagine it just took a couple of lists to get the SPAM and with
the number of Nortel folks subscribed to the various lists to hit the
corporate threshold for SPAM from the specific IP address. 

I imagine other corporate filters have also filtered these and are
perhaps less zealous about bouncing than the Nortel filter. For example,
I don't even get a chance to moderate the SPAMs that are arriving on the
lists I manage until I get the daily report about SPAM as those are
being caught by my corporate filter.

I have sent you separately more details from our IS Security prime -
again, I'm not at all an expert here. 

And, yes, the information page would be wonderful - it would be far
easier just to point our IS security folks to that than have me serve as
the middleman here. 

Mary. 

-----Original Message-----
From: James M Galvin [mailto:[email-masked]] 
Sent: Monday, February 04, 2008 2:19 PM
To: Barnes, Mary (RICH2:AR00); [email-masked]
Subject: RE: SPAM filtering on IETF lists

There should never be SPAM that actually gets distributed to a mailing
list.  If that is a problem it is a different problem than SPAM in the
moderation queues.  This discussion is about SPAM in the moderation
queues, which affects only administrators/moderators.

Please do let me know what list is getting SPAM distributed so we can
investigate what is happening.

Jim

---

From: James Carlson <[email-masked]>
Date: Mon, 4 Feb 2008 15:57:39 -0500

> had since TMDA was put on the IETF mailing lists. 

Why would a useful filter ever _bounce_ spam?  What's the point?  The
sender is just as forged as the rest of the message, so trying to
bounce it back usually just results in generating back-scatter spam --
annoying unrelated people for no apparent reason.

I'd suggest getting that fixed first.

-- 
James Carlson, Solaris Networking              <[email-masked]>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

---

From: Joe Abley <[email-masked]>
Date: Mon, 4 Feb 2008 16:18:46 -0500

> really is not a suitable technology for putting in front of almost  

The latent sysadmin in me is confused by this. Surely once you've  
figured out how to turn on TMDA for one list, turning it on for all  
lists is minimal additional work. Perhaps no additional work, if list  
creation/removal is automated, as I presume it would be for a service  
that involves hundreds of lists.

On the IAB mailing list ([email-masked]) we had TMDA turned on because of  
the volume of mail arriving in the moderator queue that was back- 
scatter due to [email-masked] being used as the from address on spam sent  
by third parties. There were several thousand messages of this type  
arriving every day. Dealing with that through the mailman moderator  
queue interface was most certainly impractical.

Joe

---

From: "Mani, Mahalingam (Mani)" <[email-masked]>
Date: Mon, 4 Feb 2008 14:20:11 -0700

I completely concur - unless the spam filter has reason to correlate and
confirm this bounce is going back to a "trusted" domain (from which it
originated) that can handle the erring client.

-mani
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of James Carlson
Sent: Monday, February 04, 2008 12:58 PM
To: Mary Barnes
Cc: [email-masked]
Subject: RE: SPAM filtering on IETF lists

---

From: James M Galvin <[email-masked]>
Date: Mon, 04 Feb 2008 16:39:56 -0500

-- On Monday, February 04, 2008 4:18 PM -0500 Joe Abley 
<[email-masked]> wrote regarding Re: SPAM filtering on IETF 
lists --

> > That completes the explanation.  The truth is we would rather
> > not   implement TMDA for all mailing lists.  It really is
> > impractical.   While it is doable and not unreasonable for a
> > few mailing lists, it   really is not a suitable technology for
> > putting in front of almost   200 mailing lists.
>
> figured out how to turn on TMDA for one list, turning it on for
> all lists is minimal additional work.

Setting it up is only problematic insofar as you have to have users 
on the server.  True, they are just virtual users so arguably even 
this is not too significant an issue.  It's just a point worthy of 
mention particularly in a security context.

The real issue here is the operation.  Who is going to manage all 
the queues setup to manage the challenge-response messages?

And then there's the master whitelist that needs to be created and 
managed.  39,000+ entries is just silly.  Every message that comes 
in has to be compared against every email address on this list.

Yes, this begs for a database, but that's not what's there.  TMDA 
was not intended to be used in this way.  That doesn't mean it can 
not be made to work, and if that's what we really believe is 
necessary then that's what we'll do.  However, given that 
SpamAssassin is now doing a much better job than it did before, I'm 
asking if we really need it?

Of course, the IAB and IESG are special lists and they probably do 
need it for just the reasons you mention.  In addition, they are 
"private" mailing lists so handling them differently is to be 
expected regardless.

Jim

---

From: James M Galvin <[email-masked]>
Date: Mon, 04 Feb 2008 16:48:08 -0500

-- On Monday, February 04, 2008 10:42 PM +0100 Henrik Levkowetz 
<[email-masked]> wrote regarding Re: SPAM filtering on IETF 
lists --

> Hi James,
>
> Comments inline,
>
> I'm quite willing to assist with ideas and dicsussion on setting
> up
> TMDA so it works for us; I'm not willing to give it up until we've
> really worked hard at setting it up (and failed).

Here's a specific question.

How do we deal with an ever growing whitelist?

39,000 entries just seems wrong to me.  Surely at least some of 
those are no longer valid.

Should we find a way to "age" them?  Should we toss the list and 
restart every month or quarter?  Should we do something else?

And note that as far as I know, TMDA doesn't do any management of 
the whitelist automatically.  If I'm wrong about this please do let 
me know.  So, we want to manage the whitelist, we'll need to 
develop something ourselves.  This is not necessarily hard, just 
another action to complete.

Jim

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 5 Feb 2008 11:42:32 -0800

> > up TMDA so it works for us; I'm not willing to give it up 
> > until we've really worked hard at setting it up (and failed).
> 

Aging entries after a year seems reasonable.  I don't see a need
to be more aggressive than that, though.

I notice that today, after what I believe was an ietf.org email
outage (someone asked me why their post was not appearing and
discussions had apparently stopped), that we seem to not be
getting SpamAssassin tagging of posts in the moderator queue.

This means the nifty regular expression doesn't help with
identifying spam posts and reducing WG Chair's workload at
cleaning them up..

Fortunately I have a system that I can run the
http://www.waider.ie/hacks/workshop/perl/mailman.pl script on, which runs
SpamAssassin against each message for me.  

But, yes, I do miss TMDA.  And now it's lent.  Maybe we can get TMDA
re-instated by Easter?  ;-)

-d

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 5 Feb 2008 14:52:11 -0800

...
> This happened for about a day when we first did the cut-over, until 
> we realized there was a bit of configuration missing.  There should 
> not be any SPAM in the archives since 1 February.  Cleaning up 
> existing archives is another problem.

The spam "Men's health drugs for almost a quarter the price!" appears in the
AVT achives on 5-Feb:
  http://www.ietf.org/mail-archive/web/avt/current/index.html
and apparently had its own Date header of 4-Feb:
  http://www.ietf.org/mail-archive/web/avt/current/msg09214.html

-d

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 5 Feb 2008 14:53:52 -0800

> Correct.  I appreciate the advantages of challenge-response, and 
> the disadvantages.  :-)
> 
> If we decide here that challenge-response is an important component 
> for managing IETF email and mailing lists, then I would say that 
> TMDA is not the right solution.
> 
> If TMDA is all we have and people really want it back, then we'll 
> have to make do with it while we pursue another option, which may 
> mean developing our own tool.

We are willing to hear what else you might propose.

But right now we are running naked (and it is wintertime); SpamAssassin 
is not scoring messages.

-d

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 5 Feb 2008 14:56:57 -0800

> If there are a dozen entries on the list, sure.
> If there are 39000 entries, I recommend using a data 
> structure that means 
> you're still only comparing to a couple of dozen entries.  
> It's called a 
> "binary tree", and searching it requires O(log(n)) compares, 
> which is about 16 for a list that size.

Or hash it and compare once.  We don't really need 100% accurate matches, we
just need a first line of defense to prevent the moderation queues from
getting filled with 100s of messages in a day.  Some false positive whitelist
matches are okay because mailman's automatic moderation of posts by
non-subscribers will prevent the spam from being delivered to users (and
prevent the spam from being posted to the archives).

-d

---

From: Adam Roach <[email-masked]>
Date: Tue, 05 Feb 2008 17:14:04 -0600

> How bad is it to not have TMDA?  This is also an important

Very, very bad.

Prior to the TMDA implementation, the moderation queue for our list was
inundated with hundreds of messages a day. It's not something that I could
stay on top while actually performing my job duties. After a while, I just
gave up and let it pile up. I knew I shouldn't, but there wasn't anything
that could reasonably be done. I put in an automatic filter to shuffle the
moderation requests off into a separate folder -- one that I grew scared of
even opening any more. Valid messages would fall between the cracks.
Sometimes I'd get an email from someone asking me to release their message
from moderation, and I'd go wading into the 25,000 or so spams to try to
find their message so it could be released. Often, weeks late.

Then TMDA was turned on, Cullen had someone clear out our queue for us (it
simply was too large to load the web page any longer), and the problem went
away. Instantly. And completely.

When a moderation request came in, it was a legitimate message. 100% of the
time. I could take the moderation requests seriously again, and could
release messages within minutes of their arrival in the queue.

Now we're suddenly back to the torrent. And, even if I want to keep up, I
know that it's simply beyond the capabilities of a human to do so.

And legitimate messages will be lost again.

/a

---

From: <[email-masked]>
Date: Tue, 5 Feb 2008 18:16:39 -0600

Just joining late to this email party, without having any 'technical'
position
on the issue, I just want to comment that the system working before the
transition
was working perfectly: the current situation is now broken. 

I do not want to have to sort through 50-75 emails caught in the mailing
list
admin UI every day.

John 

>[mailto:[email-masked]] On Behalf Of ext Dan Wing
>Sent: 05 February, 2008 14:54
>To: 'James M Galvin'; 'Joe Abley'
>Cc: [email-masked]
>> Correct.  I appreciate the advantages of challenge-response, and the 
>> disadvantages.  :-)
>> 
>> for managing IETF email and mailing lists, then I would say 
>that TMDA 
>> is not the right solution.
>> 
>> If TMDA is all we have and people really want it back, then 
>we'll have 
>> to make do with it while we pursue another option, which may mean 
>> developing our own tool.
>
>But right now we are running naked (and it is wintertime); 
>SpamAssassin is not scoring messages.
>
>-d
>
>

---

From: Joel Jaeggli <[email-masked]>
Date: Tue, 05 Feb 2008 14:39:19 -0800

Henrik Levkowetz wrote:
> Hi Jim,
> 
> On 2008-02-04 22:48 James M Galvin said the following:
> 
>> 39,000 entries just seems wrong to me.  Surely at least some of those 
>> are no longer valid.
> 
> Agreed. 
> I'd suggest to age them, and remove after for instance 6 months.

currently mailman suspends addresses after a couple bounces and 
unsubscribes them a month or two later as far as I can tell. We just 
created a new list there after the last meeting...

It would seem wise to model the behavior of one system on another, at 
least insure that they're roughly congruent about when someone falls off 
the edge of the earth.

>> And note that as far as I know, TMDA doesn't do any management of the 
>> whitelist automatically.  If I'm wrong about this please do let me 
>> know.  So, we want to manage the whitelist, we'll need to develop 
>> something ourselves.
> 
> Yes, I think that's right.
> 
>> This is not necessarily hard, just another action to complete.
> 
> Agreed. Fixing this on a rush basis seems like wrong prioritisation,
> but that doesn't mean giving up on it.
> 
> I have some shell scripts I use to manage the aging of the white- and
> black-lists generated from one of my Spamassassin plugins; I'm enclosing 
> them in case they would be of any help.  The 'update'
> script is a general utility, it happens to be used by the 'purgelist'
> script.  You can easily eliminate the need for it by minor tweaks to
> 'purgelist', if desired.  You'll also have to tweak the purgelist script
> to only handle one list, rather than a pair (black-, white-) of lists.
> 
> Although 'purgelist' was written to handle my exim auto white-/black-
> lists in particular, as reflected in the --help output, it should be
> able to deal perfectly well with a whitelist of email addresses rather
> than IP addresses.
> 
> 
>     Henrik
> 
> 
>