Subject: Spam in Mailing List Archives

From: Eric Rescorla <[email-masked]>
Date: Fri, 23 Jan 2009 14:42:28 -0800

Hi,

I've lately started noticing that a number of the mailing list archives
maintained at IETF archives are full of spam--this never appeared on
the list, it's been forged directly into the archives.

Can something be done about this please?

Thanks,
-Ekr

---

From: Barry Leiba <[email-masked]>
Date: Fri, 23 Jan 2009 17:52:27 -0500

ekr said...
> the list, it's been forged directly into the archives.

Indeed.  I've looked at some of the messages, and it seems to me that a quick 
check of the Return-Path and List-ID headers should filter these.  That is, the 
mailing list archiver should not accept any messages that do not have the correct 
(matching) values in these fields for the mailing list that it's archiving.

Barry

--
Barry Leiba, DKIM Working Group chair  ([email-masked])
http://www.research.ibm.com/people/l/leiba

---

From: Ralph Droms <[email-masked]>
Date: Fri, 23 Jan 2009 18:07:12 -0500

I submitted a report about this problem a few weeks ago and heard no  
response beyond the automated "We've received your request".

- Ralph

---

From: James M Galvin <[email-masked]>
Date: Fri, 23 Jan 2009 18:25:53 -0500

Eric,

This is a known problem and is actively being fixed.  There will 
shortly be a fix that will stop the spam from entering the archives, 
after which AMS will begin the process of removing the spam that is in 
the archives.

This is an artifact of the email architecture that AMS inherited.

Although the fix to stop it coming will be forthcoming within the next 
week, getting rid of the spam already in the archives will take a bit 
more calendar time to complete, probably measured in at least weeks if 
not months.

Jim

-- On January 23, 2009 2:42:28 PM -0800 Eric Rescorla 
<[email-masked]> wrote regarding Spam in Mailing List 
Archives --

> Hi,
>
> I've lately started noticing that a number of the mailing list
> archives maintained at IETF archives are full of spam--this never
> appeared on the list, it's been forged directly into the archives.
>

---

From: James M Galvin <[email-masked]>
Date: Fri, 23 Jan 2009 18:37:45 -0500

-- On January 23, 2009 5:52:27 PM -0500 Barry Leiba 
<[email-masked]> wrote regarding Re: Spam in Mailing List 
Archives --

> Indeed.  I've looked at some of the messages, and it seems to me that
> a quick check of the Return-Path and List-ID headers should filter
> these.  That is, the mailing list archiver should not accept any
> messages that do not have the correct (matching) values in these
> fields for the mailing list that it's archiving.

You're right, this could work.  However, the fix of choice is to "fix 
the architecture."

What is happening is that the archives are fed by "special" email 
addresses.  Each mailing list has two special subscribers, one for the 
text archive and one for the web archive.

There are two reasons for this, one of which is that it is a carryover 
from when the IETF was supposed to get a copy of all email messages for 
lists it did not host.  The other reason is because the IETF uses 
MHonArc for its web archive, i.e., we do not use the built-in Mailman 
archive.  There's more to that decision but I won't go in to that here 
and now.

Thus, historically, these aliases "needed" to exist and, historically, 
access control of them was minimal because nothing was needed.  As you 
well know this is no longer the case.

Anyway, long story short, the solution is to eliminate these "extra" 
aliases.  Personally, I never would have solved the archive issue in 
this way the first time around but, then again, in all fairness, I'm 
sure it was an easy solution that made perfect sense 10 years.

So, we're going to make it "better, stronger, faster than before."

Jim

---

From: "Hannes Tschofenig" <[email-masked]>
Date: Sat, 24 Jan 2009 17:37:45 +0200

I noticed this as well. 
See, for example, the GEOPRIV mailing list archive:
http://www.ietf.org/mail-archive/web/geopriv/current/maillist.html

>-----Original Message-----
>From: [email-masked] 
>[mailto:[email-masked]] On Behalf Of Eric Rescorla
>Sent: 24 January, 2009 00:42
>To: [email-masked]
>Cc: [email-masked]
>Subject: Spam in Mailing List Archives
>
>Hi,
>
>archives maintained at IETF archives are full of spam--this 
>never appeared on the list, it's been forged directly into the 
>archives.
>

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Sat, 24 Jan 2009 21:15:30 -0500

--On Friday, January 23, 2009 06:37:45 PM -0500 James M Galvin 
<[email-masked]> wrote:

> lists it did not host.

Uh, as far as I know, that requirement has not gone away.  What plans 
exist, if any, for changes to the way archives are handled for lists that 
are not hosted by the IETF?

-- Jeff

---

From: James M Galvin <[email-masked]>
Date: Sun, 25 Jan 2009 10:13:18 -0500

-- On January 24, 2009 9:15:30 PM -0500 Jeffrey Hutzelman 
<[email-masked]> wrote regarding Re: Spam in Mailing List Archives --

> > There are two reasons for this, one of which is that it is a
> > carryover from when the IETF was supposed to get a copy of all
> > email messages for lists it did not host.
>
> exist, if any, for changes to the way archives are handled for lists
> that are not hosted by the IETF?

Sorry, I didn't mean to suggest that this would change.  Here is the 
complete picture.

The current plan is to change the architecture to eliminate the 
"archive" addresses for all lists hosted at the IETF.  The aliases will 
remain for the external lists initially.

There is a plan being considered to transition the aliases used by 
external lists to something else.  Since I'm the one who proposed the 
plan I'll describe it here, as long as you keep in mind it is subject 
to change as the details are reviewed.

Going forward, I'm proposing to create lists at the IETF for all the 
lists hosted elsewhere (without subscribers).  Then we coordinate with 
the external lists to change the subscribed address from the "archive" 
address to the name of the list created at the IETF.

The benefit here is operational.  The IETF uses one system -- one set 
of controls, one archive model, etc. -- to manage everything.  We 
already have in place solutions for dealing with spam, authorization, 
and archives on the IETF lists.

As a side benefit, if any list wants to move to being hosted at the 
IETF, everything is ready to go; you just move the list of subscribers 
and you're done.

On the policy side, there may be an "issue" with RFC2418, which says 
that working groups with lists hosted elsewhere are required to 
maintain the "archive" alias on their list of subscribers.  Of course, 
this is already a "little bit wrong" because lists actually have two 
archive aliases, one for the text archive and one for the web archive, 
but this really is a minor operational distinction.  I think the 
important point here is that operationally the IETF needs a copy of all 
messages.

Comments welcome,

Jim

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 25 Jan 2009 07:54:46 -0800

At Fri, 23 Jan 2009 18:37:45 -0500,
> So, we're going to make it "better, stronger, faster than before."

I don't mean to be difficult, but this seems like making the best
the enemy of the good. I don't doubt that it would be nice to find
a better archiving system than that currently used by the IETF,
but the immediate problem (and one that has been happening for
at least a month now) is spam in the archives. 

The fix that Barry proposed (or any of about 5 others) could have been
executed in hours to days, and didn't carry the (substantial) risk
that something would go wrong in the transition. Unless I'm missing
something, this plan has resulted in substantial delay, thus allowing
the spam in the archives to continue to accumulate. What additional
benefits are we obtaining as a result of the choice to rearchitect the
system rather than just fix the immediate problem? Why wouldn't it
have been better to apply a simple fix and then rearchitect at
leisure?

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 25 Jan 2009 08:00:55 -0800

At Fri, 23 Jan 2009 18:25:53 -0500,
> more calendar time to complete, probably measured in at least weeks if 

Can you explain why this will take so long? This is MHonArc, right?
It seems like a relatively simply matter of a perl script which
executes Barry's suggested filter [*] and then invokes RMM on the
offending messages? 

-Ekr

[*] It looks to me like MHonArc may have the original headers, but even
if it doesn't you should be able to extract them from the text archives,
which would need similar scrubbing in any case.

---

From: James M Galvin <[email-masked]>
Date: Sun, 25 Jan 2009 10:54:53 -0500

-- On January 25, 2009 7:54:46 AM -0800 Eric Rescorla 
<[email-masked]> wrote regarding Re: Spam in Mailing List 
Archives --

> something, this plan has resulted in substantial delay,

I'm not sure what the start date, i.e., if it was earlier than 4 days 
ago.  I expect the addition of new spam will stop on Monday, Tuesday at 
the latest.

> thus allowing

I've been thinking about Barry's proposal and had an offline discussion 
with Paul Hoffman about it too.  Searching for the "archive" alias is 
insufficient but there may be other things that could work.  In any 
case, the problem with any filter is getting it wrong.  If you're going 
to put a filter in place you have to include a "backup" mechanism that 
you can review to make sure it worked correctly.

Also, I agreed with AMS that making any kind of change to email on 
Thursday or Friday (i.e., before the weekend) was probably not a good 
idea given how important email is in general.  It's better to collect a 
few more spam messages that risk "breaking" something.

The change under consideration simply redirects the flow of email.  It 
is a different kind of change than a filter change, which is 
subjective.  We know the email works in the flow direction we have so 
the plan is to stick to that.

In other words, yes it is a few days and yes there is some additional 
spam in the archives, but it is a simpler, less risky solution.  At 
least that's what I think.

Jim

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 25 Jan 2009 08:28:49 -0800

At Sun, 25 Jan 2009 10:54:53 -0500,
James M Galvin wrote:
> ago.

Well, it's not difficult to discover. It goes back at least a 
month:

http://www.ietf.org/mail-archive/web/ietf/current/msg54696.html

Moroever, as Ralph Droms indicates, he reported it weeks ago:

http://www.ietf.org/mail-archive/web/wgchairs/current/msg06908.html

Are you saying that AMS just started working on this four days ago?

> insufficient but there may be other things that could work.

I'll let Barry argue this point if he likes, but I'm not sure what
"insufficient" means. I would observe that a filter that removed 90%
of the problem would be 90% good.

>  In any 

Of course.

> the plan is to stick to that.

And you think that this doesn't need a backup in case something
goes wrong? That seems optimistic.

-Ekr

---

From: Barry Leiba <[email-masked]>
Date: Sun, 25 Jan 2009 12:02:30 -0500

> of the problem would be 90% good.

I won't argue much about it -- I think we've made the point: we really want AMS 
to take care of the spam yesterday (both new and old), and then any clean-up of 
the archive architecture can be done.

The only thing I'll add is that if an email address is only supposed to receive 
mail from one source, it's really trivial -- and safe from errors -- to filter it 
by casually tossing any mail that doesn't (appear to) come from that source.  For 
this particular case, there's really no risk of false positives.  And the only 
false negatives will come from attacks that are so specifically targeted that 
they have the right SMTP MAIL FROM *and* the right LIST-ID header.  In that case, 
well, waddyagonnado?  There are other criteria, but it hardly seems worth 
worrying about them to a first approximation.

Barry

---

From: James M Galvin <[email-masked]>
Date: Mon, 26 Jan 2009 10:48:45 -0500

-- On January 25, 2009 8:28:49 AM -0800 Eric Rescorla 
<[email-masked]> wrote regarding Re: Spam in Mailing List 
Archives --

> > > Unless I'm missing something, this plan has resulted
> > > in substantial delay,
> >
> > I'm not sure what the start date, i.e., if it was earlier than 4
> > days  ago.
>

I can not answer this.  I can only say that I was contacted 4 days ago 
(now 6 it seems) and talked at length with one of the IT guys at AMS 
about what was happening, why it was happening, and the various things 
that could be done to "fix" it.

Truth be told, even if it has been a month, that doesn't bother me 
personally.  Spam will always be with us, so taking extra time to deal 
with a "spike" now and then is no big deal.

> of the problem would be 90% good.

The factor that adds complexity is that these aliases are also used by 
the live system, i.e., for those lists hosted at the IETF.  Also, all 
of the aliases use the same script for archiving.

So, if you change the script to add filtering, as you suggest, then all 
lists are affected.  This spam problem is worse on some lists than 
others.  My point is we do not want to risk false positives across all 
lists.

Of course they could add a new script, but you still have to worry 
about false positives.  I'm worried about how many of the 90% are false 
positives.  Very worried about it, in fact.  So, I'll say again, I'd 
rather a few more spam messages in the archive while we setup the best 
solution we can rather than risking losing some email.

> > The change under consideration simply redirects the flow of email.
> > It  is a different kind of change than a filter change, which is
> > subjective.  We know the email works in the flow direction we have
> > so  the plan is to stick to that.
>

In a word, no.  Let me explain.

The difference is we do not need a new backup.  The existing system 
already does backups, as it should.  If we add spam filtering as has 
been suggested, we need a system in place for evaluating the rejected 
messages.  That's the backup we need to add for what's been proposed 
here, and the backup we're not doing with what we're going to do 
instead.

Mailing lists hosted at the IETF already have a policy and an 
implementation of that policy to handle the spam.  Rather than add a 
component to the system, which is to say we should add a spam filter 
for dealing with the archive of externally hosted lists, we want to 
direct those messages through the existing mailing list system instead. 
To do this requires a different change to the system but it is not a 
change that risks losing anything that policy does not already allow.

Also, we get a few benefits from the fix chosen over and above fixing 
the problem.  Plus, in my opinion, this fix is more robust than a 
filtering fix, which I consider one of the riskiest things to do.

I know that email filtering can be "done right".  The IETF already does 
it right in the filtering that is already being done.  The proposal is 
to make use of that rather than adding a new filtering component.

I'm sure you will agree that filtering can go seriously bad in short 
order, even when you're well intentioned.  Why risk it if you don't 
have to?  Email is too important to the IETF.  That's all I'm saying.

Jim

---

From: Robert Sparks <[email-masked]>
Date: Thu, 29 Jan 2009 13:54:55 -0600

Is there a place where progress on this is being reported that folks  
can point me to?

(fwiw, spam is still accruing. I'm particularly interested on the  
estimate for when we'll have the lists scrubbed).

RjS

On Jan 25, 2009, at 9:54 AM, James M Galvin wrote:

>
>
> -- On January 25, 2009 7:54:46 AM -0800 Eric Rescorla <[email-masked] 
> > wrote regarding Re: Spam in Mailing List Archives --
>
> days ago.  I expect the addition of new spam will stop on Monday,  
> Tuesday at the latest.
>
>
>> thus allowing
> I've been thinking about Barry's proposal and had an offline  
> discussion with Paul Hoffman about it too.  Searching for the  
> "archive" alias is insufficient but there may be other things that  
> could work.  In any case, the problem with any filter is getting it  
> wrong.  If you're going to put a filter in place you have to include  
> a "backup" mechanism that you can review to make sure it worked  
> correctly.
>
> Thursday or Friday (i.e., before the weekend) was probably not a  
> good idea given how important email is in general.  It's better to  
> collect a few more spam messages that risk "breaking" something.
>
> In other words, yes it is a few days and yes there is some  
> additional spam in the archives, but it is a simpler, less risky  
> solution.  At least that's what I think.
>
> Jim
>

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 30 Jan 2009 09:58:42 -0800

At Thu, 29 Jan 2009 13:54:55 -0600,
> estimate for when we'll have the lists scrubbed).

+1 

-Ekr