Subject: Re: IETF Mailing List Issue

From: Alexa Morris <[email-masked]>
Date: Fri, 05 Sep 2008 11:46:50 -0700

Paul,

I think that Henrik will be able to respond to your questions much more
competently than I can, but from what I understand, this is what happened:

The program that Henrik developed for the IETF -- postconfirm, which
replaced the very ill-suited TMDA -- unfortunately failed to start properly
after a reboot.  And it sounds like there is currently a problem in
postconfirm that was causing it to throw away mail on a failure, rather than
holding it and retrying it.  So, in actuality, IETF mail transport was up,
and passing email to Postconfirm, but because postconfirm was jammed it
didn't hold mail, it threw it out instead.  The server was up and running;
mailman itself was up and running just fine -- it just didn't receive any
messages from postconfirm to send out to the lists.

Alexa

On 9/5/08 10:46 AM, "Paul Hoffman" <[email-masked]> wrote:

> At 9:02 AM -0700 9/5/08, Alexa Morris wrote:
>> Last night, after a server reboot, the postconfirm spam checker that was
>> recently put in front of IETF mailing lists failed to start.  Unfortunately,
>> this resulted in a huge influx of spam to the IETF ticket system, and a
>> simultaneous failure of IETF list email. Everything is back online now and
>> functioning; however, email messages sent to IETF lists in the last 16 hours
>> were lost, and will need to be resent.
>> 
>> We have now installed a direct watcher on postconfirm, to force a restart of
>> it when it fails. In addition, Henrik is working on the system now to find
>> out why it failed to start in the first place, and why its failure caused
>> mail loss rather than mail deferral.
>> 
>> I apologize for any inconvenience this may have caused you. As always,
>> please feel free to contact me if you have any questions or concerns.
> 
> With all due respect: why did the flood of spam to the IETF ticket
> system cause mailing list traffic to be lost? Those seem to be
> unrelated.
> 
>  From a WG chair perspective: what other kinds of server problems can
> cause mailing list traffic to be lost (as compared to being delayed)?
> 
> --Paul Hoffman, Director
> --VPN Consortium
>

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 05 Sep 2008 21:06:59 +0200

Paul, all,

Following up on Alexa's mail with additional details:

The loss of mail wasn't caused by the RT queues overflowing with spam,
but there was a common root cause between that and the loss of mail to
the lists.

The integration between postfix, postconfirm and mailman is handled by a glue
script, a wrapper.  Postfix calls the wrapper, which runs postconfirm and then
forwards to mailman if the mail is legit. When, after the reboot, postconfirm
failed to start, the wrapper script should have caught that and passed mail
through to mailman unfiltered.  Unfortunately, the version of the wrapper
script was old -- newer versions catch errors correctly -- and so the
failure of postconfirm to start caused dropped mail instead of pass-through.

The integration between postfix, postconfirm and RT was done through
procmailrc, and did the pass through correctly.  This method would have
been preferable for mailman too, except that mailman doesn't lend itself
easily to integration through that method.

All in all, this event was quite unfortunate, and some of the blame for
the wrong wrapper script being in place is mine.  There is, however, no
external couplings where failure of unrelated systems will case mail loss.
What happened here is completely explained by failure in the postconfirm
integration.

And, to put this into perspective -- when I checked the relation between
spam and legitimate messages a couple of days ago, the spam percentage of
the inflow to the mailing lists was above 99%...  So even if I'm really
sorry this happened, I'm not sorry to have spent time on building this ,-/

	Henrik