Subject: Mailman password reminder emails

From: Glen <[email-masked]>
Date: Wed, 1 Aug 2018 09:08:00 -0700

Greetings -

A number of you - myself included - will have received the dreaded
Mailman password reminder email this morning.   This is obviously a
bad thing: the script, part of the Mailman distribution, sends Mailman
passwords out to users in regular clear email.  Worse, its operation
cannot be disabled via configuration option, it can only be disabled
by patching the Mailman source files:  commenting out the relevant
cron entry, and removing the script.

AMS regularly applies OS-provided software updates and security
patches to the IETF servers as a part of our ongoing maintenance
duties.  We've seen Mailman updates before install without issue;
however, the most recent update silently re-enabled the cron entry and
restored the script... and we did not catch it.  So, last night, the
flood began.

Since we all love technical details, some of you might ask, "How could
a security patch re-enable a cron entry?"  Mailman has its own crontab
file, a copy of which is kept in its operating directory.  The Mailman
start script re-copies this file into /etc/cron.d whenever Mailman
starts.  So, the patch updated the crontab copy, containing the
offending (or offensive) line, and the file overwrites the live copy
on the next server startup.  What fun!

Our engineers have already disabled and removed the script again, and
I've asked them to add a specific monitoring rule to our monitoring
systems that will continuously check for the presence of this script
and alert our team immediately if it is ever restored again by a
future update.

I apologize for the disturbance and the noise.

Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)