Subject: Re: List moderation requests

From: Glen <glen@amsl>
Date: Wed, 05 Mar 2008 10:15:53 -0800

Dear WG Chairs -

I've received word here that some of you are not receiving moderation 
alerts.

Please assist me by verifying the following:

#1. Please make sure your list is configured to send these alerts.  In 
your list admin page general options, make sure "Should the list 
moderators get immediate notice of new requests, as well as daily 
notices about collected ones?" be set to "Yes".  You should also take 
the opportunity to check and update all the other settings in your list 
setup.  We are on the latest version of mailman - a much newer version 
than the old NSS version - so some options may be different or new.

#2. Make sure your personal email filters are set to route your mail 
appropriately.  Under NSS, the single list server had many names (lists, 
odin, optimus, megatron,...) which could cause many problems and issues. 
  As announced at cutover, we flattened the mail server into one name, 
"mail.ietf.org".  (We are aware that some requests came from 
"service1.amsl.com" - we have corrected that issue.)  At any rate the 
point is that the headers and from addresses of moderation messages WILL 
HAVE CHANGED with the cutover, so you should check your filters accordingly.

#3. Make sure your ISP isn't trapping or filtering or tagging mail. 
We've seen at least one reliable instance in which, with the cutover, a 
moderator's ISP is now tagging moderation mail as Spam (with quite a 
high score, interestingly), when it never did that before the cutover. 
New whitelisting and/or other rules might be needed.

#4. Make sure that your correct, current email address is configured in 
as the list owner and/or moderator.

#5. Finally, after doing all of the above, go in and totally clear out 
your moderation queue.

Having done all of those things, your lists should work.  If they still 
do not work after that, please send me an email message, including your 
name and your list name, and I will check it out, send a test, and send 
you a mail transcript of the moderation announcement (or fix whatever 
problem still exists with your list.)

PLEASE NOTE: I do not normally receive tickets directly, nor do I 
subscribed to any IETF lists, including this one.  Normal issues should 
go through ietf-action, as usual, with as many useful details as can be 
included.  To expedite this particular issue, you email me directly, but 
I do not normally see the discussions on this list.

Thank you,
Glen Barney
IT Director
AMS

---

From: Glen <[email-masked]>
Date: Wed, 05 Mar 2008 10:35:25 -0800

Robert Sparks wrote to me privately and said:
>> the opportunity to check and update all the other settings in your 
>> list setup.  We are on the latest version of mailman - a much newer 
>> version than the old NSS version - so some options may be different or 
>> new.

> We should still be getting the daily nags if this setting is "No". This 
> setting just affects getting an immediate notice with each message 
> instead of the periodic nag.
> (From the manual 
> at http://www.gnu.org/software/mailman/mailman-admin/node13.html - note 
> the "will always get a daily summary"
> admin_immed_notify
>     List moderators get notifications of pending administrative actions, 
> such as subscription or unsubscription requests that require moderator 
> approval, or posted messages that are being held for moderator approval.
>     List moderators will always get a daily summary of such pending 
> requests, but they can also get immediate notifications when such a 
> request is made. 
>     The admin_immed_notify variable controls whether these immediate 
> notifications are sent or not. It's generally a good idea to leave this 
> set to Yes.

Okay, I wasn't aware of that; however, I'll still need specifics to do a 
trace.

For everyone's info, I just reset the cron job that runs the automatic 
checks to 4AM Pacific Time (currently that would be noon GMT).  This 
will make it easier for us all to watch for and find.

But if you are not getting moderation messages, let's at least validate 
the immediate setting first, if you would.

Once I get all the people who have immediate mode turned on working, we 
can then address the cron job, if that's still not working.

I will check the transcripts tomorrow for the 0400 run coming up, and 
see what I see there.

Thanks,
Glen

---

From: David R Oran <[email-masked]>
Date: Wed, 5 Mar 2008 13:48:36 -0500

On Mar 5, 2008, at 1:15 PM, Glen wrote:

> Dear WG Chairs -
>
> I've received word here that some of you are not receiving  
> moderation alerts.
>
> #1. Please make sure your list is configured to send these alerts.   
> In your list admin page general options, make sure "Should the list  
> notices about collected ones?" be set to "Yes".  You should also  
> take the opportunity to check and update all the other settings in  
> your list setup.  We are on the latest version of mailman - a much  
> newer version than the old NSS version - so some options may be  
> different or new.
>
I don't follow this. why should I have to set this? Before I got one  
notification a day with this option set off. Now I get none. Seems a  
really bad tradeoff to receive 10-20 of these notices a day due to  
getting a message every time a new post gets added to the moderator  
queue.

If you want me to set this for testing purposes, that's fine, but it  
really does seem like there's a root cause that isn't getting analyzed  
by this recommendation.

> appropriately.  Under NSS, the single list server had many names  
> (lists, odin, optimus, megatron,...) which could cause many problems  
> and issues.  As announced at cutover, we flattened the mail server  
> into one name, "mail.ietf.org".  (We are aware that some requests  
> came from "service1.amsl.com" - we have corrected that issue.)  At  
> any rate the point is that the headers and from addresses of  
> moderation messages WILL HAVE CHANGED with the cutover, so you  
> should check your filters accordingly.
>
Checked - my filters re fine.

> We've seen at least one reliable instance in which, with the  
> cutover, a moderator's ISP is now tagging moderation mail as Spam  
> (with quite a high score, interestingly), when it never did that  
> before the cutover. New whitelisting and/or other rules might be  
> needed.
>
Nope.

> #4. Make sure that your correct, current email address is configured  
> in as the list owner and/or moderator.
>
yup.

> #5. Finally, after doing all of the above, go in and totally clear  
> out your moderation queue.
>
Done. Twice - once last week and once yesterday. Today there were 5  
new messages in the queue and no  notificaiton.

> Having done all of those things, your lists should work.  If they  
> still do not work after that, please send me an email message,  
> including your name and your list name, and I will check it out,  
> send a test, and send you a mail transcript of the moderation  
> announcement (or fix whatever problem still exists with your list.)
>
This email should serve that purpose.

> subscribed to any IETF lists, including this one.  Normal issues  
> should go through ietf-action, as usual, with as many useful details  
> as can be included.  To expedite this particular issue, you email me  
> directly, but I do not normally see the discussions on this list.
>
> Thank you,
> Glen Barney
> IT Director
> AMS

---

From: Glen <[email-masked]>
Date: Wed, 05 Mar 2008 11:08:33 -0800

David R Oran wrote:
> getting a message every time a new post gets added to the moderator queue.

Yes, you're right.  I hadn't been told that it was only the cron-based 
alerts that were failing.  I had only been told that "nobody was 
receiving moderation messages."

That distinction has now been made.

We DO see a problem with the cronjob.  I do not yet know what it is.  I 
will be testing it and working to resolve it.  Many lists are receiving 
daily notifications, including my test list.  Several lists are not. 
Mailman isn't even trying for those lists.  I have no idea why - yet.

Until I get this repaired, you MAY wish to turn on immediate notification.

Or you may wish to check your lists regularly, until I can get this 
resolved.

Thank you all for your assistance,
Glen

---

From: David R Oran <[email-masked]>
Date: Wed, 5 Mar 2008 14:14:22 -0500

On Mar 5, 2008, at 2:08 PM, Glen wrote:

>> I don't follow this. why should I have to set this? Before I got  
>> one notification a day with this option set off. Now I get none.  
>> Seems a really bad tradeoff to receive 10-20 of these notices a day  
>> due to getting a message every time a new post gets added to the  
>> moderator queue.
>> If you want me to set this for testing purposes, that's fine, but  
>> it really does seem like there's a root cause that isn't getting  
>> analyzed by this recommendation.
>
> Yes, you're right.  I hadn't been told that it was only the cron- 
> based alerts that were failing.  I had only been told that "nobody  
> was receiving moderation messages."
>
> We DO see a problem with the cronjob.  I do not yet know what it  
> is.  I will be testing it and working to resolve it.  Many lists are  
> receiving daily notifications, including my test list.  Several  
> lists are not. Mailman isn't even trying for those lists.  I have no  
> idea why - yet.
>

> Thank you all for your assistance,
Thank YOU! This is starting to make sense, at least...

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 06 Mar 2008 10:03:42 +0000

Glen and WG Chairs,

I would like to point out that there seem to be a default setting in the 
mailman interface that isn't in accordance with IETF moderation policy 
for mailing lists.

At the bottom of the general page there is a setting for:

"Discard held messages older than this number of days."

This was at least for the TSVWG mailing list set to 14 days. That needs 
to be 0 as automatic discarding of held messages isn't okay.

Glen, can you fix this globally or do each chair need to do it?

Cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: James M Galvin <[email-masked]>
Date: Thu, 06 Mar 2008 07:20:49 -0500

The setting is in accordance with IETF policy.  The default setting 
is 14 days.

With lists with public archives, providing an up-to-date archive of 
accepted postings so the originator can check the archive to see if 
their message has been distributed is permitted.  This is stated 
here:

    <http://www.ietf.org/IESG/STATEMENTS/mail-submit-policy.txt>

Jim

-- On Thursday, March 06, 2008 10:03 AM +0000 Magnus Westerlund 
<[email-masked]> wrote regarding Automatic discard 
of held messages in mailman --

> Glen and WG Chairs,
>
> I would like to point out that there seem to be a default setting
> in the mailman interface that isn't in accordance with IETF
> moderation policy for mailing lists.
>
> This was at least for the TSVWG mailing list set to 14 days. That
> needs to be 0 as automatic discarding of held messages isn't okay.
>

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 06 Mar 2008 12:30:51 +0000

.

James M Galvin wrote:
> The setting is in accordance with IETF policy.  The default setting is 
> 14 days.
>
> their message has been distributed is permitted.  This is stated here:
>

Although 14 days is less than a normal long vacation, so if both chairs 
take the same time off, mail gets dumped.  We should really have a 21day 
minimum for mail.

Stewart

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 06 Mar 2008 13:28:58 +0000

>    <http://www.ietf.org/IESG/STATEMENTS/mail-submit-policy.txt>

James, what I am complaining about is that I don't see how default drop 
of unattended moderation queues are within this policy. Because the 
moderator isn't doing his job in a timely manner the messages put into 
the moderation queue are dropped. The messages should be held at least 
long enough for occasions when all chairs are off-line for vacations, 
which means at least more than 3 weeks. I also would prefer that we ADs 
could get indications of when queues gets long and have a time to sort 
things out with chairs before the messages are being lost.

Cheers

Magnus Westerlund

---

From: James M Galvin <[email-masked]>
Date: Thu, 06 Mar 2008 09:55:54 -0500

-- On Thursday, March 06, 2008 12:30 PM +0000 Stewart Bryant 
<[email-masked]> wrote regarding Re: Automatic discard of held 
messages in mailman --

> Although 14 days is less than a normal long vacation, so if both
> chairs take the same time off, mail gets dumped.  We should
> really have a 21day minimum for mail.

The policy also states that moderated messages should be processed 
within 2 business days, which I would suggest means that if the 
complete set of moderators are unavailable for an extended period 
that they should find a substitute.

Jim

---

From: James M Galvin <[email-masked]>
Date: Thu, 06 Mar 2008 10:09:21 -0500

-- On Thursday, March 06, 2008 1:28 PM +0000 Magnus Westerlund 
<[email-masked]> wrote regarding Re: Automatic 
discard of held messages in mailman --

> James M Galvin skrev:
> > The setting is in accordance with IETF policy.  The default
> > setting is  14 days.
> >
> > With lists with public archives, providing an up-to-date
> > archive of  accepted postings so the originator can check the
> > archive to see if  their message has been distributed is
> > permitted.  This is stated here:
> >
> James, what I am complaining about is that I don't see how
> default drop of unattended moderation queues are within this
> policy. Because the moderator isn't doing his job in a timely
> manner the messages put into the moderation queue are dropped.

So I think there are two comments you're making here.

First, whether or not dropping messages from the moderation queue 
is within the policy and, second, what to do about moderators that 
are not doing their job.

With respect to dropping messages in the queue, I think that 
whether or not a message is in the queue is not that important.  I 
say this because the SPAM filtering is to simply drop messages with 
a high SPAM scores.  A message had to get past this action before 
getting in the queue.

I realize that a message in the queue has a higher probability of 
being a "valid" message that needs to be approved, but I don't 
think the fix here is to extend the "wait" time.  I think the fix 
is to address your second point, which is what to do about 
moderators that don't do their jobs.

>From a technical perspective, Mailman does not provide a convenient 
way to monitor moderators.  One feature it does have though is that 
there is an option that says "forward messages that would be 
discarded to the administrator."  However, I believe this means it 
would forward all the SPAM it discards so this is probably not an 
effective option.

>From a policy point of view, there are no consequences for failing 
to moderate in a timely fashion, at least none that are stated 
explicitly.  There's the usual nuclear option of replacing the 
Chair of the working group (who is the default moderator who adds 
other folks to help with the process), but I doubt we want to go 
there too often.  A Chair should simply replace the moderators.

I agree it's a problem but I don't see an easy fix.

Jim

> The messages should be held at least long enough for occasions
> when all chairs are off-line for vacations, which means at least
> more than 3 weeks. I also would prefer that we ADs could get
> indications of when queues gets long and have a time to sort

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 06 Mar 2008 15:35:53 +0000

James M Galvin skrev:
> <[email-masked]> wrote regarding Re: Automatic discard 
> First, whether or not dropping messages from the moderation queue is 
> within the policy and, second, what to do about moderators that are not 
> doing their job.
> 
> With respect to dropping messages in the queue, I think that whether or 
> not a message is in the queue is not that important.  I say this because 
> the SPAM filtering is to simply drop messages with a high SPAM scores.  
> A message had to get past this action before getting in the queue.
> 
> I realize that a message in the queue has a higher probability of being 
> a "valid" message that needs to be approved, but I don't think the fix 
> here is to extend the "wait" time.  I think the fix is to address your 
> second point, which is what to do about moderators that don't do their 
> jobs.
> 
> there is an option that says "forward messages that would be discarded 
> to the administrator."  However, I believe this means it would forward 
> all the SPAM it discards so this is probably not an effective option.
> 
> explicitly.  There's the usual nuclear option of replacing the Chair of 
> the working group (who is the default moderator who adds other folks to 
> help with the process), but I doubt we want to go there too often.  A 
> Chair should simply replace the moderators.
> 

 From my perspective I think there needs to be a bit more room for 
failure cases to avoid messages being dropped. I totally agree that it 
should be extremely rare cases when messages becomes trapped for more 
than 14 days, or even more than a few days.

Chairs in general needs to think more about ensuring that there are 
backup moderators. However, there are holidays seasons when large 
percentage of WG participants are on holiday.

Cheeer

Magnus Westerlund

---

From: James M Galvin <[email-masked]>
Date: Thu, 06 Mar 2008 10:46:08 -0500

-- On Thursday, March 06, 2008 3:35 PM +0000 Magnus Westerlund 
<[email-masked]> wrote regarding Re: Automatic 
discard of held messages in mailman --

> failure cases to avoid messages being dropped. I totally agree
> that it should be extremely rare cases when messages becomes
> trapped for more than 14 days, or even more than a few days.
>
> Chairs in general needs to think more about ensuring that there
> are backup moderators. However, there are holidays seasons when
> large percentage of WG participants are on holiday.

Works for me.

Again, the Mailman default is 14 days, although folks can set that 
to anything they want.

So you think the IETF default should be 21 days, or how about a 
month, 30 days?

That's a real question.  As part of the transition I'm documenting 
"little details" like this as guidance for the Secretariat.  I was 
leaving things at the Mailman default of 14 days but if folks 
generally believe the default should be more what do you recommend?

Personally, the only position I have is that the default should not 
be 0, meaning no expiration.  Given that it is permissible to drop 
messages I think that keeping messages around indefinitely has only 
marginal value.  Part of the reason for this is that if a message 
gets too old the actual discussion may have long since ended (for 
Internet definitions of long).

Jim

---

From: Glen <[email-masked]>
Date: Thu, 06 Mar 2008 09:05:03 -0800

James M Galvin wrote:
> Personally, the only position I have is that the default should not be 
> 0, meaning no expiration.  Given that it is permissible to drop messages 
> I think that keeping messages around indefinitely has only marginal 
> value.  Part of the reason for this is that if a message gets too old 
> the actual discussion may have long since ended (for Internet 
> definitions of long).

Not to mention (if I might be permitted to chime in for one comment) the 
fact that hundreds of thousands of messages in the highly-inefficient 
mailman-stores-everything-in-one-big-directory hold queue will degrade 
performance to the point of a grinding halt... and the fact that, 
realistically speaking, a moderator returning to a hold queue of 
thousands of messages on their list is just going to drop them all, or 
at least drop them all past a certain point (I obviously can't speak for 
everyone here, but there comes a point when there are just too many 
messages on hold for a moderator to ever *really* catch up on.)

So zero is a bad idea IMHO too.  You can each set your lists to whatever 
other default you want, but IMHO again, anything beyond 14 is really 
pointless except in rare cases.

Glen

---

From: Joel Jaeggli <[email-masked]>
Date: Thu, 06 Mar 2008 09:20:09 -0800

Glen wrote:
>> 0, meaning no expiration.  Given that it is permissible to drop 
>> marginal value.  Part of the reason for this is that if a message gets 
>> too old the actual discussion may have long since ended (for Internet 
>> definitions of long).
> 

I would just observe that the disruption that occurs when a converstion 
is ongoing and the moderator doesn't deal with held messages is 
considerable at 2 days let alone 14... By then people have either given 
up or found some other way to communicate. I'd consider myself derelict 
in my duties if I got behind beyond a couple days.

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 06 Mar 2008 17:39:18 +0000

James M Galvin skrev:
> Again, the Mailman default is 14 days, although folks can set that to 
> anything they want.
> 
> So you think the IETF default should be 21 days, or how about a month, 
> 30 days?
> 
> leaving things at the Mailman default of 14 days but if folks generally 
> believe the default should be more what do you recommend?

I would recommend 45 days. Simply to guarantee that any message dropped 
will definitely be of very little relevance.

> the actual discussion may have long since ended (for Internet 

Well, my initial argument was that messages ending up in the hold 
message are not automatically discardable due to that the automatic 
tests has been done and they can't with sufficient certainty decide if 
they are real and not and needs human intervention.

Regarding Joel's message. I fully agree that letting any message go 
close to being dropped shows a derelict of duty. And I do expect you all 
to do your best to ensure timely moderation of the hold messages.

Cheers

Magnus Westerlund

---

From: Joe Abley <[email-masked]>
Date: Thu, 6 Mar 2008 13:03:41 -0500

On 6-Mar-2008, at 12:05, Glen wrote:

> and the fact that, realistically speaking, a moderator returning to  
> a hold queue of thousands of messages on their list is just going to  
> drop them all, or at least drop them all past a certain point (I  
> obviously can't speak for everyone here, but there comes a point  
> when there are just too many messages on hold for a moderator to  
> ever *really* catch up on.)

We had a period during which [email-masked] was being used as the from  
address for a whole series of big spam runs, somewhere, and the  
mailman moderator queue began accumulating 5000+ messages per day in  
backscatter (e.g. challenges from remote TMDAs, "Archie Balloonmaker  
no longer works here", etc).

The mailman moderator interface available to me was woefully  
inadequate at dealing with those; even if the queue was small enough  
(a couple of thousand messages) that the web interface would function  
at all, it took several hours to page through them all.

In the end I gave up trying to handle this myself, and had Scott at  
NSS poke about under the hood to bulk-remove messages that matched  
obvious junk patterns in message body and header fields, until the  
list was small enough for me to finish the job by hand.

So anyway, it *is* possible to deal with very large moderation queues  
without having to resort to deleting everything without reading it,  
and it has certainly been done on at least one IETF list before :-)

Joe