Subject: Fwd: delays in email since early this week....

From: "Barbara B. Fuller" <[email-masked]>
Date: Thu, 24 Feb 2005 14:32:54 -0500

Dear IETF Working Group Chairs:

I am forwarding a message that Scott Blomquist of the IETF Secretariat's 
technical support team sent to the IESG earlier today.  The message 
concerns the mail delays on IETF mailing lists that many of you experienced 
this week.  The message provides information about what happened and what 
the Secretariat did...and plans to do...to address the problem.  If you 
have any questions or concerns, then please feel free to contact Scott at 
[email-masked] (and please cc me on the message :-)

Regards,

Barbara
----------------------------------------------------------------------
>Envelope-to: [email-masked]
>Date: Thu, 24 Feb 2005 06:59:35 -0500 (EST)
>From: [email-masked]
>To: [email-masked]
>X-Spam-Score: 0.3 (/)
>X-Scan-Signature: fb6060cb60c0cea16e3f7219e40a0a81
>X-Mailman-Approved-At: Thu, 24 Feb 2005 08:16:54 -0500
>Cc: [email-masked]
>Subject: delays in email since early this week....
>X-BeenThere: [email-masked]
>X-Mailman-Version: 2.1.5
>Reply-To: [email-masked]
>List-Id: iesg.ietf.org
>List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/iesg>,
>         <mailto:[email-masked]?subject=unsubscribe>
>List-Post: <mailto:[email-masked]>
>List-Help: <mailto:[email-masked]?subject=help>
>List-Subscribe: <https://www1.ietf.org/mailman/listinfo/iesg>,
>         <mailto:[email-masked]?subject=subscribe>
>Sender: [email-masked]
>X-Spam-Score: 0.3 (/)
>X-Scan-Signature: 25620135586de10c627e3628c432b04a
>
>
>Hi All,
>
>I'd like to take a moment to explain a little about the mail delays
>this week.  I have not gone through all the logs as of yet, but I will
>before the meeting and will send a follow up email if there is any
>more to add (I should be around the helpdesk/noc most of the meeting
>if anyone would like to stop by and say hi)
>
>What is our setup?
>
>    exim
>    - defer all incoming mail
>    - strict in order delivery of queued mail
>    - 5min timeout on connect
>
>What we were seeing?
>
>  - outgoing mail volume up ~1,000,000 messages delivered between Mon-Wed
>  - mail delays up to 24hours (some lists were faster since I was
>    manually pushing through on Tue-Wed)
>  - queue lengths +26000 at some times on Tue
>  - temp failure on all verizon.net and some attbi accounts (all
>    messages deferred re-queued and tried again) This started happening
>    sometime over the weekend
>
>Why did this cause such a failure?
>
>My current feeling is that the volume and the verizon.net deferrals
>caused much of the problem.  The deferrals were not failures so each
>time the queue ran (in order) it tried to hit every queued verizon.net
>address again.  At one point we had thousands of deferred @verizon.net
>messages queued.  If each connect takes 5mins to timeout one can see
>where the problem can compound quickly.
>
>Are we ok now?
>
>We seem to be ok at the moment.  There are no waiting (non
>first_delivery) messages in the queue as of 6am EST.
>
>What did we do to put us back on track?
>
>It took some time to figure out where we were getting stuck.  Once I
>found the failure mode of the verizon.net addresses and found a
>solution, things started getting better.  It turns out sometime over
>the weekend verizon.net started using some sort of asynchronous
>callout to verify senders.  There have been some recent complaints
>about this on the internet.  As soon as we started accepting these
>callouts verizon.net messages started getting delivered, but by that
>time the damage was done and the queue was just going to take time to
>run.
>
>A couple of the settings on the system pushed the problem into the
>foreground more quickly.  The one that hurts the most is "strict in
>order delivery of queued mail".  For the moment I have turned this
>off.  I also changed "defer all incoming mail" to defer non local mail
>and changed the "5min timeout on connect" to 30s.  All these changes
>should help in the future with this problem.
>
>What about the future?
>
>The plan I am considering is implementing another mail queue which
>would handle any non local/non first_delivery mail deferred longer
>than 1hour.  This should alleviate most of the problem we were seeing.
>Mail that is fast should go out fast.  Mail that is slow will get sent
>to the other queue and go out slow.
>
>That's about it.  Please feel free to send mail if you have any
>questions or concerns.
>
>Thanks
>
>sb. Scott Blomquist