Subject: Web service slowdown Sunday evening

From: Glen <[email-masked]>
Date: Sun, 26 Jun 2011 19:27:06 -0700

Dear I-* Leadership:

On Sunday evening at approximately 5:15 PM Pacific time, the IETF experienced
a significant slowdown in web services, which lasted for about 30 minutes.

I was online during the last half of that period, trying to determine the
cause (i.e. "catch it in the act".)  It appears that the Apache web server
was experiencing a spike in usage on the datatracker, had exhausted all of 
the server memory available to it, and was trying to cope with that fact.

I restarted the web server which cleared the problem.  I also did a reboot
of the full server, just as a precautionary step.

I have made some tuning changes both to server/kernel memory management, as
well as the apache web server, and have a number of more aggressive steps I
can take if this problem occurs again.  For now, I will be performing 
additional monitoring and fine-tuning this evening, and we will of course
continue to monitor going forward.

As always, any questions, let me know.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)