Subject: Slowdown recurring - Potential datatracker issue causing web server failure

From: Glen <[email-masked]>
Date: Mon, 27 Jun 2011 07:08:27 -0700

All -

Good morning.

As many of you are aware, we had two more slowdowns following the one I
reported.  This is bad.

In the first case, which occurred at about 0440 Pacific time, the server 
rebooted itself (correctly) before I could get to it and came back online.
This is good, because at least we know the server reboot system appears to
work.

In the second case, which occurred at 0645, I caught it in the act; and,
although the server was heavily impacted, I was able to get control of it
before a reboot.  This is also good, because I was able to catch things in
the act.

As was the case yesterday, the impacted process was Apache.  This time, I
was able to also confirm that a python process being kicked off by apache
was failing.  This process caused the apache footprint to go from a normal
resident memory size of 55M to a size of 12GB.  Swap was almost completely
exhausted, and the server was crawling.

11744 wwwrun    20   0 19.2g  12g 5140 D    6 78.2  22:11.82 httpd2-prefork     
13641 wwwrun    20   0  578m  57m 6700 D    8  0.4   4:51.13 httpd2-prefork     
16574 wwwrun    20   0  569m  55m 6436 S    0  0.3   1:52.44 httpd2-prefork     
17010 wwwrun    20   0  574m  59m 6584 S    0  0.4   1:50.91 httpd2-prefork     

It has been mentioned by Steven that he was trying to enter a position at
the time, and the server choked right when he tried to do so.  Regardless
of whether his action triggered this, the hints I'm seeing make me believe
at this point we have this down to something within the datatracker, and we
need to work further to find out what is happening.

I am copying the Tools Team, and Henrik, and will be asking them to help with
log analysis to see if they can determine why the webserver is crawling.  As
you can guess from the above output, I have a snapshot program running which
takes server information every five minutes and writes it to a file.  Now
that I know this is a webserver/python (and therefore datatracker) issue,
I've added additional data to the file capture.  Still, I hope that we will
be able to resolve this quickly.

Note that since nothing has changed recently in terms of Python or the
datatracker (or the server), it may take some time to localize this.  It
could be some heretofore undetected recursion issue or something similar.
This is why I will be asking the tools team to step in and assist.

Thanks,
Glen
Glen Barney
IT Director
AMS (IETF Secretariat)