Subject: Overnight slowdown report

From: Glen <[email-masked]>
Date: Thu, 26 Apr 2012 10:40:52 -0700

Good morning -

Last night at about 10:50PM Pacific time, traffic out of the AMS primary data
center in Fremont became extremely slow.  During that time, web pages for
all sites may have been served slowly, or become stalled.  Our engineers
came online and performed circuit and server checks, and before any cause 
could be positively identified, normal speeds had been restored.  The total
slowdown period was 39 minutes.

During the testing we did determine that all of our servers were up and 
functioning properly; in addition, our circuit path from our center to
AT&T was also up and functioning properly.  (We were able to connect remotely
through our backup circuits without issue.)  We therefore suspect that 
something upstream was occurring, and have opened an inquiry with AT&T to 
see if the cause can be determined.

No machines were compromised, no email or data was lost, and no damage 
occurred.  As always, any questions, please feel free to let me know.

Thank you,
Glen

Glen Barney
IT Director
AMS

---

From: S Moonesamy <[email-masked]>
Date: Thu, 26 Apr 2012 10:56:37 -0700

Hi Glen,
>slowdown period was 39 minutes.

I was going to send an email about that.  The ietf.org and 
rfc-editor.org network was not reachable during that time.  The last 
reachable hop was cr81.sj2ca.ip.att.net (12.122.1.118).

As the known point of contact were unreachable, I gave up on sending 
a notification.

BTW, there is a similar issue right now.

Regards,
S. Moonesamy

---

From: Glen <[email-masked]>
Date: Thu, 26 Apr 2012 12:02:03 -0700

> BTW, there is a similar issue right now.

We are not experiencing any issues at the moment, nor have we all morning.

If you are having issues, you are welcome and encouraged to open a ticket
by emailing [email-masked] and the duty staff will assist you.

Thanks,
Glen