Subject: Outage on the way, one more time

From: Glen Barney <[email-masked]>
Date: Sun, 24 Apr 2011 21:54:40 -0700

Dear IETF Leadership -

For the past two weeks we have been operating out of our backup colocated
datacenter, for the dual purposes of (a) testing and verifying that such 
operation would work as expected, and (b) moving us to upgraded server
hardware and operating software.

I'm pleased to report that, having reached the end of the two-week period,
it appears that everything has gone extremely well.

Apart from one individual who couldn't follos us over because he was using 
an old hostfile-override, a brief delay in getting IPV6 reverse DNS
delegation in place, and a needed spam filter update to allow the telechat
agendas to pass through, I've been very pleased with the LACK of problems
pertaining to this test and upgrade.  Everything has gone better than I
hoped.

It is now time to bring things back to our primary data center in Fremont,
and to the now newly-upgraded-and-cloned servers ready to go here, and we
will perform that process today, Monday, April 25th.

As before, we will bring the Orlando server down into a standby mode (the
server will be up but services like web, FTP, mail and rsync will be halted), 
do a final controlled synchronization of data from the Orlando server to the 
Fremont server, update the zone files, and then bring the Fremont server 
online.

This procedure will begin at 1PM Pacific, 4PM Eastern, 2000GMT, and is
expected to take approximately two hours.  It may take less, or more, but
if things are going to go beyond FOUR hours I will send a direct message
to each member of these lists, advising of the delay.  Failing over, the
entire process took two hours and five minutes, so I'm estimating that as
a target time for fail-back as well.

Again, please remember - although I *hope* for no problems, I always *expect* 
that there will be problems.  So, as before, we will of course be monitoring 
things from here, just to make sure everything works.  Please report problems 
in the usual manner (to IETF-ACTION).  If you feel the need, you can email me 
directly out of band at [email-masked].

Thank you all for your patience as we bring this upgrade and test to a close.
As always, if there are any questions, please let me know.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)