Sun Jul 21 07:24:36 PDT 2002 — Night Operations Complete. We’ve made a massive move of our core storage architecture and associated servers. The migration went very well, and downtime of most services was quite a bit shorter than planned. Servers were taken offline at 1:30am. Web hosting was down for a bit over an hour. Mail took longer due to the backlog of inbound and outbound mail, and took about three hours to complete. Peak load average observed on one of the mail servers during this time was 992.99 (compare to a typical of under 5.00). FTP was down for about an hour and a half, later in the morning.
The clock on our primary administrative box came up with the incorrect time, and executed some scheduled tasks far out of schedule. This resulted in some invoices for colocation, disk usage and bandwidth usage being run when they shouldn’t have. If you have an email in your inbox regarding billing for one of these type of services this AM, please disregard! No actual charges were made.
We had some unexpected problems with one of our nameservers, and a few other minor challenges, but things look quite healthy now and we’re very pleased with how smoothly this transition went. This completes the majority of our move into our new datacenter facility, and we’re very excited.
If you should observe any odd behavior, please post to news:sonic.net, or email support@sonic.net, or call support at 707-547-3400 and explain what you’re observing. Ask support not to wake us if it’s not a critical item. =)
Thanks to the team here for all of their help! While I might have to ask forgiveness for missing someone, here’s a list of the folks who worked on this tonight. Ops: Nathan, Kelsey, Scott, Russ, Steve, Matt and myself. Techs: ChrisM, Jeff, Aaron, MattS, ScottB, ChrisB, Dan, Kavan, Bryan. Guest helpers: JenM DustinM
-Dane (very happy to be almost completely moved, and looking forward to my vacation which begins on Wednesday.)