Wed Apr 18 07:46:24 PDT 2007 — Power issue at our Santa Rosa datacenter. This morning at approximately 6:49am one of the UPSes that feeds our Santa Rosa colocation facility dropped its critical load, causing the 15 customers connected to the UPS to lose power. When we arrived on-site to investigate, the power room where the UPS is located smelled of burnt plastic and the input circuit breaker to the UPS was tripped. We placed the load into external bypass around 7:10am, at which time power was restored.
At this time, we surmise that a massive internal failure inside the UPS caused the fault. A UPS technician is currently en route to do further diagnostics, required repairs, and to start the unit back up. Until fixed, we will be operating with the UPS’ critical load supported by our Automatic Transfer Switch. Our generator is running in the event that our building PG&E feed fails.
On the plus side, Sonic.net services such as mail, web, ftp as well as our core networking equipment was unaffected by the outage. All of this critical infrastructure is fed by both of our datacenter UPSes to handle circumstances such as this.
-Nathan, Kelsey, Dane, Jen, Clay and everyone in Support