Our Santa Rosa datacenter facility is currently running on our backup generator due to problems with the automatic power transfer system. While we do not anticipate this will cause any impact for customers, it’s certainly a failure of a sort. Normally, the transfer switch starts the generator during a utility failure, but today it has triggered without cause. We are working to get back onto primary utility now.
Update: We’re back on utility, but managed to cause a fault our air handlers in the process of going back and forth multiple times. This bumped temperature in the datacenter from the typical 69 degrees up to over 90. All systems are back online now, and temperatures are now close to normal. There was no customer impact during this partial failure. -Dane, Russ, Kelsey, Don, Nathan and Jen
Update: ATS Fault Analysis and Repair. After reviewing the situation with a GE/Zenith support tech last night it was concluded that the modular timer responsible for automatic genset exercising and transfer tests was triggering the erroneous transfers to emergency power. This timer was ‘off’ when it failed and, indeed, never been used as we prefer to manually initiate our weekly genset exercise tests. The faulty timer has been removed and we have every confidence that our ATS will work as expected from now on. -Kelsey, Russ and Nathan
“Hal, resume operations using primary utility.”
“I’m sorry, Dane, but I cannot let you do that…”