Fri Nov 1 14:23:47 PST 2002 — BroadLink had a scheduled power outage at one of their tower sites this morning, but the UPS system failed. They replaced the equipment quickly to get customers back online. -Dane
Update, Fri Nov 1 17:32:08 PST 2002: A second outage occurred, and has been resolved. We expect at least one more once PG&E competes their work. As it’s both informative and funny, I’ll include an excerpt from the internal Sonic.net/BroadLink staff discussion list that explains the trouble. The following was written by BroadLink’s wonderful Jason Kane:
Regarding what happened:
As noted in the previous message PG&E was putting up a new power poll across the street from the tower site. As a result everyone in that area lost power for the day. It’s my understanding that wire line power will be restored in a few hours.
We originally believed that the scheduled power outage would not effect our customers since we have battery backup and a generator to re-charge it. The UPS failed immediately and the tower went dark. That was this morning.
To fix the problem we replaced the UPS with a mostly-charged unit, gassed up the generator, plugged our hardware into the UPS and plugged the UPS into the generator. Everything came back up and we figured our only problem was making sure the generator had plenty of gas. But the universe decided that today would be a good day for tweaking with the otherwise idyllic lives of BL and it’s faithful customers. As you may be aware, the tower took a nose-dive about twenty minutes ago. We had Tim nearby so he checked it out. The assumption was (of course) that the generator ran out of gas. But low and behold the generator was still cranking along without a hitch (a questionable metaphor but you get the idea). The UPS on the other hand was in a world of it’s own. And in that world, restarting every few seconds is some sort of imperative.
When we unplug the UPS from the generator it emerges from this malady and runs the tower off battery. But as every little gelfling knows you can’t run on batteries forever, even if you’re a pink bunny. So we rework the setup and plug everything into the generator. Fingers crossed the switches are flipped and the information age continues.
It is my belief that the power from the generator was not smooth/clean enough for the UPS. While it’s rather strange to have a UPS that’s more picky about clean juice than the switches, radios, management units and imported dancing hula girl lamps that are stuffed into the tower we’re forced by irrevocable circumstance to continue living without an adequate answer to such questions.
Here’s the basic sequence of events:
8:20am – PG&E disconnects power to tower 8:23am – broadlink battery backup system fails … outage 9:33am – battery system replaced, generator added 2:43pm – power supplied by generator kills replacement battery system … outage 3:33pm – tower rewired to run directly off generator, service restored
PG&E is supposed to finish their work today so we can rewire to run off primary power once again. We’ll test the battery system and replace it if needed to prevent a similar problem from recurring. We also now know that our generator can’t be used to re-charge our battery backups.
-Jason (BroadLink)
I hope that you found this as amusing as I did. -Dane