Sun Dec 15 12:45:25 PST 2002 — BroadLink Tower Five outage due to power failure. From BroadLink:
Ok, it _is_ a power problem, it looks like power went down around 2:00pm (Saturday), our battery system and generator kept it up until around 11:00pm. We’re headed up the mountain to pull the system back online. Tim will be onsite in the next few hours and we should be able to get the generator running again (he’s bringing another generator just in case) The tower itself is our most remote site, when it’s bone dry it takes a 4×4 to reach it. In this weather we may have to hike a portion of the way. PG&E doesn’t have any information on the outage, no word as to the cause but lightning is most likely. They also don’t have any information regarding power restoration. As soon as more information is available I will relay it to sb-ops (Sonic.net). -Jason (BroadLink)
Update: Sun Dec 15 12:49:23 PST 2002, The repair team’s vehicle is stuck in the mud, they are now hiking. More news as it unfolds. -Jason
Update: Sun Dec 15 14:53:20 PST 2002, It looks like BL’s generator isn’t working and we can’t move a replacement on foot. We’re in contact with the property owner to find another option (there are working generators on-site owned by the property, our eqpt isn’t hooked up with them). No useful information from PG&E yet. -Jason
Update: Sunday, December 15, 2002 4:23 PM, PG&E has finally confirmed the outage, they have no time frame for restoration. -Jason
Update: Sun, 15 Dec 2002 19:35:53 -0800, Tower 003 has recently lost power. The tower is currently operating on battery reserves. If the outage is long enough we may have a new problem.
With regards to tower 005, we still have one vehicle stuck midway up the hill. The rescue 4×4 with winch was not able to pull it free. The rain and wind are expected to continue until Tuesday and we’ve no useful data from PG&E regarding the restoration of land-line power to either tower. We’re adding a generator to 003 right now (the ‘real’ generator for tower 003 is halfway up Barham mountain in the mud-locked vehicle). Surprise surprise you can’t buy a generator in S.R. tonight for love or money. The generator should allow us to avoid knocking any additional customers offline. We’re actively looking for a solution to the tower 005 problem. A visual inspection did not reveal any damage which is probably a good sign. When power is restored we expect full functionality to return.
So, in summary: It’s possible tower 003 may have trouble in the next 4-12 hours. We’re doing what we can to avert it.
Tower 005 is still down, unless PG&E repairs that portion of the power grid the tower will be down until we can get a working generator online. We’re trying to find a way to accomplish it but regardless of what we work out ‘down here’ nothing can actually be done until someone can get up the darn hill. With rain continuing the road conditions have steadily degraded from pretty bad to practically insane. Finding a solution that doesn’t require us to wait for the weather to improve (which would probably mean Wednesday or Thursday) is clearly our top priority. -Jason
Update: Sun, 15 Dec 2002 21:14:06 -0800, PG&E power back up to 003 so UPS is re-charging and ready to go.
I will continue to work on the 005 in the AM (MT. Barham) issue that serves all of Rincon and Bennett Valley customers as well as some connected in western Santa (both north and south). Once I get my vehicle out of the way (presently blocking the entire muddy one-lane road) I am seeking to rent an ATV to pull the generator the rest of the way the hill to the tower to restore the power.
PS – We took advantage of Dane’s generous offer to use the hand-cart generator. I will use that on the 003 site if the power dumps again. -Tim (BroadLink)
Update Mon Dec 16 09:53:00 – Tim at Broadlink has sourced a generator that can be connected onsite, with the help of KXFX Radio! They estimate that they will have the Barham tower site back online later today. – Eli
Update Mon Dec 16 16:32:55 – KXFX’s onsite generator has been connected to the Barham Tower site, and the radios are associated with the customer radios. However, the backhaul into the rest of Broadlink’s network has not come up. BL’s engineers fear that the backhaul equipment may have been damaged, and work continues. – Eli and Broadlink
Final Update Thu Dec 19 08:37:47 – After much hard work, Barham is fully operational, and service has been restored! – Eli and Broadlink