Sun Dec 15 12:45:25 PST 2002 — BroadLink Tower Five outage due to power failure. From BroadLink:

Ok, it _is_ a power problem, it looks like power went down around 2:00pm (Saturday), our battery system and generator kept it up until around 11:00pm. We’re headed up the mountain to pull the system back online. Tim will be onsite in the next few hours and we should be able to get the generator running again (he’s bringing another generator just in case) The tower itself is our most remote site, when it’s bone dry it takes a 4×4 to reach it. In this weather we may have to hike a portion of the way. PG&E doesn’t have any information on the outage, no word as to the cause but lightning is most likely. They also don’t have any information regarding power restoration. As soon as more information is available I will relay it to sb-ops (Sonic.net). -Jason (BroadLink)

Update: Sun Dec 15 12:49:23 PST 2002, The repair team’s vehicle is stuck in the mud, they are now hiking. More news as it unfolds. -Jason

Update: Sun Dec 15 14:53:20 PST 2002, It looks like BL’s generator isn’t working and we can’t move a replacement on foot. We’re in contact with the property owner to find another option (there are working generators on-site owned by the property, our eqpt isn’t hooked up with them). No useful information from PG&E yet. -Jason

Update: Sunday, December 15, 2002 4:23 PM, PG&E has finally confirmed the outage, they have no time frame for restoration. -Jason

Update: Sun, 15 Dec 2002 19:35:53 -0800, Tower 003 has recently lost power. The tower is currently operating on battery reserves. If the outage is long enough we may have a new problem.

With regards to tower 005, we still have one vehicle stuck midway up the hill. The rescue 4×4 with winch was not able to pull it free. The rain and wind are expected to continue until Tuesday and we’ve no useful data from PG&E regarding the restoration of land-line power to either tower. We’re adding a generator to 003 right now (the ‘real’ generator for tower 003 is halfway up Barham mountain in the mud-locked vehicle). Surprise surprise you can’t buy a generator in S.R. tonight for love or money. The generator should allow us to avoid knocking any additional customers offline. We’re actively looking for a solution to the tower 005 problem. A visual inspection did not reveal any damage which is probably a good sign. When power is restored we expect full functionality to return.

So, in summary: It’s possible tower 003 may have trouble in the next 4-12 hours. We’re doing what we can to avert it.

Tower 005 is still down, unless PG&E repairs that portion of the power grid the tower will be down until we can get a working generator online. We’re trying to find a way to accomplish it but regardless of what we work out ‘down here’ nothing can actually be done until someone can get up the darn hill. With rain continuing the road conditions have steadily degraded from pretty bad to practically insane. Finding a solution that doesn’t require us to wait for the weather to improve (which would probably mean Wednesday or Thursday) is clearly our top priority. -Jason

Update: Sun, 15 Dec 2002 21:14:06 -0800, PG&E power back up to 003 so UPS is re-charging and ready to go.

I will continue to work on the 005 in the AM (MT. Barham) issue that serves all of Rincon and Bennett Valley customers as well as some connected in western Santa (both north and south). Once I get my vehicle out of the way (presently blocking the entire muddy one-lane road) I am seeking to rent an ATV to pull the generator the rest of the way the hill to the tower to restore the power.

PS – We took advantage of Dane’s generous offer to use the hand-cart generator. I will use that on the 003 site if the power dumps again. -Tim (BroadLink)

Update Mon Dec 16 09:53:00 – Tim at Broadlink has sourced a generator that can be connected onsite, with the help of KXFX Radio! They estimate that they will have the Barham tower site back online later today. – Eli

Update Mon Dec 16 16:32:55 – KXFX’s onsite generator has been connected to the Barham Tower site, and the radios are associated with the customer radios. However, the backhaul into the rest of Broadlink’s network has not come up. BL’s engineers fear that the backhaul equipment may have been damaged, and work continues. – Eli and Broadlink

Final Update Thu Dec 19 08:37:47 – After much hard work, Barham is fully operational, and service has been restored! – Eli and Broadlink

Sun Dec 15 05:55:00 PST 2002 — Sebastopol POP trouble. We are experiencing problems with the Sebastopol POP affecting customer who dial 707-823-8812. We have a ticket open with Pac Bell and are working to get this resolved. Please remember that we have alternate dial-up numbers available at www.sonic.net/cgi-bin/pops.pl -Matt

Update: Sun Dec 15 11:12:26 PST 2002, Power outage in Sebastopol resolved. A long power outage in Sebastopol caused our POP to go off-line for a few hours this morning. Power has been restored and the POP is answering modem calls again.

Sat Dec 14 10:29:05 PST 2002 — Intermittent “connection refused” from mail.sonic.net. For a few minutes this morning, connections to mail.sonic.net were sometimes refused. By the time we had logged onto the boxes to correct the problem, the problem had gone away. Five servers in our mail cluster serve mail.sonic.net.

We are soon to deploy a new mail server architecture, which will ensure that mail.sonic.net always answers connections. -Scott and Matt

Sat Dec 14 17:38:47 PST 2002 — Sonic.net absorbs SBC-ASI DSL customers of DSL Designs. On Saturday December 7th, in an agreement between DSL Designs and Sonic.net, we have taken over the responsibility for serving the majority of DSL Designs SBC-ASI connected DSL customer base. DSL Designs will continue operations, serving Verizon connected customers.

Customers who migrate from DSL Designs to Sonic.net can have their email forwarded to avoid any loss of correspondence. Sonic.net staff will coordinate the email forwarding with the DSL Designs management as subscribers migrate. Many of the new customers are now online with us, and others are coming online in the first few days of next week. -Dane

Sat Dec 14 14:28:03 PST 2002 — Nokia Rooftop in Rohnert Park ‘R’ Section down. The T1 that serves Nokia Rooftop customers in Rohnert Park ‘R’ Section is down. A ticket has been opened with Pac Bell and we expect a reasonable repair in spite of the weather. -Matt

Update: Sat Dec 14 14:56:50 PST 2002, Power outage in Rohnert Park ‘M’ and ‘R’ Sections. After calling down to Rohnert Park, it was determined that the power is out in much of this area. We expect service to be restored when PG&E resolves the issue.

Fri Dec 13 15:47:09 PST 2002 — Our unix shell server, bolt.sonic.net is currently having problems, and the ops team is working on it. -Ops

Update: We’ve restored shell services on a standby server and will be investigating bolt’s failure. -Kelsey, Eli and John.

Fri Dec 13 14:53:26 PST 2002 — Last chance for pre-Christmas DSL installation. If you’re interested in obtaining new DSL service prior to Christmas, order today! We’re still offering free equipment for new customers, and the pricing starts at a total delivered cost of $49.95, just $31 more than a basic Sonic.net account.

You can order at: www.sonic.net/sales/dsl/pacbell/

If you are interested in giving DSL as a gift, phone our sales group at 707-522-1000 to make special arrangements.

SBC-ASI is scheduling installations for orders placed today for December 20th. We’ll make our best effort to get you a pre-Christmas installation date if you’ll get your order in by the end of the day today. Installation time-frame cannot be guaranteed by Sonic.net.

Thu Dec 12 12:37:38 PST 2002 — Vi upgraded on Bolt (shell.sonic.net). We have updated the vi/vim package on Bolt. This is a bugfix. -Kelsey and Scott

Thu Dec 12 12:35:30 PST 2002 — New mysql server program on Timber (sql.sonic.net). We have upgraded the mysqld on Timber. This was an urgent, unannounced upgrade to fix a security vulnerability. Downtime for the server was less than a minute. -Kelsey and Scott

Thu Dec 12 11:21:16 PST 2002 — Two of our name servers had been responding slowly to DNS request resulting in occasional failures and apparently slow interactive performance for some customers. They had not slowed down enough to trip an alert in our monitoring software. After it was brought to our attention, we took some brief traces to catch the failure and then restarted both of the servers. -Kelsey and Nathan