Mon Mar 19 12:44:54 PDT 2007 — Santa Rosa colocation stays up. Early Friday morning we experienced an outage on one of the two AT&T-provided Gigabit links leaving our Santa Rosa datacenter. Our Network Operations Center worked to isolate the trouble, which turned out to be a failed regenerator card in a Fremont central office. The card was replaced around 4:00pm, bringing the link back into service. The end result was a total of 2 seconds of packet loss for colocated customers, plus a very happy NOC staff. It’s always fun to see our carefully planned redundancy work in the real world! -Nathan and the NOC (Jared, Zeke, and Matt)

Thu Mar 15 13:34:37 PDT 2007 — PG&E Service Outage at our Santa Rosa datacenter and colo facility. We lost PG&E electrical services briefly this morning; Our backup power systems functioned perfectly. By the time we’d made it to the electrical room our genset had already been automatically started and the ATS had just switched over to it. There was no interruption of any services in or provided by our datacenter. It was exciting and reassuring to see everything work exactly like it is supposed to! -Sonic.net Operations Staff

Wed Mar 14 19:04:41 PDT 2007 — Partial DSL outage. A software problem in our provisioning tools caused a ‘sync no surf’ condition for about 8% of DSL subscribers. The duration was about 15 minutes. We are working to identify and correct the bug as we speak. – Eli, Nathan, Ross

Mon Mar 12 12:47:39 PDT 2007 — Sacramento Area DSL Circuit Upgrade. Early Wednesday morning at approximately 12:01 AM, the circuit that handles all DSL traffic for the Sacramento area is going to be migrated to a new, larger circuit. Customers may experience up to 30 minutes downtime as they are migrated in batches from the old to new circuit. -Nathan and Kelsey 😉

Thu Mar 8 19:56:02 PST 2007 — MOTD Correction. The DSL maintenance the MOTD reported as being scheduled for early Saturday Morning is actually going to be taking place in several hours, not tomorrow night. We apologize for any confusion. -Kelsey

Wed Mar 7 16:23:26 PST 2007 — Planned power and DSL maintenance. During the day tomorrow we will be doing DC power plant maintenance at one of our colo facilities in San Francisco. There should be no associated services interruptions of any kind; however, there is the potential that power could be lost to some of our DSL termination equipment. In the event that power is lost to any of our equipment the associated outage would be brief.

Early Saturday at 12:01AM, a number of planned events may affect a number of short outages to some DSL customers terminated in San Francisco. Jobs to be done include, installation of additional ATM switch ports, turn up of additional DSL termination equipment, and restoration of one of the ATM OC-12s to it’s primary path through our in building patching and switching. Affected customers should see a total of no more than 15 minutes of combined outage spread over a number of short events. -Sonic.net Network Operations

Wed Feb 28 15:52:46 PST 2007 — DSL Termination Upgrades. Tonight, shortly after Midnight, some DSL circuits terminated on our equipment in San Francisco may experience a 5 to 10 minute outage as we perform some upgrades and regular maintenance on the termination equipment. -Kelsey and Nathan

Update Thu Mar 1 01:07:21 PST 2007 — We’re done! Maximal customer downtime was 4 minutes, with most folks seeing only a minute or two of connectivity loss. -Nathan and Kelsey

Wed Feb 21 13:37:08 PST 2007 — Planned down time for maintenance. Tonight at midnight one of our core switches will be taken down briefly; during this time reachability to Sonic.net services may be interrupted briefly. Also during this time the customer Database System (custsql.sonic.net) will be taken down for hardware maintenance; expected down time is less than 15 minutes. -Augie and Jared.

Update Thu Feb 22 01:31:14 PST 2007 — Maintenance complete and all services are now restored. -Augie and Jared.

Tue Feb 20 18:48:53 PST 2007 — Brief Web Cluster Instability. For a short period some issues in our webcluster caused some sites to fail to load. Service were restored quickly. -Kelsey and Nathan

Fri Feb 16 18:30:26 PST 2007 — DSL Service disruption. One of the OC-12s that terminates DSL subscribers in LATA 1 (most of Northern California) is offline. Our NOC staff is already onsite at the POP and is working with AT&T and the facility providers to ensure a speedy fix. -Kelsey, Alan and the 7065 NOC.

Update Fri Feb 16 19:40:40 PST 2007 — Service restored. As of approximately 7:30pm all services on the affected OC-12 have been restored. The problem was determined to be faulty cross-floor fiber patching at our 200 Paul Ave. facility. To expedite service restoration we brought some otherwise idle gear into service, which allowed us to bring customers back by eliminating intermediate fiber connections. We will be scheduling a maintenance window in the near future to clean the cross-floor run and return the circuit to its original configuration. We apologize for the late update to this entry, not to mention the downtime! -Nathan, Zeke, Matt and Juston