Stockton DSL outages.

Wed Jul 12 14:07:15 PDT 2006 — Stockton DSL outages. We have confirmed DSL outages throughout the Stockton area. Customers may be experiencing sync with no ability to surf, or severe slow downs in service. AT&T is working to resolve the issues, but we have no ETR at this time. -Kavan, John, and Support Update: Thu Jul 13 15:25:45 — Connectivity to the affected customers has been restored. -Eli, Support

Network maintenance complete.

Wed Jun 21 01:03:05 PDT 2006 — Network maintenance complete. The scheduled network maintenance tonight went off without a hitch. Affected colo customers would have experienced less than 10 minutes of total downtime as we reloaded our distribution switch onto new code, and migrated it to gigabit uplinks. -Jared, Zeke, Nathan, and Matt

Brief System Event.

Tue Jun 20 14:16:21 PDT 2006 — Brief System Event. An administrative tool gone awry briefly impacted the file server that handles /home. A large number of systems depend upon /home being available and snappy including inbound mail, POP3/IMAP and webmail, as well as shell and FTP. -Jared

Upcoming network maintenance for colo.

Mon Jun 19 12:57:48 PDT 2006 — Upcoming network maintenance for colo. We will be performing a software upgrade on our primary colo distribution switch in Santa Rosa at 12:01 AM Wednesday, June 21. Most colo customers will experience 5-10 minutes of intermittent connectivity as we reload the switch onto new code. -Zeke, Jared, and Nathan

Sonic.net is currently experiencing network…

Wed Jun 14 08:40:42 PDT 2006 — Sonic.net is currently experiencing network instability we are investigating the problem and will be working to resolve it shortly. -Sonic.net Operations

Update: Wed Jun 14 09:07:45 PDT 2006 — This morning a major DoS attack created instability in part our network in Santa Rosa. Colocation and DSL customers served out of the Santa Rosa facility were affected. A reboot of one of our distribution switches halted the attack and we will continue to investigate. -Matt, Nathan, Augie, Clay and Dan

Emergency customer router maintenance.

Wed Jun 7 15:24:57 PDT 2006 — Emergency customer router maintenance. We will be replacing hardware on one of our customer routers this evening at 12:00. The hardware replacement should only cause a few minutes of downtime for 20 of our T1 customers. We apologize for any inconvenience this may cause. -Jared

Update: Thu Jun 8 00:38:58 PDT 2006 — Router maintenance complete. The maintenance went as expected. The card replacement and all associated downtime should have taken less than 5 minutes. All T1 lines on the affected hardware have been confirmed up and running. -Jared and Matt

DSL outage in LA.

Tue Jun 6 11:33:55 PDT 2006 — DSL outage in LA. Our ATM DS3 that serves DSL to the Los Angeles region went down for 5-10 minutes at approx 11:15 this afternoon. The circuit is back up now and all customers should be up and running again. We are investigating the cause with the provider of the DS3. -Jared, Russ, and Nathan

Update: Wed Jun 7 13:56:21 PDT 2006 — The LA region DSL circuit began to show errors again for about 30 minutes this afternoon. The transport provider was able to see the problem and has their engineers working on the problem now. At this time all customers should be up and running. -Jared

Update: Sat Jun 10 16:12:44 PDT 2006 — The LA region’s circuit is once again showing errors. We’re working with our transport provider to fix the link. Unfortunately, we don’t have an ETR, but are hopeful the problem will be resolved soon. -Nathan

Update: Sat Jun 10 20:35:16 PDT 2006 — The circuit has been repaired. Our transport provider found a failure in one of their long-haul transmission cards. The card has been replaced and we expect no further interruption on this link. -Nathan

Broadlink backhaul outage.

Sat May 20 18:35:00 PDT 2006 — Broadlink backhaul outage. Broadlink reports that on May 19th at about 2PM, the Broadlink backhaul went down without warning. After many hours of troubleshooting, they were able to remove the faulty radio equipment and are in contact the vendor to find the cause of the sudden failure. Another radio link was put up as a replacement, but latency and throughput problems continue.

There is no current ETR for getting all circuits back up to full capacity, but Broadlink’s operations department is working diligently to find a solution for this service interruption.

-Tony and Jason

Update: Mon May 22 10:04:38 PDT 2006 — Broadlink backhaul outage update. The backhaul path is being rerouted this morning. We expect to have the new link in place by 11am and customers back up by Noon. Thanks for your patience. -Matt and Jason

Brief service interruption.

Thu May 18 23:52:02 PDT 2006 — Brief service interruption. Some routine maintenance this evening inadvertently caused about 3 minutes of limited reachability to some hosts in our datacenter. Traffic is now flowing smoothly once again, and impact should have been minimal. -Jared and Nathan

Dial Up Service Issues.

Thu May 18 10:40:12 PDT 2006 — Dial Up Service Issues. One of our CT3’s of inbound dial up lines is currently out of service. We suspect a failure in one of the CAC WideBank muxes and have a tech en route to San Francisco now. We do not have an ETA at this time. -Kelsey, Nathan and Russ