Wed Mar 26 08:33:47 PST 2003 — The T3 backhaul to our SF pop is down, this has effectively isolated this pop from the rest of our network. Customers trying to dial-up to SF numbers will not be able to authenticate, and customers currently online will find that they are unable to contact our DNS servers. Our UUNet transit is also picked up at the SF pop so at this time all transit traffic is being handled by our other transit providers. We’ll have an ETA shortly. -Kelsey and Nathan.

We have been informed by Focal that there has been a fiber cut and SBC has told is there has been ‘major’ system failure at their equipment at 650 Townsend. We know that SBC techs are probably already on site working to resolve the problem, whichever is the case. Meanwhile, this outage has also affected all of our voice lines as well; tech support and all of our other voice lines are not available. -Kelsey, Nathan, Jared, Matt and John.

Sun Mar 23 12:03:18 PST 2003 — Intermittent busies on 9811 numbers. A bad PRI card in our dial-up gear has caused anyone dialing a number ending in 9811 to get occasional busy signals. We are working with our provider to disable the PRI until we can replace the card. -Matt, Kavan and John F.

Tue Mar 18 10:36:17 PST 2003 — Etherswitch failure. Currently web and mail services are unreachable due to an etherswitch failure. We are in the process of replacing it. –Kelsey, Nathan and John

The etherswitch has been replaced and the mail system is slowly recovering. It will probably take a couple of hours before things are back to normal, due to dealing with the backlog. –Kelsey, Nathan and John (and Zeke!)

Tue Mar 18 13:28:43 PST 2003 — Mail server load problems. The failure of the switch this morning triggered a cyclical failure of our already taxed mail cluster. We’ve been busy all morning doing what we can to assure smooth mail delivery and have, at this point, restored most services. Customers may experience some delays in delivery of email, however no email will or has been lost. Customers may also still encounter some difficulties sending or retrieving mail while we continue to work on the servers to improve the situation.

In order to increase NFS server performance we’ve added three more disks to the Netapp volume that handles mail. These additional spindles have already helped reduce mail server load. We’ve also added a secondary MX server to handle mail for sonic.net when our cluster is off line; this keeps the mail under our control and ensures more rapid delivery of delayed mail. -Kelsey, Nathan, Zeke, et al.

Fri Mar 14 12:07:03 PST 2003 — Graymail updates: We deployed a rewritten graymail notification and expiration processor a on Wednesday night. The new processor resolves the performance problems that the old one was having, speeding up a single run from a 10+ hour job to just 78 minutes.

Currently, over 11,000 of our members are enjoying the pleasures of a SPAM free inbox. We strongly urge anyone who hasn’t enabled SpamAssassin and Graymail on their account already, to do so now at sonic.sonic.net/membertools/spamcan/. -Kelsey and Nathan

Tue Mar 11 06:17:22 PST 2003 — The scheduled maintenance to move former Inreach/PacBell DSL customers in the Stockton area to Sonic’s network was completed successfully this morning. Welcome to Sonic! -Nathan, John and Andy from Inreach

Fri Mar 7 21:07:29 PST 2003 — Connectivity with one of our upstream providers is experiencing problems. Traffic is being rerouted while we determine and fix the problem. Customers should not notice any connectivity issues as our redundant provider is picking up the slack. -Matt and Nathan

Wed Mar 5 18:01:18 PST 2003 — We are postponing the scheduled Thursday maintenance. It is now tentatively scheduled for Tuesday, 3/11, between 5 and 7 AM. -John and Nathan

Tue Mar 4 10:05:38 PST 2003 — Former Inreach DSL customers in the Stockton (LATA 9) area will be moved to the Sonic network during a maintenance window Thursday, 3/6/03, between 5 and 7 AM. The actual outage should be much shorter than that. -John and Nathan

Sun Mar 2 09:34:46 PST 2003 — Border router failure. A VIP in one of our core Cisco 7500 routers failed this morning causing routing instability until it was removed. This VIP was been used to provide redundancy in our core and is not needed for normal operation. We will be investigating why the the failure of one of the router’s redundant interfaces interfered with it’s operation; it should have gone relatively unnoticed. While the router was having difficulty, reachability to or from our network may have been spotty and some of our prefixes were dampened by remote networks. At this time, all of the router’s services have been fully restored and our network has returned to normal. -Kelsey and Nathan