Tue May 16 18:05:38 PDT 2006 — Shell server offline. Our shell server, shell.sonic.net was offline briefly today due to a power cord failure. Total downtime was less than 5 minutes -Jared

Fri May 12 08:37:42 PDT 2006 — Brief Santa Rosa DSL router outage. Our DSL router in Santa Rosa stopped forwarding traffic at approx. 7:40AM. Service was restored at approx. 7:55AM. We are investigating the cause of the failure. – Clay and Network Operations

Sun May 7 13:02:25 PDT 2006 — Brief DoS Attack. There was a brief DoS attack directed at a customer on one of the DSL terminators in San Francisco. The attack appears to be over. -Kelsey and Nathan

Thu May 4 13:53:19 PDT 2006 — Santa Rosa DSL Outage. At approximately 1:05pm today, DSL service in the Santa Rosa area ceased to function. The outage lasted roughly 45 minutes, manifesting as an ability to sync up, without the ability to surf. Service should be restored at this time, and we are checking individual customer circuits to confirm. — John F and John K

Update Thu May 4 17:47:26 PDT 2006 — ASI confirms that this outage was caused by a failure at their Santa Rosa central office, they are taking steps to prevent this from recurring. — John F

Wed May 3 07:30:23 PDT 2006 — We are currently experiencing packet corruption on DSL circuits terminated in San Francisco. Customers that are not able to surf to sites other than Sonic.net are likely affected. Operations are aware of, and working to resolve the issue. -Kavan and Nathan

Update Wed May 3 07:39:57 PDT 2006 — We have isolated the issue to a failed Juniper ATM interface in one of our routers at 200 Paul Ave. The interface was passing small-sized packets without incident but began corrupting frames beyond a total size of 428 bytes. This caused TCP sessions to initiate properly but fail when large packets were transferred. Once the problem was isolated, we swapped traffic to a secondary, redundant path which fixed the issue. We will be scheduling replacement of the failed card soon to bring us back to full redundancy. -Nathan

Fri Apr 28 10:57:56 PDT 2006 — Tonight, shortly after Midnight, we are going to perform maintenance and expansion on the NetApp filers that handle mail spools and web sites. Every effort will be taken to minimize downtime of these critical services however each filer will have to be rebooted at least once to perform OS upgrades and add additional storage capacity. -Kelsey and Clay

Update: Sat Apr 29 02:27:04 PDT 2006 — The upgrades went smoothly as planned. Customer impact remained minimal. -Kelsey and Augie.

Fri Apr 28 08:12:46 PDT 2006 — Shell server changes. We’ve added a program to Bolt.sonic.net, our shell server. (If you don’t know what a shell server is, you can safely ignore this notice.)

Yes, it is the return of “pkiller.pl”, the process monitor. However, it is much tamer now — for instance, the “screen” program is exempt from being killed by the monitor, as well as any program with a controlling tty…

Our goal is to prevent crufty, old, idle processes from building up on the shell server, freeing those resources for folks who are actively using the shell server.

Discussion about the shell server itself is certainly welcome in the sonic.net newsgroup, available here: news://news.sonic.net/sonic.net -Scott

Mon Apr 10 22:54:46 PDT 2006 — DSL Router Maintenance. We will be performing hardware maintenance on our DSL router in Santa Rosa at approx. midnight tonight. Expected downtime is 10-15 minutes for any customers terminated on this router. -Nathan, Kelsey, and Jared

Update Tue Apr 11 00:44:35 PDT 2006: Maintenance is complete. Total downtime was approximately 13 minutes. Earlier today we diagnosed an obscure hardware failure in Santa Rosa’s DSL termination gear that lead to the lockup and subsequent reboot around 8pm. Tonight’s hardware maintenance was a complete swap of the entire unit with our on-site spare. We’ll keep a close eye on its performance over the next few days. -Nathan, Kelsey and Jared

Mon Apr 10 20:02:45 PDT 2006 — DSL Router Lockup. Our DSL router in Santa Rosa stopped forwarding traffic at approx 7:50 PM. It has been rebooted and traffic is flowing normally again. We have this device scheduled for hardware replacement in the very near future. -Jared and John

Sun Apr 9 18:04:39 PDT 2006 — Router Reload. One of our customer-serving routers in Santa Rosa reloaded without warning at approx. 5:45 this afternoon. The router is back up and forwarding traffic at this time. Any T1 or WBA customers served by this router would have been offline for 5-10 minutes. We are investigating the cause behind this unexpected reload. -Zeke and Jared