Month: May 2006

Sat May 20 18:35:00 PDT 2006 — Broadlink backhaul outage. Broadlink reports that on May 19th at about 2PM, the Broadlink backhaul went down without warning. After many hours of troubleshooting, they were able to remove the faulty radio equipment and are in contact the vendor to find the cause of the sudden failure. Another radio link was put up as a replacement, but latency and throughput problems continue.

There is no current ETR for getting all circuits back up to full capacity, but Broadlink’s operations department is working diligently to find a solution for this service interruption.

-Tony and Jason

Update: Mon May 22 10:04:38 PDT 2006 — Broadlink backhaul outage update. The backhaul path is being rerouted this morning. We expect to have the new link in place by 11am and customers back up by Noon. Thanks for your patience. -Matt and Jason

Thu May 18 23:52:02 PDT 2006 — Brief service interruption. Some routine maintenance this evening inadvertently caused about 3 minutes of limited reachability to some hosts in our datacenter. Traffic is now flowing smoothly once again, and impact should have been minimal. -Jared and Nathan

Thu May 18 10:40:12 PDT 2006 — Dial Up Service Issues. One of our CT3’s of inbound dial up lines is currently out of service. We suspect a failure in one of the CAC WideBank muxes and have a tech en route to San Francisco now. We do not have an ETA at this time. -Kelsey, Nathan and Russ

Tue May 16 18:05:38 PDT 2006 — Shell server offline. Our shell server, shell.sonic.net was offline briefly today due to a power cord failure. Total downtime was less than 5 minutes -Jared

Fri May 12 08:37:42 PDT 2006 — Brief Santa Rosa DSL router outage. Our DSL router in Santa Rosa stopped forwarding traffic at approx. 7:40AM. Service was restored at approx. 7:55AM. We are investigating the cause of the failure. – Clay and Network Operations

Sun May 7 13:02:25 PDT 2006 — Brief DoS Attack. There was a brief DoS attack directed at a customer on one of the DSL terminators in San Francisco. The attack appears to be over. -Kelsey and Nathan

Thu May 4 13:53:19 PDT 2006 — Santa Rosa DSL Outage. At approximately 1:05pm today, DSL service in the Santa Rosa area ceased to function. The outage lasted roughly 45 minutes, manifesting as an ability to sync up, without the ability to surf. Service should be restored at this time, and we are checking individual customer circuits to confirm. — John F and John K

Update Thu May 4 17:47:26 PDT 2006 — ASI confirms that this outage was caused by a failure at their Santa Rosa central office, they are taking steps to prevent this from recurring. — John F

Wed May 3 07:30:23 PDT 2006 — We are currently experiencing packet corruption on DSL circuits terminated in San Francisco. Customers that are not able to surf to sites other than Sonic.net are likely affected. Operations are aware of, and working to resolve the issue. -Kavan and Nathan

Update Wed May 3 07:39:57 PDT 2006 — We have isolated the issue to a failed Juniper ATM interface in one of our routers at 200 Paul Ave. The interface was passing small-sized packets without incident but began corrupting frames beyond a total size of 428 bytes. This caused TCP sessions to initiate properly but fail when large packets were transferred. Once the problem was isolated, we swapped traffic to a secondary, redundant path which fixed the issue. We will be scheduling replacement of the failed card soon to bring us back to full redundancy. -Nathan