This morning at approximately 5:01 AM, one of our ATM switches suffered an interface card lockup. We are about to initiate an emergency reboot of the ATM switch chassis to clear the problem. The ATM switch reboot will disrupt DSL, Business-T, and FRATM connectivity to some customers in the Bay Area, as well as the Stockton, Chico, and San Diego areas.
-Jared
Update: The ATM switch reboot was completed at 5:33 AM. All service through that ATM switch is now restored.
Starting at about 2:24 PM today, a distributed denial of service attack was launched at one of our DSL subscribers. This DDoS was large enough to disrupt service to the Redback SMS DSL aggregation router that served this customer and several hundred others. We have blocked the DDoS attack at our edge, thus restoring service to the affected customers.
-Jared
Starting at about 1:33 PM today, a distributed denial of service attack was launched at one of our DSL subscribers. This DDoS was large enough to disrupt service to the Redback SMS DSL aggregation router that served this customer and several hundred others. We blocked the DDoS attack at our edge within 5 minutes, thus restoring service to the affected customers.
-Jared
ATT has confirmed that there is a DSL outage in San Francisco. This is not a complete outage, but rather intermittent performance problems. Thank you for your patience while they work to resolve the issue.
Our offices will be closed from 12pm to 5pm today for our 15th Annual Holiday Party. Should you need assistance, please leave a voice mail or email support@sonic.net. As always, Network and Systems Operations are available on their respective hot lines. -Sonic.net
We apologize for the delay in posting this notification, but our systems operations staff are very busy trying to fix the problem. Thank you for your patience, web services should be restored soon!
Update: Starting at 7:30AM this morning a DDOS directed at one of our hosted websites started to destabilize the cluster causing intermittent failures. Our initial response was misdirected as we thought it to be a known failure scenario with our hardware load balancers. We rebooted the load balancers and all services came back up and we believed that the issue had been resolved. It wasn’t until several subsequent failures occurred that the root cause of the problem was correctly identified and fixed. Services were fully restored shortly after 9:00AM. -Kelsey, William, Joey and Nathan
Today at 9:58AM one of the routers that serves T1 customers in the Sonoma County area unexpectedly reloaded. Service has been restored at this time and we are investigating the cause of the reload. Approximate downtime was 5 minutes.
-Tim and Nathan
Tonight at approximately 1:20 AM, one of our DSL aggregation routers that serves the greater Bay Area had it’s network uplink interface lock up. This problem was resolved by reloading the affected router. At this time, the router is back up and fully operational, and all affected customers should be back online. Total downtime was approximately 15 minutes for affected customers
-Jared
Wednesday (09, Dec.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.
-Augie, Aaron
We are currently experiencing an outage affecting our Los Angeles area aggregation equipment. This is affecting all DSL and ATM-based Business-class services. We are working on restoring service and will provide an update shortly.
-Jared and Tim
Update: The outage was caused by a crashed management card in our core ATM switch in Los Angeles. The ATM switch does have a redundant management card, but that card did not load its configuration properly. The ATM switch had to be completely power failed to restore it to service. At this time the switch is up and functioning properly, and all connectivity should be restored. We have also isolated and fixed what caused the backup management card to not load its configuration properly.