Tonight, February 5 at approximately 12:25 AM, one of the ATM switches that serves legacy DSL to the greater bay and Chico areas suffered a card software failure. The switch was rebooted to clear the problem, and all affected customers should be back online. We will be investigating this issue with the equipment vendor.
-Jared
We will be performing immediate emergency maintenance on equipment serving a number of legacy DSL customers in the LATA1 area. This will impact affected customers for 5 to 10 minutes while the maintenance is performed.
-Tim and Jared
Update: The source of the problem was a bad port on a single DSL aggregation router in our ATM network. We have moved the affected connection to a new port and re-routed all traffic to the new port. Affected customers should be back online at this time.
This Thursday, starting at 9AM, one of the three UPSes that serves our Santa Rosa datacenter will have all of it’s batteries replaced. This is not a service impacting event and the UPS will be online for the duration of work. If necessary, the battery replacement may continue Friday morning. -Kelsey and Russ
Tonight, Thursday, January 26 at 11:01 PM, we will be replacing the second core router in our datacenter. The switchover should only take a few minutes, and traffic will be routed around the affected router. If there is any disruption, it should not last more than 5-10 minutes at worst.
-Jared
Update: The router replacement was completed without incident. There should have been no impact to customer traffic.
Tonight, Tuesday January 24 at 12:01 AM, we will be replacing the core router in our datacenter which failed earlier this morning. The switchover should only take a few minutes, and traffic will be routed around the affected router. If there is any disruption, as routing updates, it should not last more than 5-10 minutes at worst.
-Jared
Update: The router replacement has been completed without incident. There should have been no impact to customer traffic. In a few days we will schedule a replacement of the other in the pair of redundant routers in our Santa Rosa datacenter.
At 5:32 AM today, one of our core routers in our Santa Rosa datacenter unexpectedly restarted its routing processes. This caused 5-10 minutes of impacted connectivity to our datacenter as the network recovered from the event. This would have affected access to mail, Sonic.net hosted websites, and colocated servers.
Ironically, we had already scheduled retirement and replacement of our core routers in the Santa Rosa datacenter for this upcoming week. We will accelerate our plans due to this event, and apologize for any impact it may have caused.
-Jared
Today we have updated our member tool authentication code. This is a transparent change in our back end code and does not require any end user changes. We believe this will resolves an intermittent bug that was causing users to occasionally have to re-authenticate their session and also results in a slight performance increase while using the tools.
-William
This Thursday, December 29 at 12:01 AM we will be performing core configuration changes on the routers at the head end of our Legacy DSL aggregation network. We will be taking all possible precautions to prevent any outages, but when doing this type of deep configuration change, the possibility of a problem is always present. The reconfiguration should take between 30 and 60 minutes, and we will be able to roll back quickly in the event of an issue.
-Jared and Nathan
Update: The reconfiguration has been completed. There were a couple brief, unexpected hiccups which disrupted connectivity to a small subset of our legacy DSL customers. We apologize for any inconvenience this may have caused.
This Wednesday, December 28 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.
-Jared
The server that handles ssl.sonic.net (our legacy shared https web hosting server) suffered a double disk raid failure (in a mirror) this afternoon and was offline for approximately 23 minutes while we restored to replacement disks from our backups. -Kelsey and William