Tonight, Thursday, January 26 at 11:01 PM, we will be replacing the second core router in our datacenter. The switchover should only take a few minutes, and traffic will be routed around the affected router. If there is any disruption, it should not last more than 5-10 minutes at worst.
-Jared
Update: The router replacement was completed without incident. There should have been no impact to customer traffic.
Tonight, Tuesday January 24 at 12:01 AM, we will be replacing the core router in our datacenter which failed earlier this morning. The switchover should only take a few minutes, and traffic will be routed around the affected router. If there is any disruption, as routing updates, it should not last more than 5-10 minutes at worst.
-Jared
Update: The router replacement has been completed without incident. There should have been no impact to customer traffic. In a few days we will schedule a replacement of the other in the pair of redundant routers in our Santa Rosa datacenter.
This Thursday, December 29 at 12:01 AM we will be performing core configuration changes on the routers at the head end of our Legacy DSL aggregation network. We will be taking all possible precautions to prevent any outages, but when doing this type of deep configuration change, the possibility of a problem is always present. The reconfiguration should take between 30 and 60 minutes, and we will be able to roll back quickly in the event of an issue.
-Jared and Nathan
Update: The reconfiguration has been completed. There were a couple brief, unexpected hiccups which disrupted connectivity to a small subset of our legacy DSL customers. We apologize for any inconvenience this may have caused.
Tonight, Friday November 4 at 12:01 AM, we will be reconfiguring approximately 50 Flexlink Ethernet customers to smooth the way for future network upgrades. This work will disrupt connectivity to those customers for less than 5 minutes apiece. No work or changes will be necessary for these customers to restore service.
-Jared, Tomoc, Robbie, and Phil
Update: All customer reconfigurations are complete. No customers should have been offline for more than 5 minutes. If you are experiencing any problems with your Flexlink Ethernet service, please call the NOC Hotline at 877-706-5662.
We were notified this morning of issues dialing numbers in Northern California on our Fusion Voice service. The issue was traced to one of our upstream phone providers in this area. Calls are now routed out redundant capacity and the issue is resolved. There was no impact to 911 service.
-Nathan
Today, Saturday September 24, at approximately 10:55 AM, we will be performing an emergency reload of one of our ATM switches. The ATM switch appears to have suffered an error which is affecting traffic flow through the box, and the only possible resolution is a full reload of the switch. All customers served by this switch will experience 5-10 minutes of downtime.
-Jared and Nathan
Update: The ATM switch has been reloaded and all affected customers should be back online at this time. If you are still experiencing problems with your legacy DSL connection, please contact our support team.
We are in the process of diagnosing a Legacy DSL outage that appears to be affecting much of Northern California. Â We are actively working with our vendor to resolve this outage as soon as possible. Â At this time there is no estimated time of repair.
Update: Our vendor has diagnosed the cause of this outage as a defective ATM card within their network. Â The card has been replaced and service has been restored to all affected users.
-Chris
This morning at 12:24 PM, a large DDoS was aimed at a server in our colocation facility in Santa Rosa. The DoS was large enough to disrupt connectivity to our Santa Rosa datacenter, which would have affected access to mail and web services hosted by Sonic. The DoS was blocked by 12:32 PM, and no further ill effects should have been felt after that point. -Nathan
This Tuesday, June 28 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.
-Jared
We are currently investigating a problem within our telecom network. Customers on it may experience intermittent reachability to other hosts both in and outside of our network. We will provide an update shortly once we have further diagnosed the issue.
-Matt and the NOC
Update: Service should now be stabilized for any effected customers. We have determined there was a fiber cut in Palo Alto, CA that impacted a number of our backbone links. That in turn caused reachability issues while our network rerouted around the down circuits. Furthermore, a router bug prevented a number of prefixes from properly rerouting, thus causing additional impact to a limited number of Flexlink Ethernet and Flexlink Long Range customers in the North Bay. We are working with the hardware vendor to diagnose and resolve this issue. All in all, this was a very successful test of our redundancy scheme, which kept packets flowing despite a group outage impacting a significant number of circuits.