As of approximately 10:40am, a small subset of copper based customers in San Francisco may be currently experiencing intermittent connectivity issues. We are working to identify the cause of this problem and repair it. More information to follow.
Update(1:02pm): We have narrowed this issue down to a single device in our network and are no longer experiencing any instability. We are continuing to investigate the root cause.
-Sonic NOC
Update: 4:15pm – Today at 12:28pm Sonic experienced a network instability event of an unknown origin at the time. Â Through troubleshooting we were able to narrow down the issue to two transport links going from one of our datacenter locations to core network equipment. Â These links were receiving replicated traffic from the transport providers network equipment, sent back at us. The replicated traffic overloaded the CPU on our core routers. We have taken steps to prevent the replicated traffic from affecting our network, and we have contacted our provider for further diagnosis. Apologies for the delay in getting issue resolved, it was a very difficult problem to troubleshoot, and we have never seen anything like this happen before.
Update 3:42pm – We believe everything is restored. Â We will release a RFO soon with more specific information.
Update 2:46pm – We are still working to mitigate the DoS. All of our engineering staff is currently engaged in this issue. We will post more details as they become available.
As of 12:30pm, the Sonic network is experiencing reachability issues to the outside world. A large DoS attack is the suspected cause but we are still working to identify and mitigate the problem ASAP. More information will follow.
-Sonic NOC
Update: 1/23/2017 – 4:05pm – A bug has been identified in the vendors network that is related to the CPE that is put on site and the router in the vendors core network. Â Our vendor has been issuing upgrades to the CPE which has the potential expose this bug on the core router, causing traffic to loop. Â When this occurs, the traffic is perceived as BUM traffic and is policed causing speeds to be capped at 2 Mbps. Â We are working with our vendor to create solutions to help identify and act quickly when this bug happens.
Update: 1/23/2017 – 9:58am – We are continuing to actively work with our vendor’s advanced Engineering and Management teams to resolve the remaining affected customers. Please report outstanding SSE service impacting issues to NOC for us to investigate.
Update: 1/22/2017 – 3:30pm We are receiving some more reports this afternoon regarding this issue being unresolved for a portion of our customers and have escalated with our vendor.
Update: 1/21/2017 – 8:30pm We believe this issue has been located and fixed at this point. Please contact the NOC if you are experiencing any further trouble.
Update: 1/21/2017 – 5:40pm We are continuing to escalate and work with our vendor on this issue.
Update: 1/21/2017 – 11:22am We are still working with high level escalation with our vendor. Â Currently they are tracing aggressively through their network in an attempt to locate any issues or anomalies.
Update: 1/21/2017 – 6:46am We are working direct with upper management and 2 different teams. Â We will update soon when they get some more data to us.
Update: 10:38pm – Escalation has moved to a higher level department and our issue is currently being worked. Â We are awaiting the results of their testing. Â No ETR.
Update: 8:11pm – We are still on working with our vendors and have escalated the issue again. There is currently no ETR.
Update: 4:45pm – Our issue is currently being worked by our vendors Engineering staff, they suspect an issue in the core of their network. Â No ETR yet.
Update: 2:09pm – We are still working with our vendor and have assisted in narrowing down the issue. There is currently no ETR.
Update: 10:39am – We are working with our fiber vendor to isolate and resolve this issue. There is currently no ETR. As soon as we have one or know more, this post will be updated.
~ Network Operations
We are investigating reported packet loss and high latency spikes on our enterprise SSE fiber circuits. We will update this once we know more.
~ Network Operations
Update (1:38AM): Maintenance Complete
Beginning tonight at midnight, we will be performing upgrades on equipment serving a small subset of Fusion and Flexlink customers in San Francisco. Expected downtime for this operation can be up to 2 hours, but is typically less than 30 minutes for most customers. This maintenance will not affect any Fusion Fiber customers.
-Network Engineering
Update (12:42AM): Maitenance complete.
Beginning tonight at midnight, we will be performing upgrades on equipment serving a subset of Fusion and Flexlink customers in San Francisco. Expected downtime for this operation can be up to 2 hours, but is typically less than 30 minutes for most customers. This maintenance will not affect any Fusion Fiber customers.
-Network Engineering
Tonight, starting at midnight, we will be performing maintenance on equipment serving our Northern California Telecom network. Expected downtime is less than 10 minutes.
Update(2:31am): Maintenance complete
-Tomoc and Brandon
Update (1:56AM): Maintenance complete
Beginning tonight at midnight, we will be performing upgrades on equipment serving Fusion and Flexlink customers in Petaluma. Expected downtime for this operation can be up to 2 hours, but is typically less than 30 minutes for most customers.
-Network Engineering
Tonight, starting at midnight, we will be performing maintenance on equipment serving our Northern California Telecom network. No downtime is expected, however some customers may experience brief periods of routing instability.
Update(11:11pm): Maintenance postponed until tomorrow night at midnight
Update(12/1 4:51am): Maintenance is now complete. An equipment failure during this maintenance caused a disruption of service for customers with Fiber SSE services.
-Tomoc
Update(1:50AM): Maintenance complete.
Beginning tonight, at midnight, we will be performing maintenance on equipment that provides transport for all Fusion/FlexLink and legacy DSL services in the Sacramento area. We do not expect any downtime from this operation though users may notice some routing changes.
Update (1:13AM): Maintenance complete
Beginning tonight at midnight, we will be performing upgrades on equipment serving Fusion and Flexlink customers in Ukiah. Expected downtime for this operation can be up to 2 hours, but is typically less than 30 minutes for most customers.
-Network Engineering