Author: jack.wanke@sonic.com

Outage in Oakland

Update (Tue Jun 25 04:30:00 PM PDT 2024): Emergency maintenance on the affected equipment in the Oakland area has been completed successfully. Stability looks to have returned to nominal levels. While we are confident in the resolution, we will continue to monitor for any abnormal behavior to ensure long-standing stability on our network. If you are actively experiencing issues with your connection, we advise you to call into our technical support team so that we may investigate further. Thank you.

Update (Tue Jun 25 02:10:00 PM PDT 2024): We’re doing emergency maintenance on the affected equipment within the Oakland area in order to resolve the ongoing outage at 3:00PM. This will result in roughly 15 to 30 minutes of downtime for our services once the maintenance begins. The maintenance window is 2 hours.

Update (Tue Jun 25 11:00:00 AM PDT 2024): Our Network Engineers are continuing to work with our vendor in order to solve the issue at hand. While no fix has been implemented yet, efforts are ongoing. More updates to follow.

Update (Tue Jun 25 08:30:00 AM PDT 2024): We are now working in tandem with our equipment vendors to further diagnose this issue. Once successful, a final fix will then be implemented to restore stability indefinitely. There is still no estimated time of restoral, but updates will continue to be provided.

Update (Mon Jun 24 08:50:00 PM PDT 2024): Unfortunately, the fix we implemented to our affected equipment earlier this afternoon did not solve the issue in its entirety. Intermittent instability with services is still being observed. We are aware of the issue and are actively assessing this further in an attempt to mitigate the problem in its entirety. Due to the complexity and extent of the issue, we do not have an estimated time of restoral. We will continue to provide updates as more information becomes available.

Update (Mon Jun 24 01:05:00 PM PDT 2024): The issue affecting our equipment in the Oakland area is resolved. If you still have problems with your connection, we recommend calling into our technical support team.

Update (Mon Jun 24 12:23:03 PM PDT 2024): We are still investigating the issue in the Oakland area. Changes made to the equipment on site appear to have restored service for most of our customers. More actions are being taken to ensure stability.

Update (Mon Jun 24 11:01:33 AM PDT 2024): We’ve observed issues on the affected equipment in the Oakland area after the initial restoral of connectivity. We are continuing to make changes to the equipment in order to stabilize the situation.

Update (Mon Jun 24 10:29:01 AM PDT 2024): Services are being restored on the affected equipment. The services are still being closely monitored to ensure stability.

Update (Mon Jun 24 10:03:37 AM PDT 2024): We are continuing to investigate the issue. We have dispatched a technician to the Oakland data center in order to perform troubleshooting on site.

We’re currently investigating an issue with our network in the Oakland area that has resulted in customers being unable to reach the internet. We will update this post with more information as the investigation continues.