Today, August 7th 2024, starting at 5:30pm, a router in one of our transit provider networks began dropping traffic destined for Sonic customers. These traffic drops manifested in connectivity issues from many internet destinations, as well as voice issues for our Southern California customers. We took steps to mitigate the affected upstream routing problem and restore voice as quickly as possible. Our transit provider resolved their issue no later than 7pm and all traffic has been restored.
Author: Tomoc Field
Nor-Cal Core Network Maintenance
Tuesday, April 27th, starting at 11:59pm PST, we will be performing updates to transport equipment serving our core network. These upgrades are expected to be non-intrusive. This maintenance window ends at 4am.
-Network Engineering
Core Network Maintenance
Update(12:25AM): Maintenance complete.
Monday, January 11th, starting at 11:59PM, we will be performing non-intrusive maintenance on equipment servicing the core of our network. This maintenance window is 4 hours.
-Network Engineering
San Bruno/Burlingame Fusion Fiber Maintenance
Update(12:41am): Maintenance complete
Tuesday, October 20, starting at 11:59pm, we will be performing maintenance on Fusion Fiber circuits in the Burlingame and San Bruno areas. Some customers may experience up to 10 minutes of service interruption. Maintenance window is 4 hours.
-Network Engineering
Burlingame Area Fusion Fiber Maintenance
Update(1:15am): Maintenance complete
Thursday, October 8th, starting at 11:59pm, we will be performing intrusive maintenance on Fusion Fiber customers in the greater Burlingame area. Expected down time is 15 minutes. Maintenance window is 4 hours.
-Network Engineering
Legacy DSL Emergency Maintenance
Update(12:42am): Maintenance complete
Tonight, September 10th, starting at 11:59pm, we will be performing an emergency migration of a small subset of Legacy DSL subscribers to different aggregation equipment. This migration is to mitigate a small amount of packet loss to the affected equipment. Customer down time is expected to be less than 10 minutes.
-Network Engineering
LATA 1 Legacy DSL Maintenance
Update(12:39am): Maintenance complete. All services should be restored, so if any service issues remain, please contact support in the morning for individual assistance.
Tonight, September 3rd, starting at 11:59pm, we will be performing maintenance on a small subset of Legacy DSL subscribers in Northern California. Expected service interruption for the affected users is 10 minutes.
-Network Engineering
Northern California Legacy DSL Outage
Update(9:30AM): All suspect hardware has been replaced. We believe affected customers should now be operational and will continue to monitor the situation.
Update(7:15AM): We are still working towards resolution with the remaining customers that are down
Update(3:03AM): We were able to restore most subscribers via equipment reboot, however a subset of circuits cannot be restored without hardware replacement. We are dispatching with replacement hardware with an ETA of 5AM.
At approximately 10:40PM, equipment serving a subset of legacy DSL subscribers in Northern California had a hardware failure. We are working to restore traffic as quickly as possible but anticipate resolution will require an emergency maintenance reload of the device. Updates to follow.
– Network Engineering
Partial Network Outage
Update(11:08am): It appears the affected networks were much more isolated than we initially believed. Affected network segments include a portion of our 2260 Apollo Way data center, as well as a number of internal monitoring and management networks. Another update will be provided tomorrow during business hours.
Tonight, August 4, starting around 9:49PM, a core router rebooted, causing wide-spread intermittent connectivity issues. This device was in a redundant configuration, will will receive some scrutiny as to why graceful failover did not occur.
We believe service has been mostly, if not completely restored and will provide updates as we have them.
-Netwokr Engineering
Southern California Edge Maintenance
Update(3:00am): Maintenance complete, no customer impact
Tuesday, July 21st, starting at 11:59pm, we will be making changes to our transit configuration serving our Southern California network. Customers may experience up to 10 minutes of routing instability as BGP across the internet re-converges. This maintenance window is 4 hours.
-Network Engineering