Author: Michael Tardif

Berkeley Backbone Fiber Damage

Friday, 12/9/2022 at approximately 11:00am PST a power line fell onto a backbone fiber cable causing extensive burn damage to the entire cable serving a Local Convergence Point (LCP) for a neighborhood in the Berkeley area. Upon customer down-detection – in combination with calls to our support line – we dispatched repair crews to evaluate the extent of the damage. Unfortunately, the scope of the damage was extensive and required a full backbone cable re-splice to restore service. Crews worked for 27 hours to fully rebuild the cable to restore service to our customers. We believe all fibers to be fully restored. If you are continuing to experience a service issue, please contact support, methods here: https://www.sonic.com/support.

To modify your SMS number on file for directly messaged outage notifications, please visit https://members.sonic.net/ and navigate to ‘Account’, then to  ‘Member Information’.

Upstream Routing Reachability Instability

Sonic’s network just experienced a brief service disruption from certain upstream routes. We have identified the cause and have remedied. This was a transient routing issue and we do not expect further impact.

~Network Engineering

Fusion Fiber Service Disruption

Update: 11:37pm – The majority of all affected customers are now restored. We are manually restoring the ones that remain. The cause of this outage was a planned vendor maintenance event for power work that caused an equipment power redundancy failure on a core router despite being fully powered on backup feeds. The interruption in traffic while routes shifted caused certain customer routers to flood our DHCP servers beyond capacity. If your internet service is still down, reboot your router. If problems persist please reach out to customer support during business hours.

Update: 10:51pm – Services are restoring now across our platforms.

We are investigating service disruptions on our residential Fusion Fiber services. Issues include intermittent data and voice connectivity. We are working quickly to restore services. Updates to follow.

-Network Engineering

Fusion Fiber Maintenance – San Francisco

Update(4:17) Maintenance is now complete.

Tomorrow morning, 9/3/21 at 2:00am PDT, we will be performing intrusive maintenance on a small subset of customers in San Francisco. Data and voice services will be intermittent for the duration of the maintenance window while work is performed. Full restoral is expected for all users at the end of the window.

-Network Engineering

Fusion Fiber Maintenance – San Francisco, Berkeley Areas

Update: This work concluded at 3:30 A.M with all post-checks completing successfully.

Update: This work has been postponed to 7/29/21 @ 2 A.M.

Tomorrow morning (7/28/21) at 2 A.M we will be performing routine system maintenance on a small subset of Fusion Fiber customers in the San Francisco and Berkeley areas. Data and voice services will be interrupted briefly. The maintenance window is 3 hours.

~Network Engineering

Telephone Reachability Issues

Update: 6/24/21 3:15 PM – Multiple inbound carriers have made re-routes to work around the reachability issues we have been experiencing. Due to the nature of telephone network routing we do not have control over how calls make it to our network. We will continue to collect examples, escalate, and test with the other telephone companies involved  If anybody continues to experience dialing delays when calling Sonic customers please open a trouble case with your phone company. We are actively engaged with Verizon, Comcast, and Bandwidth to discover which telephone carrier in the call path is causing these troubles.

Update: 9:37 AM – We have made progress with inbound carriers and our testing has yielded an improved call completion rate. We are continuing to test and work with other inbound routes to isolate and resolve any lingering dialing issues.

We are currently investigating inbound call issues to Sonic customers. The problem does not seem to be systemic at this time, but with some calling parties to Sonic telephone numbers. Symptoms include Post Dial Delay (PDD) after dialing into a Sonic telephone number, and occasional error messages played back to the calling party. We have escalated the issue with the call paths that lead to us and are working to resolve.

~Network Engineering

Sonic Fiber Network Maintenance

Update: As of approximately 10:15am all services, both data and voice were restored. RFO to follow.

Update: 8:40am – For the remaining customers that are still without service, we are currently working with our hardware vendor’s engineering team to resolve this as quickly as possible.

Update 7:29am – A large portion of the remaining subscribers have had their services restored, however we are still pushing to bring back the remaining circuits. At this time the root cause of the problem is unknown, but we have a procedure to bring subscribers up incrementally that we will continue to exercise.

Update 6:29am – The service migration has failed to restore service for a number of customers. We are working to restore service as quickly as possible and will provide updates hourly.

Update 4:46am – We have isolated the cause of the issue and are working to resolve by migrating services to restore connectivity.

Update: Some customers in the Albany area are experiencing intermittent connectivity. We are investigating the root cause and will post updates as they become available. The remaining customers in other regions are back online.

Tuesday, 04/27/21, starting at 2:00 AM, we will be performing intrusive maintenance affecting residential and small business Fusion Fiber service in the east bay and Burlingame regions. Expected telephone and internet access downtime within the maintenance window is approximately 30 min as we perform system upgrades on the affected central office hardware. The maintenance window is 2 hours.

San Carlos & San Bruno – Intrusive Maintenance for Fusion Fiber Services

Update 4/26/21 3:49AM: This maintenance is now complete.

Monday, 04/26/21, starting at 2:00 AM, we will be performing intrusive maintenance affecting residential and small business Fusion Fiber service. Expected telephone and internet access downtime within the maintenance window is approximately 30 min as we perform system upgrades on the affected central office hardware. The maintenance window is 2 hours.

-Network Engineering

San Carlos – Intrusive Maintenance for Fusion Fiber Services

Update: This op is being postponed.

 

Wednesday, 04/14/21, starting at 2:00 AM, we will be performing intrusive maintenance affecting residential and small business Fusion Fiber service. Expected telephone and internet access downtime within the maintenance window is approximately 30 min. The maintenance window is 2 hours.

-Network Engineering

Brentwood & Petaluma Area Fusion Fiber Intrusive Maintenance

Update: As of 2:25 AM all circuits have recovered from the interruption of service. All post-maintenance monitoring has shown circuits as stable. This maintenance is complete.

03/10/2021 starting at 2:00 AM PST, Fusion Fiber customers in the Petaluma and Brentwood areas will experience a brief service interruption while we perform some maintenance. The expected down time is approximately 30 minutes for internet access only. Telephone services will not be affected unless using 3rd party VoIP on the affected data connection. The maintenance window will be 3 hours.