Northern California Private Ethernet Transport Instability

Earlier this morning Sonic Engineers discovered some instability in the network that’s affecting Private Ethernet Transport services. At this point we’ve isolated the issue to traffic passing through some core network equipment in Santa Rosa. Network Engineers are actively investigating the issue. There is no ETR at this time.

Update: (Wed Jun 7 12:14 PM PDT 2023) Sonic is aware of some routing issues within the Santa Rosa area. Engineers are currently performing emergency maintenance on equipment in Santa Rosa in order to stabilize the network.

Update: (Wed Jun 7 12:52 PM PDT 2023) Emergency maintenance is still ongoing. This may impact some services including internet access, voice, and private ethernet transports as routing shifts across the network.

Update: (Wed Jun 7 02:36 PM PDT 2023) Sonic Engineers have identified and addressed the root cause of the original instability, however the team is still working towards restoring services that were affected during the emergency maintenance.

Update: (Wed Jun 7 04:21 PM PDT 2023) Sonic Engineers have been monitoring network activity and the situation has stabilized after the emergency maintenance performed earlier this afternoon.

Northern California Fiber ONT Upgrades

Update: 1:50AM: This maintenance is now complete.

Tonight at 11:59pm, we will be pushing updates to another subset of Northern California Sonic fiber customers served by AdTran 622V ONTs. Expected service interruption will be 5-10 minutes as the ONTs reboot to load their new firmware. The maintenance window is until 4AM. An update to this MOTD will be provided when the work is completed.

-Network Engineering

Northern California Fiber ONT Upgrades

Update: 2:08AM: This maintenance is now complete.

Tonight at 11:59pm, we will be pushing updates to a subset of Northern California Sonic fiber customers served by AdTran 622V ONTs. Expected service interruption will be 5-10 minutes as the ONTs reboot to load their new firmware. The maintenance window is until 4AM. An update to this MOTD will be provided when the work is completed.

-Network Engineering

Southern California Core Maintenance

Friday, 6/9/23, starting at 12:01 AM, we will be performing maintenance on core routing equipment in our southern California network.

This work is not expected to impact any customer connectivity. The maintenance window is open until 4am.

-Network Engineering

Cloverdale, Healdsburg, Napa, Occidental, Santa Rosa – Fusion DSL and Flexlink Ethernet Intrusive Maintenance

Update: 3:37AM: This maintenance is now complete.

Tuesday(6/6/23) morning, starting at 2:00AM, a small subset of Fusion DSL and Flexlink Ethernet customers in the Cloverdale, Healdsburg, Napa, Occidental and Santa Rosa area’s will experience a brief service interruption while we perform some maintenance. The expected down time is 1 hour. The maintenance window will be 3 hours.

– Network Engineering

Northern California Core Maintenance

We will be performing nightly maintenance on core routing equipment in our northern California network next week. The work happens on Tuesday 6/6/23 through Friday 6/9/23 night starting at 12:01AM. This work is not expected to impact any customer connectivity. The maintenance window ends at 4:00AM.

-Network Engineering

Albany, Berkeley, Brentwood – Core Router Maintenance

Update: 3:06AM: This maintenance is now complete.

Friday, June 2nd, starting at 12:01AM, we will be performing maintenance on core equipment serving the Albany, Berkeley and Brentwood areas. No customer impact is expected. This maintenance window is 4 hours.

-Network Engineering

Northern California Fiber ONT Upgrades

Update: 2:32AM: This maintenance is now complete.

Tonight at 11:59pm, we will be pushing updates to a subset of Northern California Sonic fiber customers served by AdTran 622V ONTs. Expected service interruption will be 5-10 minutes as the ONTs reboot to load their new firmware.  The maintenance window is until 4AM.  An update to this MOTD will be provided when the work is completed.

-Network Engineering

Connectivity issue in Santa Rosa

 We are investigating a connectivity issue that affected services in Santa Rosa and related voice services. Services look to have been restored but we are still looking into the cause currently.
Update(6:30AM): Services have been confirmed restored after rebooting a core router in Santa Rosa. The affected device began generating suspicious log messages right at the time connectivity issues began. We will be continuing root cause investigation in order to prevent this type of failure in the future.
~Network Engineering