Our Operations Team has found and corrected a problem which caused inbound emails to sonic.net (and other domains for which Sonic handles email) to take longer than expected to be delivered to their corresponding mailboxes.

Per our logging, inbound email delivery would have experienced delays between approximately 3:10pm and 6:20pm PST today. (July 25, 2024) Outbound email processing was not impacted. We are not aware of any lost messages due to these delays. We will investigate the root cause of this incident and will be adding monitoring to ensure we can swiftly respond in the future.

— Joe M
Sonic System Operations

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems. Any downtime to affected public services should be brief, and we expect to complete the maintenance within 2 hours.

An update will be posted after work is complete.

Sonic Support will be closing early at 5pm tomorrow 7/04. The Network Operations team will be open in a reduced capacity and focusing on urgent and critical issues. Billing and Inside Sales will be closed the entire day. All Sonic departments return at 8am on 7/05.

Have a happy holiday!

Update (Tue Jun 25 04:30:00 PM PDT 2024): Emergency maintenance on the affected equipment in the Oakland area has been completed successfully. Stability looks to have returned to nominal levels. While we are confident in the resolution, we will continue to monitor for any abnormal behavior to ensure long-standing stability on our network. If you are actively experiencing issues with your connection, we advise you to call into our technical support team so that we may investigate further. Thank you.

Update (Tue Jun 25 02:10:00 PM PDT 2024): We’re doing emergency maintenance on the affected equipment within the Oakland area in order to resolve the ongoing outage at 3:00PM. This will result in roughly 15 to 30 minutes of downtime for our services once the maintenance begins. The maintenance window is 2 hours.

Update (Tue Jun 25 11:00:00 AM PDT 2024): Our Network Engineers are continuing to work with our vendor in order to solve the issue at hand. While no fix has been implemented yet, efforts are ongoing. More updates to follow.

Update (Tue Jun 25 08:30:00 AM PDT 2024): We are now working in tandem with our equipment vendors to further diagnose this issue. Once successful, a final fix will then be implemented to restore stability indefinitely. There is still no estimated time of restoral, but updates will continue to be provided.

Update (Mon Jun 24 08:50:00 PM PDT 2024): Unfortunately, the fix we implemented to our affected equipment earlier this afternoon did not solve the issue in its entirety. Intermittent instability with services is still being observed. We are aware of the issue and are actively assessing this further in an attempt to mitigate the problem in its entirety. Due to the complexity and extent of the issue, we do not have an estimated time of restoral. We will continue to provide updates as more information becomes available.

Update (Mon Jun 24 01:05:00 PM PDT 2024): The issue affecting our equipment in the Oakland area is resolved. If you still have problems with your connection, we recommend calling into our technical support team.

Update (Mon Jun 24 12:23:03 PM PDT 2024): We are still investigating the issue in the Oakland area. Changes made to the equipment on site appear to have restored service for most of our customers. More actions are being taken to ensure stability.

Update (Mon Jun 24 11:01:33 AM PDT 2024): We’ve observed issues on the affected equipment in the Oakland area after the initial restoral of connectivity. We are continuing to make changes to the equipment in order to stabilize the situation.

Update (Mon Jun 24 10:29:01 AM PDT 2024): Services are being restored on the affected equipment. The services are still being closely monitored to ensure stability.

Update (Mon Jun 24 10:03:37 AM PDT 2024): We are continuing to investigate the issue. We have dispatched a technician to the Oakland data center in order to perform troubleshooting on site.

We’re currently investigating an issue with our network in the Oakland area that has resulted in customers being unable to reach the internet. We will update this post with more information as the investigation continues.

Starting yesterday afternoon after around 3pm, email that had an autoresponse configured started bouncing with errors about a missing SpamAssassin module. This was due to a mistake on our part that lead to the incorrect package automatically installing on a system it should not have. The problem was identified and erroneously considered fixed a few hours after it happened, but persisted until approximately 9:30am this morning when it was ultimately resolved.

We’re sorry for any problems this caused, and if you’re still experiencing any sort of problems related to this, please contact support@sonic.net or call support so we can help you resolve them.

-System Operations

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

Impacted systems include the shell server as well as various other systems. A follow-up notification will occur once all maintenance is complete.

Update 11:34pm: Maintenance compete

Update: 6:26am – Power has been restored and our equipment is back up.

We received reports of a power outage in the Santa Rosa Airport area and we noticed our equipment was a part of it. Currently we are trying to get services up with a back up generator while waiting for the power outage issue to get resolved.
ETA 2 hours for back up power.

-Network Operations

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

The systems include, among others:

• VPN servers
• Various public facing applications

Update: 12:40 AM Maintenance is complete

June 6th, at 12:01 AM we will be performing maintenance on our phone switch. No voice service interruptions are expected as a result of this maintenance. The window for this maintenance is 5 hours.
~Network Engineering