Author: Mike Ely

Update: Maintenance complete

Tonight beginning at 11pm we will begin routine maintenance of multiple public-facing systems, including webmail and VPN. Downtime for each system should be a few minutes, and the maintenance window is one hour.

Tonight (May 6 2021) beginning at 10PM we will be applying updates to multiple systems including our VPN servers. Any downtime should be very brief, and we expect to complete the maintenance within 2 hours.

Beginning at midnight tonight we will be upgrading systems related to address qualification. During this time looking up certain addresses on sonic.com/availability may return an error. Our maintenance window for this work is one hour.

Update 12:45am: Work is complete.

Beginning at 11pm tonight the Sonic VPN servers, both production and beta, will be migrated to new infrastructure. This will require brief downtime (roughly 5 minutes) for each server.

Update: This work was completed successfully before 11:15pm.

Starting at 11pm this evening the Sonic VPN server will be taken offline for updates. We expect this outage to last no more than fifteen minutes. An update will be posted here once maintenance is complete.

Starting at 10pm tonight System Operations will conduct planned maintenance on a host serving bandwidth graphs for certain legacy products. This will impact specific applications that display graphs for those legacy products – no other impact is expected. Total downtime should be approximately one hour.

Update: As of 12:10am maintenance is complete and all systems are in good working order.

Beginning at midnight tonight System Operations will be performing maintenance on various servers, including the shell server. Expected downtime for each system is brief, and operations should be completed within one hour.

Tonight at midnight System Operations will perform maintenance to the shell server. Downtime for this operation should only last a few minutes. An update will be posted here once maintenance is complete.

Update: Maintenance was inadvertently completed early. We apologize for the inconvenience.

Tonight beginning at 11pm Sonic System Operations will conduct physical maintenance at one of our remote datacenters. There is a possibility that this work will have a brief impact on multiple customer-facing systems including the Sonic VPN server. Any downtime encountered is expected to be brief. An update will be posted once maintenance is complete.

Update: This maintenance, previously scheduled for midnight, has been changed to begin at 11pm in order to adequately schedule needed resources. We apologize for any disruption this change may cause.

Conclusion: Maintenance is complete. No outages were observed.

Due to a switch configuration issue, various customer-facing systems are currently offline or not functioning as intended. System Operations and Network Engineering are working to resolve the issue. This post will be updated when more information is available.

Update: Network Engineering has isolated the problem to a code upgrade in a core switch cluster. After rolling the upgrade back functionality has been restored. Work to determine the specific failure mode and future mitigation steps will be conducted during business hours.