Author: mike.ely@sonic.com

Systems Maintenance

Update: Maintenance complete. Some unexpected instability was encountered with the mail cluster which caused the maintenance window to run long.

Beginning at 11pm tonight (4/2/2025), System Operations will be performing updates to the following systems:

  • sonic.com
  • Webmail/Webmail Beta
  • Forums
  • Speedtest

Maintenance work for each system is expected to be very brief, and the overall maintenance period is scheduled for one hour. An update will be sent upon completion.

Systems Maintenance 2/19/2025

Between 11 and midnight tonight the System Operations team will be transitioning a key backend server to our virtualization platform. While most systems will continue to run normally during this shift, some instability with various elements of our Member Tools may be encountered. An update will be posted once work is complete.

Update: Maintenance is complete, all systems appear healthy.

Systems Maintenance

Update: This maintenance has been rescheduled for 10pm tonight (September 5th). We apologize for any inconvenience.

Tonight (September 4th) at 11:30pm PST, we will be performing maintenance on various services including the Sonic VPN cluster. Downtime for each service is expected to be brief, and maintenance should be completed within one hour.

Systems Maintenance

Update: Due to unforeseen circumstances, this work has been delayed until tonight, 8/6/24. Apologies for any inconvenience.

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems. Any downtime to affected public services should be brief, and we expect to complete the maintenance within 2 hours.

Systems Maintenance

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems. Any downtime to affected public services should be brief, and we expect to complete the maintenance within 2 hours.

An update will be posted after work is complete.

Systems Maintenance

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

Impacted systems include the shell server as well as various other systems. A follow-up notification will occur once all maintenance is complete.

Update 11:34pm: Maintenance compete

Partial mail service disruption: 3/26-3/27

Yesterday afternoon at approximately 4:45PM we completed the turn up and migration of services to a new set of geographically redundant proxy servers used by our internal mail infrastructure to communicate between the various clusters that support email services. Unfortunately one of the new proxies was left out of an ACL which prevented that proxy from being able to connect to the mail forwarding cluster.  This resulted in about 50% of forwarded mail (typically configured in the “Forwarding” section of Sonic Member Tools) being rejected.  These rejected messages would have resulted in non-deliverable messages being returned to the original sender instead of being delivered as intended.

We apologize for any issues this may have caused and will review our systems for any opportunities to improve monitoring to ensure that if something like this happens again it is immediately detected and resolved.

System Maintenance

Update: Work is complete.

Tonight (3/14/24) beginning at 11pm we will be conducting maintenance on various Sonic systems. Downtime on individual systems will be limited to a few minutes each, and the maintenance work overall is scheduled to last for approximately one hour.