Author: Kelsey Cummings

Update 22:56:37 PDT: Everything looks good and all services appear to be functioning.

Update 21:37:03 PDT: The final cohort has been enabled.  We’ll continue to watch the cluster for next several hours to ensure that everything is working as intended.  It will still take some time before we’ve regained expected performance.

Update 19:17:35 PDT: Approximately 5/6 of user’s access to mail spools has been restored.  We’re still on target for having services fully restored in a few hours.

Update 17:39:57 PDT: Approximately 2/3 of user’s access to mail spools has been restored and while users can expect slow performance, the cluster and storage pools appear to be stable and chugging along.  At this point, we expect to have services fully restored later tonight.  Please note that no email was lost during this outage and our inbound servers were able to deliver mail without issue.

Update 16:19:00 PDT: The status of our email system is continuing to improve – we will post an update as our systems return to normal operation.

Update 13:00:56 PDT: In order to allow the cluster to recover we’ve taken steps to reduce the total number of user who can log in at once.  This appears to be having positive results and is allowing the storage pools to catch up with the load.  We still don’t have an ETA but the situation is improving.

Update: We believe the underlying issues have been been resolved.  However, it will take some time for IMAP and POP services to return to normal.  The fix has resulted in substantially increased IO load on our mail spool storage as all mailbox indexes have to be rebuilt.  This rebuild process can take a long time to complete, especially for large mailboxes, and it will be some time before the process is complete for all active mailboxes.

Update: We’re continuing to restore services for effected users.  Please note this may also affect some Fusion DSL user’s voicemail services.

Update: mailboxes for many users have been fixed and are working properly.  We’re continuing to repair remaining effected users.

Unfortunately, the mail server upgrade and rollback has caused issues for our IMAP cluster and it looks like most IMAP sessions are failing.  We’re working to restore services ASAP and will have more information forthcoming.  -Kelsey and Grant.

Our Santa Rosa Datacenter UPS’ will be undergoing their annual preventative maintenance on Thursday May 10th, starting at 10:00 AM.  This is a fully scripted event with our vendor and should not result in any service interruptions.

Update:  This maintenance was postponed and rescheduled to June 6th starting at 10AM.

– Sonic Facilities and Systems

UPDATE: Certificates deployed.

Do to unexpected issues with the IMAP cluster upgrades we neglected to deploy the new certificates to the old servers,.  Unfortunately the certs on the old servers just expired.  The new certs are being deployed now.  We are sorry for any inconvenience or concern this may have caused you.

-Sonic Operations

Early this morning a library used by many of our backend rpc services was updated as part of our nightly automatic upgrade process.  This particular update caused an issue that required services which depended upon it to be restarted in order for them to function correctly under some specific circumstances.  Unfortunately most of those specific circumstances are not tested for by our monitoring suite and so we didn’t discover the issue until this morning.

During this period we were unable to process some payments and the Fusion Voicemail system wasn’t able to process inbound calls correctly.  All services have been restored now and we’re continuing to dig into the underlying cause of the bug and what can be done to prevent it from happening again.

-Kelsey and the rest of System Operations

We will be performing updates and maintenance to the Sonic Member Tools at https://members.sonic.net tonight at 11 PM.

Maintenance is expected to last 30 minutes to 1 hour during which time access to the various member services such as bill payment, Graymail, and voice tools accessed through members.sonic.net will be unavailable.

– System Operations

Tonight (Wednesday, November 8th) at 11:59 PM, Webmail (https://webmail.sonic.net) will be unavailable for a brief time while we install a minor version upgrade.

Maintenance is expected to last 15 to 30 minutes.  No other mail services are affected, and you can use our Webmail Beta application to check your mail during the maintenance.

UPDATE: 12:14 AM.  Maintenance complete.

On Monday, August 21st at midnight, Sonic will be upgrading it’s Webmail platform. The upgrade is expected to take no more than 1 hour, during which time Webmail will be unavailable. Our beta platform will be available as a backup and other mail services will not be affected.

The upgrade includes numerous improvements, including improved performance opening folders and reading messages. If you are a frequent Webmail user, there are some impacting changes to be aware of.

Please see our forum post at https://forums.sonic.net/viewtopic.php?f=5&t=4791 for detailed information regarding the update if you use Webmail. You can also post questions, comments, or suggestions there or by emailing support@sonic.net.

The default layout has changed to be more functional on larger screens, including widescreen layouts. If you prefer to keep the current 2-column layout, refer to the forum post above for instructions on changing the settings.

In order to bring new features, support for Internet Explorer 9 and below will be dropped! If you use Webmail with IE 9 or below, it will no longer function properly. This change only affect users on Windows XP or Windows Vista. You can continue to use Webmail on XP or Vista by downloading Mozilla Firefox or an older version of Google Chrome. However, if you still use Internet Explorer 9 or below, consider upgrading as these versions are no longer safe to use and makes your computer more vulnerable to viruses and other threats.

We would encourage you to try the new version at https://webmail-beta.sonic.net prior to the upgrade to familiarize yourself with the changes and let us know if you encounter any problems using the new platform, have questions, or would like to provide feedback or suggestions.

– Sonic System Operations