Update: Maintenance complete.
Tonight at 11pm we will be performing updates to several systems; Impacted customer facing services will include mail and member tools. Â Any interruptions to service should be brief and the maintenance should be completed within 1 hour. Thank you.
Update: All updates were complete as of 1:00am this morning.
System Operations will be performing updates tonight starting at 11:59pm. The following services may be temporarily unreachable for a few minutes as servers are rebooted:
— Joe and the SOC
Update: Maintenance complete
Tonight starting at 11:59pm SOC will be updating software on some of our core systems. The following services may experience brief interruptions:
We will also be upgrading the SSL certificate for imap.sonic.net from SHA1 to SHA256. This is the last of our SSL certificates that we need to upgrade so we don’t expect most clients to have problems, but very old mail clients may not support the new certificate.
-Grant, Joe, and SOC
Over the past several months we’ve watched a marked increase in the spam making its way through our filtering.  And we’re not alone.  Spam volume has increased over 6 times since October 2013 globally and many spammers have become increasingly sophisticated in thwarting filters, explicitly testing and tuning delivery through common systems to ensure their spam reaches their targets.  In response, we’ve completed several upgrades that have had a big impact reducing spam in our inboxes.
We finished a migration to our next generation MX servers last week.  The new MX servers feature a completely rewritten custom policy enforcement engine.  Like most MX filters, it makes heavy use of a collection of reputation services in conjunction with policies like rate limits and checks for basic compliance.  There’s one important difference with the new system: it only allows customers to opt-out of the filtering where our old MX filters allowed for some per user customization.  When a customer opts out, we’ll make our “best effort” at excluding a customer’s inbox from any filtering.
We’ve also made a couple of  important changes to our SpamAssassin installation including the integration of a commercial plugin.  This, along with a redoubled effort to maintain our own set of custom rules, has significantly increased its accuracy. We hope these changes have made as big an impact on your inbox as they have on ours.
Customers may opt-out of the new MX filters and adjust their SpamAssassin preferences using the Spam Configuration membertool.  Please join us in the forums if you have any questions.
-Kelsey
One of the interfaces in the load balancer that handles, mail.sonic.net, pop.sonic.net and imap.sonic.net had an interface lock up and stop passing traffic. During the few minutes this was an issue, these services would have been intermittently available. Traffic has been manually rerouted through the standby load balancer. Under nearly every other failure possible the standby load balancer would automatically take over but, unfortunately, its ability to deal with this lockup is not one of them. -Kelsey
Update:
All work is complete as of 1:00am on Wednesday, April 9th.
Tonight, April 8th, at 11:59pm, System Operations will be updating and rebooting several systems, including load balancers. This may cause a brief interruption in any web-based services and mail.
— Joe and the SOC
UPDATE: All work is complete.
Tonight at 11:59pm we will be preforming updates on several system clusters. Â Impacted services include Member Tools, Webmail, mySQL databases, and customer sites. The downtime for each system is expected to be very brief.
Grant K.
System Operations
UPDATE: Upgrades are complete.
Tonight, 12/9/13 at 11:59 PM we will be performing storage upgrades across all mail spools. We do not anticipate any interruption in mail services during this upgrade.
-William
Operations has observed that emails from sonic.net customers to sbcglobal.net addresses are currently being refused. We are working with sbcglobal.net and Yahoo to resume delivery of mail.
Update:
Mail resumed delivery to sbcglobal yesterday (Dec 2) around 10:30pm.
— Joe and the SOC
Tonight, at 12:00AM, we will migrate “mail.sonic.net” to a new cluster. Â Users may be unable to send mail through mail.sonic.net during the migration which is expected to take 10-15 minutes to complete. Â There’s no new exciting features to tout, just a fresh OS and faster modern hardware.
In other news, our MX server cluster now supports SSL encryption for inbound connections.  This allows the session between the connecting server and our’s to be fully encrypted preventing messages from being available “on the wire” in clear text.  Our outbound servers have supported this kind of encryption when sending mail to destinations that support it for more than a decade.  At this time it looks like approximately 15-20% of inbound email is encrypted in transit.
-Kelsey
Update: Â Wed Nov 13 00:48:37 PST 2013, the upgrades are finished and went over as planned.