Category: Security

System Maintenance

Update: Maintenance complete

Starting at 11:59pm this evening, SOC will be running updates to multiple customer facing services. The following may experience hiccups during the maintenance window:

  • Webmail interfaces
  • Membertools
  • Forums
  • Sonic.com
  • Wiki.sonic.net

This post will be updated when the maintenance is complete.

-SOC

System Maintenance

Update: Maintenance Complete

 

Tonight at 11:59 pm we will be running a maintenance window to apply software updates to multiple customer facing systems. The following services may experience brief interruptions:

  • SMTP/IMAP/POP3
  • Customer tools (Webmail, Membertools)
  • Hosted MySQL Databases.
  • IPv6 Tunnels

The maintenance window is expected to last 1 hour.

-SOC

SSL Certificate Issues

UPDATE: Our CA has reissued our certificates, and they have been installed.

The Extended Validation (EV) certificates we installed last week have blocked a small, but significant, subset of users from accessing our websites due to the use of Elliptic Curve Cryptography (ECC).  In most cases, using Firefox instead of IE, Safari or Chrome resolves the problem.  We recognize this isn’t always and acceptable solution and have asked the Certificate Authority (CA) to reissue the certs using RSA and will install them as they become available.  However, we strongly encourage customers using Windows XP or old version of OSX to upgrade their operating systems to a modern supported version.  Using an old unsupported browser from an old unsupported operating system is a “Bad Idea.”  It is also exceedingly likely that future steps taken to ensure the privacy and security of our customers at large will cause similar issues for these old systems.

System Maintenence Tonight.

Update: Maintenance complete

Tonight starting at 11:59pm SOC will be updating software on some of our core systems. The following services may experience brief interruptions:

  • Website hosting
  • IPv6 tunnels
  • Incoming and outgoing mail

We will also be upgrading the SSL certificate for imap.sonic.net from SHA1 to SHA256. This is the last of our SSL certificates that we need to upgrade so we don’t expect most clients to have problems, but very old mail clients may not support the new certificate.

 

-Grant, Joe, and SOC

Connectivity Issues to Santa Rosa Datacenter

This morning, beginning around 3:45 AM, a host within our Santa Rosa datacenter became the target of a very large DDoS attack. We are currently in the process of battling this attack and users may notice reachability problems to services such as our email servers and member tools during this time.

– Robbie

The Heartbleed Bug and You: Change Your Passwords!

We always keep your privacy and security in mind.

By now you’d be hard pressed to have missed coverage of The Heartbleed bug in OpenSSL.  At this point, Sonic.net is joining many other providers and recommending that you change your passwords for your online services.  This is important for high value accounts like banking and finance or other accounts that protect your personal information and data.  Do not forget to change your ISP and email account passwords!  These are especially important since access to your email account can be used to gain access into most of your online services.

We do not have any reason to believe that we, or any of our users, were targeted.  However, this attack was undetectable and the cautious response is to assume that sensitive information has been leaked.  In the interest of full disclosure we are providing a complete list of affected services and systems.  It should be noted that all of the vulnerable services support PFS wherever possible and should our private keys have been leaked, they cannot be used to decrypt any past traffic in most cases.

Customers may change their passwords in the membertools using the password tool.

If you have any questions, please post them in our forums.

As of 21:45 on April 7th,  all vulnerable systems had received an update to fix this bug.

The following sites and services were vulnerable:

  • imap.sonic.net (login credentials were not vulnerable, only keys)
  • pop.sonic.net (login credentials were not vulnerable, only keys)
  • mail.sonic.net (login credentials and mailflow in/out)
  • legacy-webmail.sonic.net
  • webmail.sonic.net
  • forums.sonic.net
  • wiki.sonic.net
  • corp.sonic.net
  • newsignup.sonic.net
  • public-api.sonic.net (used by mobile apps)
  • fusionbroadband.com (used by our wholesale partners and customers)
  • srapi.sonic.net (used by our wholesale partners)

All of these systems have had their certificates replaced except for the following which are still pending reissue by our CAs:

UPDATE: All systems have had their keys replaced.

  • legacy-webmail.sonic.net
  • wiki.sonic.net
  • forums.sonic.net

The following sites and services were not vulnerable due to running an early version of OpenSSL:

  • mx.sonic.net (inbound mail)
  • members.sonic.net
  • signup.sonic.net
  • listman.sonic.net

OpenSSL Heartbleed Bug

A serious bug in OpenSSL was announced this afternoon known as the Heartbleed Bug.  An attacker, armed with the ability to exploit this bug is able to remotely read the contents of the memory of a vulnerably server.  This exposes the potential for an attacker to acquire the private key used to both encrypt the traffic and identify the server allowing them to eavesdrop on traffic as well as impersonate the server.  For a more in depth explanation of the bug and its affects see heartbleed.com  We have updated our servers with a local version of OpenSSL that disables Hearbeats to prevent an exploit pending new packages released by our OS upstream which fully resolves the issue.   -Kelsey

Update: April 8th, 17:35.  All affected public web and application servers received the fix from our OS upstream shortly after the original MOTD was posted yesterday.  Today, we’ve worked on wrapping up the upgrades on less critical systems and have reissued certificates for the bulk of the systems which had potentially exposed private keys. Ironically, we’re still waiting for all of our EV certs to be reissued.  The severity of this exploit can’t be underestimated as even earlier today Yahoo’s servers we’re still vulnerable exposing user names and passwords for the taking with little effort.  All users who run secure services should ensure that their systems are properly patched and consider having their certificates reissued by their CA.  -Kelsey and Grant