Month: October 2014

In an effort to better protect our customer's data, we have been working
on updating our SSL protected services to the industries "best
practice". As of yesterday, we have ceased the use of SSLv3 on our
member services including Member Tools, Webmail, and our mail cluster.
This was to protect against a new vulnerability in the SSLv3 protocol
that has been dubbed "POODLE". More information for that may be found
here:
https://blog.mozilla.org/security/2014/10/14/the-poodle-attack-and-the-end-of-ssl-3-0/ .

It is also recommended that users disable SSLv3 in their clients. Some
older clients may not be able to disable this protocol, and the only
course of action is to update to a client that supports TLS. For more
information on how to configure your clients, see the following link:

https://zmap.io/sslv3/browsers.html
 
In other SSL related news, the SHA-1 hashing algorithm is to be phased
out over the next few years. As a result of this announcement, we have
been reissuing our certificates to the SHA-2 hashing algorithm over the
last week.

Any discussion of SSL should be directed to the forums.
https://forums.sonic.net/

Kelsey, Grant, and SOC

Tonight, starting at 9:41pm, fiber serving a subset of Orange County Fusion/FlexLink customers went down. We are currently working to restore service as quickly as possible.

Update: Fiber has been repaired and service restored

-Tomoc

A routine configuration change on our MX cluster had some unintended consequences and took inbound mail offline about 45 minutes ago.  We’re in the process of consoling the servers now to restore services and expect to have them back up and running shortly.  Due to some quirks with how some of our internal systems function this has also impacted our member tools servers and is adding a substantial delay to login and subsequent page loads.

Update:  All services were restored shortly after the original post.  Postmortem of the failure revealed that a new log messages stream from the MX cluster caused some synchronous blocking behavior leading to excessive resource consumption and eventual lock up of all servers in the cluster.  We have also addressed delays in our initial response to the outage.  -Kelsey and Grant.