Category: Uncategorized

In an effort to better protect our customer's data, we have been working
on updating our SSL protected services to the industries "best
practice". As of yesterday, we have ceased the use of SSLv3 on our
member services including Member Tools, Webmail, and our mail cluster.
This was to protect against a new vulnerability in the SSLv3 protocol
that has been dubbed "POODLE". More information for that may be found
here:
https://blog.mozilla.org/security/2014/10/14/the-poodle-attack-and-the-end-of-ssl-3-0/ .

It is also recommended that users disable SSLv3 in their clients. Some
older clients may not be able to disable this protocol, and the only
course of action is to update to a client that supports TLS. For more
information on how to configure your clients, see the following link:

https://zmap.io/sslv3/browsers.html
 
In other SSL related news, the SHA-1 hashing algorithm is to be phased
out over the next few years. As a result of this announcement, we have
been reissuing our certificates to the SHA-2 hashing algorithm over the
last week.

Any discussion of SSL should be directed to the forums.
https://forums.sonic.net/

Kelsey, Grant, and SOC

Tonight, starting at 9:41pm, fiber serving a subset of Orange County Fusion/FlexLink customers went down. We are currently working to restore service as quickly as possible.

Update: Fiber has been repaired and service restored

-Tomoc

A routine configuration change on our MX cluster had some unintended consequences and took inbound mail offline about 45 minutes ago.  We’re in the process of consoling the servers now to restore services and expect to have them back up and running shortly.  Due to some quirks with how some of our internal systems function this has also impacted our member tools servers and is adding a substantial delay to login and subsequent page loads.

Update:  All services were restored shortly after the original post.  Postmortem of the failure revealed that a new log messages stream from the MX cluster caused some synchronous blocking behavior leading to excessive resource consumption and eventual lock up of all servers in the cluster.  We have also addressed delays in our initial response to the outage.  -Kelsey and Grant.

Starting at 11:59pm, System Operations will be performing maintenance updates to the following systems:

• Web Hosting
• IPv6 tunnels
• Email

Service interruptions should be brief.

-Grant and SOC

A portion of our Flexlink Long Range customers are currently offline in the Burlingame area. We have dispatched a technician to the CO to troubleshoot this issue. ETR is ~2.5hours.

-Tim J.

We are experiencing issues with a legacy DSL aggregation circuit causing extreme slowdowns for subscribers in the Chico area. We are working with our providers to resolve this issue as soon as possible.

Update (9:44pm) : We are continuing to work with our providers to isolate this issue. Updates will be posted as soon as more information is available.

Update (3:34am): Technicians have been dispatched to isolate the troubled span, still no ETR.

-Tomoc

Starting September 16th, mail.sonic.net will require all users authenticate using their username and password to send mail even when directly connected to our network.  This is the default configuration for most mail clients and will only affect a small group of users.  This change makes it easier for us to track and prevent spam from being sent using our mail infrastructure.  -Kelsey and Grant

UPDATE: All work is complete.

Tonight at 11:59 pm, we will be updating several customer facing systems. The services effected will include:

• Customer Wiki
• Forums
• Webmail
• Outbound mail

Any interruptions should be brief.

-Grant and SOC

Tonight, August 20th, starting at 11:59pm, we will be performing maintenance on equipment serving a subset of Legacy DSL subscribers in the Los Angeles area. Expected downtime is 15 minutes.

-Tomoc