OpenSSL Heartbleed Bug

A serious bug in OpenSSL was announced this afternoon known as the Heartbleed Bug.  An attacker, armed with the ability to exploit this bug is able to remotely read the contents of the memory of a vulnerably server.  This exposes the potential for an attacker to acquire the private key used to both encrypt the traffic and identify the server allowing them to eavesdrop on traffic as well as impersonate the server.  For a more in depth explanation of the bug and its affects see heartbleed.com  We have updated our servers with a local version of OpenSSL that disables Hearbeats to prevent an exploit pending new packages released by our OS upstream which fully resolves the issue.   -Kelsey

Update: April 8th, 17:35.  All affected public web and application servers received the fix from our OS upstream shortly after the original MOTD was posted yesterday.  Today, we’ve worked on wrapping up the upgrades on less critical systems and have reissued certificates for the bulk of the systems which had potentially exposed private keys. Ironically, we’re still waiting for all of our EV certs to be reissued.  The severity of this exploit can’t be underestimated as even earlier today Yahoo’s servers we’re still vulnerable exposing user names and passwords for the taking with little effort.  All users who run secure services should ensure that their systems are properly patched and consider having their certificates reissued by their CA.  -Kelsey and Grant

ATM Customer Aggregation Router Reload

Tonight, April 7, starting at 11:59PM, we will be performing a maintenance reload on our customer facing ATM aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Tomoc

 

fusion phone numbers deactivated incorrectly

A small subset of fusion telephone numbers were deactivated in our phone switch this morning incorrectly. We are currently working to activate the telephone numbers for all effect customers.

NOC

Update: 10:40 am 4/4/2014 – All telephone numbers have been reactivated. Some voicemail settings may have been reset to default for the affected customers. We will be sending an email out to all affected customers.

NOC

Mailing List Maintenance

Today at 13:20 PST the mailing list server will be taken offline for maintenance.  All work should be completed within 15 minutes.

– Maintenance completed

Inbound calling issue on Sonic’s fusion platform

We are currently working on an issue with a small subset of fusion customer unable to receive calls from outside Sonic’s network.
NOC

Update:  3:22pm 4/2/2014 The issue is effecting a large set of fusion customers than first reported . We are actively working to get this resolved as quickly as possible.

Update: 4:00 pm 4/2/2014 A routing mistake by one of our upstream providers caused the inbound calling issue. We are working with our upstream provider to ensure this does not happen again.

Legacy DSL Outage – Bay Area

Update (10:25AM): This problem has been resolved and affected customers are now back online.

A portion of legacy DSL users in the Bay Area are currently offline due to a DHCP server problem. We are currently working towards a resolution and will update this post when we have an update.

– Robbie

Legacy DSL Outage – Bay Area

Update: These customers are now back online.

Due to an apparent optical port failure, a small portion of our legacy DSL users in the bay area are currently offline. We will be moving these customers off the affected equipment ASAP.

– Robbie

Database Server Repair

Update: All work is complete as of 12:30am on March 16th.

Tonight, March 15th, at 11:59pm, Operations will be performing emergency maintenance on the database cluster to replace a suspected defective part. During this time, any web-based services such as Member Tools will be unavailable. Estimated time for repairs is no more than 30 minutes.

— Joe and the SOC

System updates

UPDATE: All work is complete.

 

Tonight at 11:59pm we will be preforming updates on several system clusters.  Impacted services include Member Tools, Webmail, mySQL databases, and customer sites. The downtime for each system is expected to be very brief.

 

Grant K.

System Operations

Non-intrusive Voice Maintenance

Tonight starting at 11:59PM we will be finishing a software upgrade on portions of our voice network in Santa Rosa. No customer impact is expected from this maintenance.

-Tim J.