Month: April 2014

OpenSSL Heartbleed Bug

A serious bug in OpenSSL was announced this afternoon known as the Heartbleed Bug.  An attacker, armed with the ability to exploit this bug is able to remotely read the contents of the memory of a vulnerably server.  This exposes the potential for an attacker to acquire the private key used to both encrypt the traffic and identify the server allowing them to eavesdrop on traffic as well as impersonate the server.  For a more in depth explanation of the bug and its affects see heartbleed.com  We have updated our servers with a local version of OpenSSL that disables Hearbeats to prevent an exploit pending new packages released by our OS upstream which fully resolves the issue.   -Kelsey

Update: April 8th, 17:35.  All affected public web and application servers received the fix from our OS upstream shortly after the original MOTD was posted yesterday.  Today, we’ve worked on wrapping up the upgrades on less critical systems and have reissued certificates for the bulk of the systems which had potentially exposed private keys. Ironically, we’re still waiting for all of our EV certs to be reissued.  The severity of this exploit can’t be underestimated as even earlier today Yahoo’s servers we’re still vulnerable exposing user names and passwords for the taking with little effort.  All users who run secure services should ensure that their systems are properly patched and consider having their certificates reissued by their CA.  -Kelsey and Grant

ATM Customer Aggregation Router Reload

Tonight, April 7, starting at 11:59PM, we will be performing a maintenance reload on our customer facing ATM aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Tomoc

 

fusion phone numbers deactivated incorrectly

A small subset of fusion telephone numbers were deactivated in our phone switch this morning incorrectly. We are currently working to activate the telephone numbers for all effect customers.

NOC

Update: 10:40 am 4/4/2014 – All telephone numbers have been reactivated. Some voicemail settings may have been reset to default for the affected customers. We will be sending an email out to all affected customers.

NOC

Mailing List Maintenance

Today at 13:20 PST the mailing list server will be taken offline for maintenance.  All work should be completed within 15 minutes.

– Maintenance completed

Inbound calling issue on Sonic’s fusion platform

We are currently working on an issue with a small subset of fusion customer unable to receive calls from outside Sonic’s network.
NOC

Update:  3:22pm 4/2/2014 The issue is effecting a large set of fusion customers than first reported . We are actively working to get this resolved as quickly as possible.

Update: 4:00 pm 4/2/2014 A routing mistake by one of our upstream providers caused the inbound calling issue. We are working with our upstream provider to ensure this does not happen again.