Inbound calling issue on Sonic’s fusion platform

We are currently working on an issue with a small subset of fusion customer unable to receive calls from outside Sonic’s network.
NOC

Update:  3:22pm 4/2/2014 The issue is effecting a large set of fusion customers than first reported . We are actively working to get this resolved as quickly as possible.

Update: 4:00 pm 4/2/2014 A routing mistake by one of our upstream providers caused the inbound calling issue. We are working with our upstream provider to ensure this does not happen again.

Legacy DSL Outage – Bay Area

Update (10:25AM): This problem has been resolved and affected customers are now back online.

A portion of legacy DSL users in the Bay Area are currently offline due to a DHCP server problem. We are currently working towards a resolution and will update this post when we have an update.

– Robbie

Legacy DSL Outage – Bay Area

Update: These customers are now back online.

Due to an apparent optical port failure, a small portion of our legacy DSL users in the bay area are currently offline. We will be moving these customers off the affected equipment ASAP.

– Robbie

Database Server Repair

Update: All work is complete as of 12:30am on March 16th.

Tonight, March 15th, at 11:59pm, Operations will be performing emergency maintenance on the database cluster to replace a suspected defective part. During this time, any web-based services such as Member Tools will be unavailable. Estimated time for repairs is no more than 30 minutes.

— Joe and the SOC

System updates

UPDATE: All work is complete.

 

Tonight at 11:59pm we will be preforming updates on several system clusters.  Impacted services include Member Tools, Webmail, mySQL databases, and customer sites. The downtime for each system is expected to be very brief.

 

Grant K.

System Operations

Non-intrusive Voice Maintenance

Tonight starting at 11:59PM we will be finishing a software upgrade on portions of our voice network in Santa Rosa. No customer impact is expected from this maintenance.

-Tim J.

Legacy DSL Outage – San Diego Area

Update (2:32PM): We have migrated off of the bad port and customers are now back online.

A portion of legacy DSL users in the San Diego area are currently offline due to a apparent optical port failure on a piece of our aggregation gear. We are currently working on a solution and will update this post when we have moved forward.

– Robbie

Non-Intrusive Voice Maintenance

Tonight beginning at 11:59PM PST we will begin performing a software upgrade of portions of our voice network in Santa Rosa. There should be no noticeable impact to users during this software upgrade. The upgrade is expected to take 2-3 hours.

-Tim J.

DDoS Towards Santa Rosa Data Center

Tonight, February 24, starting at 8:53pm, a large DDoS towards a collocation customer caused reachability issues for services hosted in our Santa Rosa data center. This malicious traffic has been blocked at our edge, and all connectivity has been restored.

 

-Tomoc

Mail server upgrades

Hello,

Tonight, at 11:59 pm, we will be preforming maintenance upgrades on servers in our mail cluster. Any interruptions caused by this should be brief.

Grant K.

System Operations