Month: February 2014

Non-Intrusive Voice Maintenance

Tonight beginning at 11:59PM PST we will begin performing a software upgrade of portions of our voice network in Santa Rosa. There should be no noticeable impact to users during this software upgrade. The upgrade is expected to take 2-3 hours.

-Tim J.

DDoS Towards Santa Rosa Data Center

Tonight, February 24, starting at 8:53pm, a large DDoS towards a collocation customer caused reachability issues for services hosted in our Santa Rosa data center. This malicious traffic has been blocked at our edge, and all connectivity has been restored.

 

-Tomoc

Mail server upgrades

Hello,

Tonight, at 11:59 pm, we will be preforming maintenance upgrades on servers in our mail cluster. Any interruptions caused by this should be brief.

Grant K.

System Operations

Legacy DSL Outage

Update (11:20 PM): This circuit is now back online and fully functional.

Update: We are still working with our provider to resolve the issue. We have escalated to a very high level and continue to push for resolution as soon as possible.

Today, at 10:30AM, an aggregation circuit serving Legacy DSL for the Chico area went down. We are presently working with our upstream provider to isolate the cause and resolve the issue as quickly as possible. We will provide further updates shortly.

-Robbie and Tim

 

Legacy DSL Outage

We are currently tracking an issue affecting a small subset of Legacy DSL subscribers in LATA 1. There appears to be an issue with the aggregation equipment on our end and we are taking corrective actions as quickly as possible.

-Robbie and Tim

Brief LA Area Outage

From 10:09AM PST until 10:22AM PST we experienced blackholing on traffic between our Los Angeles area and San Francisco area networks. This may have caused brief email delays and voice call failures. The issue has been resolved and we are working to determine the cause of the outage and take steps to prevent future outages.

-Tim J

DNS Performance Issues

Users may have noticed delays or even failures to resolve hosts on our name servers off an on today until the issue was tracked down and resolved this afternoon.  This is related to the ongoing DNS amplification attacks, in this case, being sourced from unsecured or exploited customer equipment on our own network.

Sonic Telecom Backbone Maintenance

Tonight, beginning at midnight, we will be performing maintenance on equipment that serves Fusion and FlexLink products in and around the San Jose and Los Altos areas. While you may notice some brief routing changes, we do not expect any downtime from this operation.

Update: Maintenance complete with no customer downtime

– Robbie and Tomoc

Legacy DSL Outage

Today, at 9:58AM, an aggregation circuit serving legacy DSL for the Chico area went down. We are working with AT&T to get this as soon as possible, and will post an update as soon as we have one available.

Update: This outage was caused by a partial fiber cut to a backhaul circuit from one of our transit carriers. Technicians have been dispatched for repairs, but we have not received an ETR as of yet.

Update: Service appears to be restored. We are working with our provider to determine the details of this outage, and how we can prevent re-occurrence in the future.

-Tomoc

Sonic Telecom Inbound Calling Issue

We are presently experiencing an inbound calling issue impacting customers on the Sonic telecom network. We are working to identify the cause of problem and repair said issue. Further update will be provided upon resolution.

-Sonic NOC

 

Update: As of 12:30 pm today the issue with inbound calls to Sonic telecom network should be resolved.

-Sonic NOC