Legacy DSL Outage – Bay Area

Update (10:25AM): This problem has been resolved and affected customers are now back online.

A portion of legacy DSL users in the Bay Area are currently offline due to a DHCP server problem. We are currently working towards a resolution and will update this post when we have an update.

– Robbie

Legacy DSL Outage – Bay Area

Update: These customers are now back online.

Due to an apparent optical port failure, a small portion of our legacy DSL users in the bay area are currently offline. We will be moving these customers off the affected equipment ASAP.

– Robbie

Database Server Repair

Update: All work is complete as of 12:30am on March 16th.

Tonight, March 15th, at 11:59pm, Operations will be performing emergency maintenance on the database cluster to replace a suspected defective part. During this time, any web-based services such as Member Tools will be unavailable. Estimated time for repairs is no more than 30 minutes.

— Joe and the SOC

System updates

UPDATE: All work is complete.

 

Tonight at 11:59pm we will be preforming updates on several system clusters.  Impacted services include Member Tools, Webmail, mySQL databases, and customer sites. The downtime for each system is expected to be very brief.

 

Grant K.

System Operations

Non-intrusive Voice Maintenance

Tonight starting at 11:59PM we will be finishing a software upgrade on portions of our voice network in Santa Rosa. No customer impact is expected from this maintenance.

-Tim J.

Legacy DSL Outage – San Diego Area

Update (2:32PM): We have migrated off of the bad port and customers are now back online.

A portion of legacy DSL users in the San Diego area are currently offline due to a apparent optical port failure on a piece of our aggregation gear. We are currently working on a solution and will update this post when we have moved forward.

– Robbie

Non-Intrusive Voice Maintenance

Tonight beginning at 11:59PM PST we will begin performing a software upgrade of portions of our voice network in Santa Rosa. There should be no noticeable impact to users during this software upgrade. The upgrade is expected to take 2-3 hours.

-Tim J.

DDoS Towards Santa Rosa Data Center

Tonight, February 24, starting at 8:53pm, a large DDoS towards a collocation customer caused reachability issues for services hosted in our Santa Rosa data center. This malicious traffic has been blocked at our edge, and all connectivity has been restored.

 

-Tomoc

Mail server upgrades

Hello,

Tonight, at 11:59 pm, we will be preforming maintenance upgrades on servers in our mail cluster. Any interruptions caused by this should be brief.

Grant K.

System Operations

Legacy DSL Outage

Update (11:20 PM): This circuit is now back online and fully functional.

Update: We are still working with our provider to resolve the issue. We have escalated to a very high level and continue to push for resolution as soon as possible.

Today, at 10:30AM, an aggregation circuit serving Legacy DSL for the Chico area went down. We are presently working with our upstream provider to isolate the cause and resolve the issue as quickly as possible. We will provide further updates shortly.

-Robbie and Tim