Tonight, starting at 9:41pm, fiber serving a subset of Orange County Fusion/FlexLink customers went down. We are currently working to restore service as quickly as possible.
Update: Fiber has been repaired and service restored
-Tomoc
MX Server Outage
A routine configuration change on our MX cluster had some unintended consequences and took inbound mail offline about 45 minutes ago. We’re in the process of consoling the servers now to restore services and expect to have them back up and running shortly. Due to some quirks with how some of our internal systems function this has also impacted our member tools servers and is adding a substantial delay to login and subsequent page loads.
Update: All services were restored shortly after the original post. Postmortem of the failure revealed that a new log messages stream from the MX cluster caused some synchronous blocking behavior leading to excessive resource consumption and eventual lock up of all servers in the cluster. We have also addressed delays in our initial response to the outage. -Kelsey and Grant.
Flexlink Intrusive Maintenance in Brea
Update:
This maintenance is now complete.
Tomorrow morning, starting at 3:30am, we will be performing intrusive maintenance on the equipment serving Flexlink customers in the Brea area. We estimate that customers will experience about 15 minutes of downtime during this maintenance.
-Clay
Legacy DSL Outage
Update(6:45PM): Affected customers have been migrated and are now back online.
A portion of legacy dsl customers in northern California are currently offline due to an atm port failure. Those customers are being migrated to a backup port and should be back online once that process is complete.
– Robbie
Legacy DSL Maintenance
Tonight, starting at 11:59PM, we will be performing maintenance on gear that aggregates some legacy DSL connections in northern California. Expected downtime is less than 10 minutes.
– Robbie
System Maintenance
Starting at 11:59pm, System Operations will be performing maintenance updates to the following systems:
- Web Hosting
- IPv6 tunnels
Service interruptions should be brief.
-Grant and SOC
Fusion/FlexLink Intrusive Maintenance
Update (12:15AM): This maintenance is now complete.
Tonight, beginning at 11:59 PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers located in eastern Santa Rosa. Expected downtime is less than 15 minutes.
– Robbie
Flexlink Long Range Outage – Burlingame
A portion of our Flexlink Long Range customers are currently offline in the Burlingame area. We have dispatched a technician to the CO to troubleshoot this issue. ETR is ~2.5hours.
-Tim J.
FlexLink Ethernet Planned Outage
A very small subset of enterprise customers served out of San Francisco will experience a breif outage tonight at midnight due to routine system maintainence.
Expected downtime will be less than 5 minutes.
-Mike Tardif
-NOC
Chico Legacy DSL Slowdown
We are experiencing issues with a legacy DSL aggregation circuit causing extreme slowdowns for subscribers in the Chico area. We are working with our providers to resolve this issue as soon as possible.
Update (9:44pm) : We are continuing to work with our providers to isolate this issue. Updates will be posted as soon as more information is available.
Update (3:34am): Technicians have been dispatched to isolate the troubled span, still no ETR.
-Tomoc