Update(12:45AM): This maintenance is now complete. A single card on a DSLAM did not upgrade properly and is now operating incorrectly. We will be dispatching a technician to replace the card this morning.
Update(11:00PM): This maintenance has been postponed until tomorrow night.
Tonight, beginning at 11:59 PM, we will be performing intrusive maintenance on equipment serving some Fusion and FlexLink customers located in San Francisco. Expected downtime is less than 15 minutes.
– Robbie
Tonight, starting at 9:41pm, fiber serving a subset of Orange County Fusion/FlexLink customers went down. We are currently working to restore service as quickly as possible.
Update: Fiber has been repaired and service restored
-Tomoc
A routine configuration change on our MX cluster had some unintended consequences and took inbound mail offline about 45 minutes ago. Â We’re in the process of consoling the servers now to restore services and expect to have them back up and running shortly. Â Due to some quirks with how some of our internal systems function this has also impacted our member tools servers and is adding a substantial delay to login and subsequent page loads.
Update: Â All services were restored shortly after the original post. Â Postmortem of the failure revealed that a new log messages stream from the MX cluster caused some synchronous blocking behavior leading to excessive resource consumption and eventual lock up of all servers in the cluster. Â We have also addressed delays in our initial response to the outage. Â -Kelsey and Grant.
Update:
This maintenance is now complete.
Tomorrow morning, starting at 3:30am, we will be performing intrusive maintenance on the equipment serving Flexlink customers in the Brea area. We estimate that customers will experience about 15 minutes of downtime during this maintenance.
-Clay
Update(6:45PM): Affected customers have been migrated and are now back online.
A portion of legacy dsl customers in northern California are currently offline due to an atm port failure. Those customers are being migrated to a backup port and should be back online once that process is complete.
– Robbie
Tonight, starting at 11:59PM, we will be performing maintenance on gear that aggregates some legacy DSL connections in northern California. Expected downtime is less than 10 minutes.
– Robbie
Starting at 11:59pm, System Operations will be performing maintenance updates to the following systems:
Service interruptions should be brief.
-Grant and SOC
Update (12:15AM): This maintenance is now complete.
Tonight, beginning at 11:59 PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers located in eastern Santa Rosa. Expected downtime is less than 15 minutes.
– Robbie
A portion of our Flexlink Long Range customers are currently offline in the Burlingame area. We have dispatched a technician to the CO to troubleshoot this issue. ETR is ~2.5hours.
-Tim J.
A very small subset of enterprise customers served out of San Francisco will experience a breif outage tonight at midnight due to routine system maintainence.
Expected downtime will be less than 5 minutes.
-Mike Tardif
-NOC