Around 5:30PM today, routine network configuration on equipment serving Brentwood triggered a router bug that broke customer connectivity. Our support staff quickly alerted network engineering and we were able to revert the configuration and restore service. The NOC will in investigate the exact cause of the issue and take steps too prevent it’s recurrence.
-Network Engineering
Tonight at 11:59PM we are performing maintenance on aggregation gear serving our Brentwood fiber network. Customer impact should be minimal, but users may experience a few minutes of downtime. The maintenance window is expected too complete by 1AM.
-Network Engineering
Monday night (July 2nd) starting at 11:59PM, we will be performing followup maintenance on hardware serving Fusion and FlexLink Fiber customers in the Brentwood area. The work should not be customer impacting and the maintenance window is 3 hours.
-Network Engineering
Update: 1:10AM, Maintenance has successfully been completed without incident.
Tomorrow night (June 28th) starting at 11:59PM, we will be performing followup maintenance on hardware serving Fusion and FlexLink Fiber customers in the Brentwood area. The work should not be customer impacting and the maintenance window is 3 hours.
-Network Engineering
Update 12:25AM: This maintenance has been postponed for tonight.
Tomorrow night (June 12th) starting at 11:59PM, we will be performing maintenance on hardware serving Fusion and FlexLink Fiber customers in the Brentwood area. Though delicate in nature, we do not expect this work to be service impacting. The maintenance window is 2 hours.
-Network Engineering
1:30AM – This work has concluded for the night. Though not customer impacting, the operation was not fully successful, and a future maintenance window will be rescheduled soon.
This Saturday (6/9) starting at 5AM, one of the San Francisco data centers we collocate equipment, will be performing a planned power maintenance. Although we have been assured by the colocation provider this work will be non-impactful, we have taken additional preparations to verify redundancy on all critical hardware, and will have a network engineer on site as a fail-safe. The maintenance window officially extends until 6PM, and we will provide update when all work is completed.
– Network Engineering
(Update 8:12am) – We had a Power supply fail in a piece of our equipment during the first phase of the datacenter power maintenance. We quickly swapped out the cabling to restore the unit. Legacy DSL customers would have been impacted but service should now be restored.
Update(1:03pm) Power maintenance complete.
Tomorrow night (5/31) starting at 11:59PM, we will be performing maintenance on hardware serving our Business Park fiber customers in the Santa Rosa area. We do not expect this work to be service impacting for customers. The maintenance window is 1 hour.
– Network Engineering
Update: 1:50AM – Maintenance was completed by 1AM.
As of around 8:30PM this evening, we were alerted of an impacted piece of hardware serving fusion customers in Palo Alto. After some time troubleshooting, we identified the issue but even with vendor assistance were unable to remotely perform a repair. A technician is presently en route to replace the failed equipment. Update will be provided on completion of service restoration.
Update(6:06am): A subset of customers continues to be affected after physical equipment replacement. We are continuing to work with our vendor to find the root cause and fully restore service.
Update(8:24am): All services have been restored and we are working to protect ourselves from a failure of this nature in the future.
-Network Engineering
Today starting at 12:00pm, we will be performing maintenance on core routers in a series of POPs, spanning between Richmond, Vallejo, Napa, and Santa Rosa. This work should be non-intrusive to customer traffic. Estimated maintenance window is 8 hours, and an update will be provided upon completion.
-Network Engineering
Tonight beginning at midnight, a small subset of our Business Park Fiber customers may experience a brief interruption in service. The maintenance window is from 12:00am to 1:00am, with a max interruption of 5 minutes.
~Network Operations