Update: Maintenance was completed at 1:45AM (10/19/18).
Thursday (10/18/18) at 11:59PM we will performing intrusive maintenance on another subset of Fusion Fiber customers located in San Francisco’s Richmond District. Approximate expected customer downtime is 30 minutes. An update to this MOTD will be provided when all customer connectivity has been restored.
-Network Engineering
The bulk of voice service was restored at 5:15 PM when an emergency repair was engineered by our team. It was determined additional manual intervention was necessary to restore voice for a small subset of customers, which we expect to be complete by 7:00PM.
At approximately 3:15 today, our support began getting reports of Fusion Fiber customers in the Mission reporting voice services down. We are working to restore connectivity as quickly as possible and will update as more details become available.
-Network Engineering
Update: Maintenance was completed at 3:30AM 10/12/2018.
Thursday (10/11/18) night at 11:59PM we will performing intrusive maintenance on a subset of Fusion Fiber customers located in San Francisco’s Richmond District. Approximate expected customer downtime is 30 minutes. An update to this MOTD will be provided when all customer connectivity has been restored.
-Network Engineering
Around 5:30PM today, routine network configuration on equipment serving Brentwood triggered a router bug that broke customer connectivity. Our support staff quickly alerted network engineering and we were able to revert the configuration and restore service. The NOC will in investigate the exact cause of the issue and take steps too prevent it’s recurrence.
-Network Engineering
Tonight at 11:59PM we are performing maintenance on aggregation gear serving our Brentwood fiber network. Customer impact should be minimal, but users may experience a few minutes of downtime. The maintenance window is expected too complete by 1AM.
-Network Engineering
Tomorrow night (June 28th) starting at 11:59PM, we will be performing followup maintenance on hardware serving Fusion and FlexLink Fiber customers in the Brentwood area. The work should not be customer impacting and the maintenance window is 3 hours.
-Network Engineering
Update 12:25AM: This maintenance has been postponed for tonight.
As of around 8:30PM this evening, we were alerted of an impacted piece of hardware serving fusion customers in Palo Alto. After some time troubleshooting, we identified the issue but even with vendor assistance were unable to remotely perform a repair. A technician is presently en route to replace the failed equipment. Update will be provided on completion of service restoration.
Update(6:06am): A subset of customers continues to be affected after physical equipment replacement. We are continuing to work with our vendor to find the root cause and fully restore service.
Update(8:24am): All services have been restored and we are working to protect ourselves from a failure of this nature in the future.
-Network Engineering
Wednesday, starting at 11:59pm, we will be performing non-intrusive maintenance on core equipment throughout the Northern California Sonic Telecom core network. Maintenance window is 4 hours.
Update (2:10am): Maintenance is now complete, with no customer impact.
– Network Engineering
Over the pass 24 hours we experienced 2 brief intervals of errors towards a piece of equipment serving a small subset of customers in the San Francisco area. We have identified the faulty equipment and it has been removed from production and we will dispatch for immediate replacement. The replacement of the equipment will cause no downtime.
-Network Engineering
Tomorrow night(2/14/18), starting at 11:59PM, a small subset of customers in the San Francisco area will experience a brief service interruption while we perform some maintenance. Expected down time is about 15 minutes while the equipment reboots. The maintenance window will be 2 hours.
– Network Engineering
Update(1:27am) Maintenance is now complete