(Update: As of 9:58 AM, all backbone repairs have completed. A few individual customers need the drop to their premises need to be re-done, and we are following up with them).
A small subset of our customers in Albany are experiencing a Fiber outage. A tree limb brought down the PG&E lines and our Fiber backbone in the area.
We have dispatched a repair team to replace the downed Fiber Strand.
ETR is 5-9 hours for complete repairs.
Update: All updates complete.
Various systems are receiving a critical kernel update and will need to be rebooted. Downtime for each system is expected to be brief, but work will progress throughout the day until updates are complete.
UPDATE 3:00AM – Maintenance successfully completed.
Thursday (5/16/19) at 11:59PM, we will be performing upgrades on equipment serving Fusion Fiber and FlexLink Fiber customers in the Sunset District of San Francisco. We do not expect any customer impact during this work. Our maintenance window will last up to 4:00AM. An update will be provided on completion.
-Network Engineering
Update(2:35am): An unexpected router bug caused a routing loop affecting traffic in our Santa Rosa data center for approximately 20 minutes. This would have affected local colo customers, as well as some internal Sonic infrastructure. Maintenance is complete and all services are restored.
Tonight, May 14, starting at 11:59pm, we will be performing maintenance on network edge equipment serving the Northern California Sonic network. Customers may experience brief intervals of routing instability as traffic is shifted to backup paths, however this maintenance should not cause any customer down time. This maintenance window is 4 hours.
-Network Engineering
This maintenance has been postponed and will be rescheduled at a later date.
Update: This maintenance has been rescheduled for Tuesday, May 14th.
Tonight beginning at 11pm we will be performing maintenance on several systems including our customer shell server. We expect any downtime to be minimal and for the maintenance to be completed within 2 hours.
Update 7:36pm: The customers that were impacted by this outage should now be restored.
Update 5:46pm: ATT has created work orders to reprovision the impacted customers. Currently no ETA on when those orders will complete. We will be checking constantly on the work order status and update when we have a more firm ETR.
A subset of our Enterprise Fiber DIA SSE customers are currently experiencing an outage. We are promptly working to get this resolved as soon as possible. Please standby for updates.
-Network Operations
Update @ 11:24pm
Maintenance complete.
—
Tonight (Wednesday 5/8/2019) between 11pm and 4am we will be performing work to our systems affecting a small subset of customers in Vallejo to repair an unexpected consequence of last night’s operation that caused an outage. Expected downtime from this work is 3-5 minutes of network connectivity issues that will resolve automatically.
4:00AM – Maintenance successfully completed.
Tonight (5/08/19) at 11:59PM, we will be performing upgrades on equipment serving Fusion Fiber and FlexLink Fiber customers in the Sunset District of San Francisco. We do not expect any customer impact during this work. Our maintenance window will last up to 4:00AM. An update will be provided on completion.
-Network Engineering
Webmail was unavailable or otherwise slow from approximately 3am – 9am this morning (May 8th). The Operations team was able to isolate the problem and is taking steps to improve monitoring going forward.
— Joe @ System Operations