Starting at approximately 8am this morning the Sonic shell server experienced an outage. The system was brought back online at approximately 8:50am. Work is underway to determine a cause and subsequent solution for the issue.
Los Angeles Flexlink Maintence
Update: This maintenance has been cancelled
Tonight, starting at 11:59PM, we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in Los Angeles. This window is scheduled for 2 hours. During this time, circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.
System Maintenance
Update: Maintenance complete.
Tonight at 11:59 pm we will be performing updates to a range of systems, including mail, member tools, and vpn. There will be brief interruptions to these services as systems are rebooted. We expect to conclude the maintenance within 1-2 hours.
Sacramento and Mendocino County Flexlink Maintence
Update: 2:04AM – This maintenance is now complete.
Tonight, starting at 11:59PM, we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in Sacramento and Mendocino County. This window is scheduled for 2 hours. During this time, circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.
South and East Bay Flexlink Maintence
Update: 2:04AM – This maintenance is now complete.
Tonight, starting at 11:59PM, we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in the South and East Bay. This window is scheduled for 2 hours. During this time, circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.
South and East Bay Flexlink Maintence
Update: 1:48AM – This maintenance is now complete.
Tonight, starting at 11:59PM, we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in the South and East Bay. This window is scheduled for 2 hours. During this time, circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.
– Network Operations
Cloverdale – Fusion DSL/Flexlink Intrusive Maintenance
Tonight, starting at 11:59PM, customers in the Cloverdale area will experience a brief service interruption while we perform some maintenance. Expected down time is about 15-30 minutes while the equipment reboots. The maintenance window will be 2 hours.
Update: 1:55am – This maintenance is now complete.
– Network Engineering
Palo Alto Fusion Service Issues
As of around 8:30PM this evening, we were alerted of an impacted piece of hardware serving fusion customers in Palo Alto. After some time troubleshooting, we identified the issue but even with vendor assistance were unable to remotely perform a repair. A technician is presently en route to replace the failed equipment. Update will be provided on completion of service restoration.
Update(6:06am): A subset of customers continues to be affected after physical equipment replacement. We are continuing to work with our vendor to find the root cause and fully restore service.
Update(8:24am): All services have been restored and we are working to protect ourselves from a failure of this nature in the future.
-Network Engineering
Voice Issue
A subset of our customers are experiencing call quality issues affecting outbound dialing, and some inbound toll free calls. Our engineering team is actively working the issue.
There is no ETR for repair. The next update will be for any meaningful progress reported.
Update 2:21 PM PST – We have isolated the issue to a specific carrier and to mitigate the issue all voice traffic is being routed around to a different path. Once repairs are made we will relay another update.
San Francisco Flexlink Maintenance
Update: 2:27am – This maintenance is now complete.
Tonight, starting at 11:59PM, we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in San Francisco. This window is scheduled for 2 hours. During this time, circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.
– Network Operations