Tonight, at Midnight, we will be performing an emergency router reload in Petaluma. This router serves Flexlink Long Range in the Petaluma area. Downtime is expected to be 5 to 10 minutes while the router is reloading.
-Clay and Tomoc
Author: admin
CLEC Intrusive Maintenance
Tuesday, July 7th, at 11:00pm, we will be performing some intrusive maintenance on equipment serving Flexlink in Petaluma. This will be a service affecting maintenance window and we expect customer downtime to be less than 15 minutes.
-Clay and Tomoc
Scheduled SQL Maintenance
Tonight starting at 10:00PM, customer and internal SQL servers will undergo a maintenance period that may cause a very brief service interruption. This period is estimated to only last 2-5 minutes at most. -Don
Update: All systems are again running normally. There was an additional interruption around 11:00PM which was caused by a problem on one of our internal SQL servers. Webmail connectivity may have been intermittent during this time.
Web and MySQL Cluster Service Interuption
A transient issue on one of our customer MySQL servers that went unnoticed by our monitoring systems caused processes to lock and stack on our web cluster. Sites hosted on our web cluster, including www.sonic.net, may have been slow or timed out while loading until the issue was properly identified and resolved. We’ve already corrected the error in monitoring the MySQL server and will take steps to ensure that it doesn’t happen again. -Kelsey and William
CLEC Intrusive Maintenance
Tuesday, June 30th at 12:01 am, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in the downtown Santa Rosa area. This will be a service affecting maintenance window and we expect customer downtime to be less than 15 minutes.
-Clay and Tim
Update: All work has been completed as planned. Downtime was less than 15 minutes.
CLEC DSLAM maintenance
This Thursday, at 12:01 am, we will be performing some intrusive maintenance on one of our DSLAM’s in San Francisco. This DSLAM serves Fusion & Flexlink ADSL2+ in the Lower Pacific Heights/ Western Addition area. This work will cause an interruption of service for about 20 minutes while we replace a suspected bad cable.
-Clay
Router Maintenance Reloads
This Tuesday, June 23 at 12:01 AM we will be performing maintenance reloads of the routers that terminate our Business-T service, and two Redbacks that terminate traditional DSL service. This will affect all Business-T customers, all Fresno DSL subscribers, and a small subset of DSL customers in the Bay Area. Expected downtime is 5 minutes.
-Jared
Update June 23 12:31 AM: The Redback reloads were completed without incident, and all DSL customers should be back online at this time. Our Los Angeles POP Business-T router locked up during the reload and required a physical reboot to come back up properly. For this reason Business-T customers in the Los Angeles area experienced approximately 15-20 minutes of downtime. Our other Business-T routers reloaded without incident. All services affected by this maintenance have been restored at this time.
Mail Cluster Service Interuption
One of the four heads in our redundant NFS filer clusters that handle all of our email storage crashed this morning due FCAL bus instabilities after a disk failure. It’s partner attempted to take over it’s operation but was unable to due to the specific nature of the failure. This is one of the few edge cases where all of the redundancy built into the system isn’t able to help as the only way to reestablish service is to powerfail all of the disk shelves to completely reset the FCAL buses. No email or data was lost and the systems otherwise performed as expected. During the 15 minutes while the filer was down approximately one quarter of our users would have been unabled to check their email. At this time all services have been restored.
Update Mon Jun 22 10:41:48 PDT: IMAP users who’s message stores were on the affected filer may have continued to be unable to check their mail until a few minutes ago due do clock skew between the filer and the servers.
CLEC DSLAM Maintenance
This Tuesday, June 16 at 12:01 AM, we will be performing a software upgrade of our DSLAM that serves Fusion and Flexlink services to customers in the Downtown Santa Rosa area. Expected downtime is 10-15 minutes for the DLSAM to be rebooted onto the new software release.
-Jared
Update: The software upgrade has been completed without incident. Downtime for affected customers was less than 5 minutes.
ATM Switch Maintenance
This Wednesday, June 17 at 12:01 AM, we will be performing a maintenance reload of one of our ATM switches that serves DSL to customers in the North Bay area. Customers served by this switch will experience 5-10 minutes of downtime.
-Jared
Update: The ATM switch was rebooted at approximately 12:20 AM and is functioning normally. Total downtime for affected customers was under 10 minutes.