Tuesday, June 30th at 12:01 am, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in the downtown Santa Rosa area. This will be a service affecting maintenance window and we expect customer downtime to be less than 15 minutes.
-Clay and Tim
Update: All work has been completed as planned. Downtime was less than 15 minutes.
This Thursday, at 12:01 am, we will be performing some intrusive maintenance on one of our DSLAM’s in San Francisco. This DSLAM serves Fusion & Flexlink ADSL2+ in the Lower Pacific Heights/ Western Addition area. This work will cause an interruption of service for about 20 minutes while we replace a suspected bad cable.
-Clay
This Tuesday, June 23 at 12:01 AM we will be performing maintenance reloads of the routers that terminate our Business-T service, and two Redbacks that terminate traditional DSL service. This will affect all Business-T customers, all Fresno DSL subscribers, and a small subset of DSL customers in the Bay Area. Expected downtime is 5 minutes.
-Jared
Update June 23 12:31 AM: The Redback reloads were completed without incident, and all DSL customers should be back online at this time. Our Los Angeles POP Business-T router locked up during the reload and required a physical reboot to come back up properly. For this reason Business-T customers in the Los Angeles area experienced approximately 15-20 minutes of downtime. Our other Business-T routers reloaded without incident. All services affected by this maintenance have been restored at this time.
One of the four heads in our redundant NFS filer clusters that handle all of our email storage crashed this morning due FCAL bus instabilities after a disk failure. It’s partner attempted to take over it’s operation but was unable to due to the specific nature of the failure. This is one of the few edge cases where all of the redundancy built into the system isn’t able to help as the only way to reestablish service is to powerfail all of the disk shelves to completely reset the FCAL buses. No email or data was lost and the systems otherwise performed as expected. During the 15 minutes while the filer was down approximately one quarter of our users would have been unabled to check their email. At this time all services have been restored.
Update Mon Jun 22 10:41:48 PDT: IMAP users who’s message stores were on the affected filer may have continued to be unable to check their mail until a few minutes ago due do clock skew between the filer and the servers.
This Tuesday, June 16 at 12:01 AM, we will be performing a software upgrade of our DSLAM that serves Fusion and Flexlink services to customers in the Downtown Santa Rosa area. Expected downtime is 10-15 minutes for the DLSAM to be rebooted onto the new software release.
-Jared
Update: The software upgrade has been completed without incident. Downtime for affected customers was less than 5 minutes.
This Wednesday, June 17 at 12:01 AM, we will be performing a maintenance reload of one of our ATM switches that serves DSL to customers in the North Bay area. Customers served by this switch will experience 5-10 minutes of downtime.
-Jared
Update: The ATM switch was rebooted at approximately 12:20 AM and is functioning normally. Total downtime for affected customers was under 10 minutes.
Our primary 800# and 707-522-1000 are currently returning a fast busy signal . We are working with our telephone system vendor to correct this as quickly as possible.
In the interim, Support is available at 707-547-3400, and we can transfer calls internally. This queue is liable to be a bit busy until this situation is corrected, and we apologize for the inconvenience.
– Eli
Update: Â Full service to our phone system has been restored, and calls to all Sonic.net phone numbers are being received.
Tonight, June 9th at 11:00 pm, we will be replacing a card in one of our routers in Santa Rosa, which provides Flexlink Long Range in the Santa Rosa area. This maintenance will cause about 10 minutes of downtime for affected customers on this card, and may require a reload of the router to complete the maintenance.
-Clay and Tim
Update: The card replacement was completed with no problems. Downtime for affected customers was less than 1 minute.
At approximately 8:20 PM today, one of the DSL aggregation routers that serve DSL customers in the Bay Area unexpectedly reloaded. This caused about 5 minutes of downtime for the customers served by this router.
-Jared
This evening, June 3rd, at 11:00PM we will be inserting a new interface card into one of our aggregation routers. This router provides FlexLink Long Range coverage for the Petaluma area. This work is not expected to cause an interruption of service, but a router reload may be required to complete this operation. If a reload is required, customers served by the affected router will experience 5-10 minutes of service interruption.
-Clay