Friday, July 17th, between 12:01 and 12:45, we will be replacing DSL cards in Rohnert Park and Santa Rosa. These cards serve Flexlink and Fusion DSL in those areas. There will be a short amount of downtime associated with each card replacement. Expected downtime per card is about one minute.
-Clay
We’ve just completed replacing all of the older dual dual-core xeon servers that make up our load balanced webcluster with new dual quad-core xeon servers with 16GB of ram each. The new servers are substantially faster than the old servers so some users may notice that their websites load faster and that their hosted applications are more responsive.
Tonight, at Midnight, we will be performing an emergency router reload in Petaluma. This router serves Flexlink Long Range in the Petaluma area. Downtime is expected to be 5 to 10 minutes while the router is reloading.
-Clay and Tomoc
Tuesday, July 7th, at 11:00pm, we will be performing some intrusive maintenance on equipment serving Flexlink in Petaluma. This will be a service affecting maintenance window and we expect customer downtime to be less than 15 minutes.
-Clay and Tomoc
A transient issue on one of our customer MySQL servers that went unnoticed by our monitoring systems caused processes to lock and stack on our web cluster. Sites hosted on our web cluster, including www.sonic.net, may have been slow or timed out while loading until the issue was properly identified and resolved. We’ve already corrected the error in monitoring the MySQL server and will take steps to ensure that it doesn’t happen again. -Kelsey and William
Tuesday, June 30th at 12:01 am, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in the downtown Santa Rosa area. This will be a service affecting maintenance window and we expect customer downtime to be less than 15 minutes.
-Clay and Tim
Update: All work has been completed as planned. Downtime was less than 15 minutes.
One of the four heads in our redundant NFS filer clusters that handle all of our email storage crashed this morning due FCAL bus instabilities after a disk failure. It’s partner attempted to take over it’s operation but was unable to due to the specific nature of the failure. This is one of the few edge cases where all of the redundancy built into the system isn’t able to help as the only way to reestablish service is to powerfail all of the disk shelves to completely reset the FCAL buses. No email or data was lost and the systems otherwise performed as expected. During the 15 minutes while the filer was down approximately one quarter of our users would have been unabled to check their email. At this time all services have been restored.
Update Mon Jun 22 10:41:48 PDT: IMAP users who’s message stores were on the affected filer may have continued to be unable to check their mail until a few minutes ago due do clock skew between the filer and the servers.
Our primary 800# and 707-522-1000 are currently returning a fast busy signal . We are working with our telephone system vendor to correct this as quickly as possible.
In the interim, Support is available at 707-547-3400, and we can transfer calls internally. This queue is liable to be a bit busy until this situation is corrected, and we apologize for the inconvenience.
– Eli
Update: Full service to our phone system has been restored, and calls to all Sonic.net phone numbers are being received.
Tonight, June 9th at 11:00 pm, we will be replacing a card in one of our routers in Santa Rosa, which provides Flexlink Long Range in the Santa Rosa area. This maintenance will cause about 10 minutes of downtime for affected customers on this card, and may require a reload of the router to complete the maintenance.
-Clay and Tim
Update: The card replacement was completed with no problems. Downtime for affected customers was less than 1 minute.
At approximately 8:20 PM today, one of the DSL aggregation routers that serve DSL customers in the Bay Area unexpectedly reloaded. This caused about 5 minutes of downtime for the customers served by this router.
-Jared