Tonight, Tuesday, Frebruary 28 at 12:01 AM we will be performing some potentially intrusive network configuration changes to our Telecom network in the Sacramento area. We do not expect any downtime, but there may be some brief outage of network connectivity for Fusion and Flexlink customers in the Sacramento area.
-Jared
Tonight, February 10th at 12:01 AM, the shared customer database servers will undergo routine maintenance which requires a reboot of the servers. Expected downtime for customer websites that use the shared database service is less than 10 minutes. This work is in addition to the internal database server upgrades scheduled tonight, but is unrelated.
-Kevan
Update: All upgrades were completed without event and customer database services are working correctly.
Tonight, after Support closes, we’re going to begin an invasive upgrade on our primary internal database servers. The upgrade necessitates a complete dump and restore to new hardware and is expected to take several hours to complete. The new servers are substantially more powerful and many of our internal systems should see a marked performance improvement. While the data is being migrated and imported, all signup and member tools will be unavailable but all other services will otherwise remain unaffected. — Kelsey and William
Update: The upgrade appears to have gone according to plan and all internal systems are back online at this time.
Tonight, February 5 at approximately 12:25 AM, one of the ATM switches that serves legacy DSL to the greater bay and Chico areas suffered a card software failure. The switch was rebooted to clear the problem, and all affected customers should be back online. We will be investigating this issue with the equipment vendor.
-Jared
We will be performing immediate emergency maintenance on equipment serving a number of legacy DSL customers in the LATA1 area. This will impact affected customers for 5 to 10 minutes while the maintenance is performed.
-Tim and Jared
Update: The source of the problem was a bad port on a single DSL aggregation router in our ATM network. We have moved the affected connection to a new port and re-routed all traffic to the new port. Affected customers should be back online at this time.
This Thursday, starting at 9AM, one of the three UPSes that serves our Santa Rosa datacenter will have all of it’s batteries replaced. This is not a service impacting event and the UPS will be online for the duration of work. If necessary, the battery replacement may continue Friday morning. -Kelsey and Russ
At 5:32 AM today, one of our core routers in our Santa Rosa datacenter unexpectedly restarted its routing processes. This caused 5-10 minutes of impacted connectivity to our datacenter as the network recovered from the event. This would have affected access to mail, Sonic.net hosted websites, and colocated servers.
Ironically, we had already scheduled retirement and replacement of our core routers in the Santa Rosa datacenter for this upcoming week. We will accelerate our plans due to this event, and apologize for any impact it may have caused.
-Jared
Today we have updated our member tool authentication code. This is a transparent change in our back end code and does not require any end user changes. We believe this will resolves an intermittent bug that was causing users to occasionally have to re-authenticate their session and also results in a slight performance increase while using the tools.
-William
This Wednesday, December 28 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.
-Jared
The server that handles ssl.sonic.net (our legacy shared https web hosting server) suffered a double disk raid failure (in a mirror) this afternoon and was offline for approximately 23 minutes while we restored to replacement disks from our backups. -Kelsey and William