This evening, Tuesday July 20th at 11PM we will be replacing the failed blade in our core switch. There should be no effect aside from a brief hit in accessibility to news.sonic.net. Any impact should be less than 10 minutes.
-Jared
Update: The card replacement was completed shortly after 11PM, and the core switch was restored to full service at approximately 11:15PM and is functioning normally at this time.
10:28 news.sonic.net is back on-line — we apologize for the inconvenience.
The Sonic.net News cluster (news.sonic.net) suffered a hardware problem and is currently being repaired; we will update you as changes occur.
Python has been upgraded to version 2.6 on our webcluster and shell server.
Access the new version by calling python2.6 instead of python.
MySQLdb has also been installed along with this upgrade.
-William
At 12:55pm today it was brought to our attention customers on our network were experiencing intermittent reachability to parts of the internet. We determined the source of this to be a routing issue with one of our upstream providers. At approximately 1:08 we routed traffic away from the affected upstream, thus mitigating the issue.
The upstream provider has since fixed their routing issue, but we are performing further tests on the link before returning network traffic back to it.
-Matt and Jared
Edit: Fixed incorrectly stated times in the above message. -Matt
Update: Our web cluster was the target of a large DDoS; the targeted host has been isolated and the cluster has been stabilized.
We are currently experiencing issues with our web server cluster. We are investigating the problem and will provide an update shortly.
-Tim, Nathan, and William
This evening, Thursday July 15, at 12:01AM, we will be migrating all DSL subscribers in the Sacramento area to a new OC-12 circuit. Affected customers may experience up to 10 minutes of downtime while the migration is performed.
-Tim, Jared, and Nathan
Update: This migration has been completed and all affected customers should be back online at this time.
This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.
-Tim, Nathan, and Jacob
Tonight, Wednesday June 30 between 1 AM and 2 AM, we will be performing a maintenance reload of one of our ATM switches to resolve a diagnostic bug we have found. Affected customers will experience about 5 minutes of downtime as the switch reloads onto new code.
-Jared
Update: The ATM switch reload has been completed without incident. Total downtime for affected customers was less than 10 minutes.
This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.
-Tim, Nathan, and Jacob
This evening, Tuesday June 29, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from one of our ATM OC-12 circuits to a less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes.
-Jared and Nathan
Update: This migration has been postponed to Wednesday June 30, at 12:01 AM.
Update: This migration has been completed successfully. All affected customers should be back online at this time.