This Friday morning, at 12:00AM on July 20th, we will be performing intrusive maintenance on DSL aggregation equipment serving customers in the LA area. Expected downtime is less than 15 minutes.
– Robbie and Tomoc
Network Maintenance
This coming Monday, July 16th, from 9:00PM to 3:00AM, we will be performing major intrusive work within one of our core points of presence in San Jose. We do not expect any service impact to our customers as a result of this operation. Fusion voice users may notice a very short delay in their calls when initially dialing out during a portion of this window.
– NOC
Update: Maintenance completed without a hitch, customers should not have experienced any service interruption.
Sonic.net Router Maintenance
Tonight, July 8th, at 12:00am, we will be performing maintenance on routers that aggregate Business-T and Frame-Relay connections. The procedure will require a reboot of the routers. Expected downtime is less than 15 minutes.
– Robbie and Tomoc
Update: As some of you may have noticed, 12:00am tonight is actually July 6th. I apologize for any confusion this may have caused.
Update 2: The router maintenance has been completed and all affected customers are now be back online. Total downtime was about 3 minutes.
Sonic Telecom Emergency Backbone Maintenance
Due to a hardware failure in the Sonic Telecom backbone we will need to perform emergency router maintenance over the next hour. We do not believe this work will have any impact on customer traffic but there is the potential for a brief service interruption. Further updates will follow shortly.
-Tim and Nathan
Update: Maintenance work has been completed. There should have been no impact to customer traffic. This evening, from 12:00AM until 1:00AM, we will be performing further maintenance of a similar nature. Again we do not expect any impact to customer traffic but there is a possibility of brief service interruptions during this time.
Fusion Outage in San Francisco
This evening during routine network maintenance, around 5:10 PM, a configuration error caused one of our DSLAMs in San Francisco to destabilize. This disrupted service for all Fusion customers on that particular DSLAM. Service has been restored to all affected customers as of around 5:50 PM. If you are still having any trouble with your Fusion service please contact our support department as usual.
– Robbie and the NOC
Coming soon: New Fusion equipment!
I’m very excited to announce that on July 1st, we will be launching a new equipment program for our residential Fusion Broadband+Phone service. The new equipment is a super-fast ASDL2+ modem/router with four Fast Ethernet ports, high-powered WiFi “N” and an easy-to-use web interface. Built-in Firewall protection for your home network is also included.
Our new Fusion equipment will also streamline our support systems, and allow us to upgrade firmware, manage configuration and roll out new features. For example, in future this capability will be used to offer IPv6 to customers.
The new equipment will be owned by Sonic.net and rented to customers as part of the Fusion home service, for $6.50 per month. This assures you’ll always have working equipment – we can remotely troubleshoot it, and replace it for you at no charge if the equipment fails.
Existing customers are of course welcome to continue to use the equipment that they already own, or you will be able to upgrade (starting next week) to new rental equipment if you find the features compelling.
Webcluster Service Interuption
A number of hosted websites (including www.sonic.net) were offline for a approximately seven minutes this afternoon. We’re investigating the initial cause that caused the servers to drop out but believe a known bug in our load balancer configuration was also triggered that prevented all of the sites from being automatically brought back online.
Update: Thu Jun 28 10:16:44 PDT 2012. We’ve seen a couple more events since this happened yesterday and we believe we have identified the DoS attack and are working to mitigate its affects now.
Update: Thu Jun 28 13:13:41 PDT 2012. We’ve installed a custom apache module that should eliminate or, at least, reduce the affects of the DoS attack on our web cluster and are hopeful that this will prevent this issue from causing problems again.
-Kelsey, William and Kevan
Modem support and assistance
In the near future we will be integrating customer premise modems into a centralized configuration platform, or ACS, using TR-069. This will allow support staff to view and update some modem settings, to view connection statistics, and for some equipment, to push firmware updates.
If you have ZTE, Motorola or Pace/2Wire equipment on your Sonic.net connection, it may be included in our remote configuration and assistance platform.
To learn more about the TR-069 standard, see Wikipedia: http://en.wikipedia.org/wiki/TR-069
DSL Aggregation Router Reload
This Friday, June 22 at 12:01 AM we will be performing maintenance reloads of two Redbacks that terminate traditional DSL service. This will affect some of our Bay Area and LA DSL subscribers. Expected downtime is 5 minutes.
Update: Reloads complete. Estimated down time was under 5 minutes.
-Tomoc
Emergency Router Reload
At 7:33am, a protocol issue between two redundant routers caused network instability which required an emergency reboot. This caused intermittent connectivity issues to our DNS servers as well as a subset of customer connections. Service has been restored as of 8:14am, and we will be working with our vendors to ensure this does not happen again.
-Sonic.net NOC