Month: June 2012

Fusion Outage in San Francisco

This evening during routine network maintenance, around 5:10 PM,  a configuration error caused one of our DSLAMs in San Francisco to destabilize.  This disrupted service for all Fusion customers on that particular DSLAM. Service has been restored to all affected customers as of  around 5:50 PM. If you are still having any trouble with your Fusion service please contact our support department as usual.

– Robbie and the NOC

Coming soon: New Fusion equipment!

I’m very excited to announce that on July 1st, we will be launching a new equipment program for our residential Fusion Broadband+Phone service. The new equipment is a super-fast ASDL2+ modem/router with four Fast Ethernet ports, high-powered WiFi “N” and an easy-to-use web interface. Built-in Firewall protection for your home network is also included.

Our new Fusion equipment will also streamline our support systems, and allow us to upgrade firmware, manage configuration and roll out new features. For example, in future this capability will be used to offer IPv6 to customers.

The new equipment will be owned by and rented to customers as part of the Fusion home service, for $6.50 per month. This assures you’ll always have working equipment – we can remotely troubleshoot it, and replace it for you at no charge if the equipment fails.

Existing customers are of course welcome to continue to use the equipment that they already own, or you will be able to upgrade (starting next week) to new rental equipment if you find the features compelling.

Webcluster Service Interuption

A number of hosted websites (including were offline for a approximately seven minutes this afternoon.  We’re investigating the initial cause that caused the servers to drop out but believe a known bug in our load balancer configuration was also triggered that prevented all of the sites from being automatically brought back online.

Update: Thu Jun 28 10:16:44 PDT 2012.  We’ve seen a couple more events since this happened yesterday and we believe we have identified the DoS attack and are working to mitigate its affects now.

Update: Thu Jun 28 13:13:41 PDT 2012.  We’ve installed a custom apache module that should eliminate or, at least, reduce the affects of the DoS attack on our web cluster and are hopeful that this will prevent this issue from causing problems again.

-Kelsey, William and Kevan


Modem support and assistance

In the near future we will be integrating customer premise modems into a centralized configuration platform, or ACS, using TR-069. This will allow support staff to view and update some modem settings, to view connection statistics, and for some equipment, to push firmware updates.

If you have ZTE, Motorola or Pace/2Wire equipment on your connection, it may be included in our remote configuration and assistance platform.

To learn more about the TR-069 standard, see Wikipedia:

DSL Aggregation Router Reload

This Friday, June 22 at 12:01 AM we will be performing maintenance reloads of two Redbacks that terminate traditional DSL service. This will affect some of our Bay Area and LA DSL subscribers. Expected downtime is 5 minutes.
Update: Reloads complete. Estimated down time was under 5 minutes.


Emergency Router Reload

At 7:33am, a protocol issue between two redundant routers caused network instability which required an emergency reboot. This caused intermittent connectivity issues to our DNS servers as well as a subset of customer connections. Service has been restored as of 8:14am, and we will be working with our vendors to ensure this does not happen again. NOC Company Picnic

Support will be closed from 11am-6pm on 6/9/2012. will be celebrating its 18th Birthday on Saturday, June 9th.  In order to allow all employees to attend the celebratory company picnic support will be closed from 11am until 6pm.  We apologize for any inconvenience this may cause.

Emergency Legacy DSL Maintenance

Tonight, at 12:01AM, we will be performing invasive maintenance on an ATM switching device that serves a subset of legacy DSL customers in the Bay Area as well as Chico. Downtime for affected customers should be less than 10 minutes.

-Tim, Nathan, and the NOC

Update: The ATM switch in question suffered a redundancy fail-over before the maintenance work could be performed this evening. Affected customers should have experienced about 5 minutes of downtime while the fail-over occurred. We are currently investigating the cause of the issue in order to prevent further recurrences.