The raised floor bracing and cabinet mounting systems at our Santa Rosa datacenter are currently undergoing a seismic retrofit. The work is being handled by a contractor that specializes in raised floor environments and will not cause an interruption in any of our services. Colo customers should note that some cabinet doors must be opened in order to provide sufficient clearance to perform the work. Please do not be alarmed at an unexpected door open event during regular business hours over the next few days.
Fusion Voice 3-1-1 Service
All Fusion Voice subscribers are able to access 3-1-1 services where available. Please note that 3-1-1 services are available only in a limited set of cities. For more information about 3-1-1, and the list of cities where 3-1-1 services are available, please see our wiki.
Flexlink Ethernet Customer Reconfiguration
Tonight, Friday November 4 at 12:01 AM, we will be reconfiguring approximately 50 Flexlink Ethernet customers to smooth the way for future network upgrades. This work will disrupt connectivity to those customers for less than 5 minutes apiece. No work or changes will be necessary for these customers to restore service.
-Jared, Tomoc, Robbie, and Phil
Update: All customer reconfigurations are complete. No customers should have been offline for more than 5 minutes. If you are experiencing any problems with your Flexlink Ethernet service, please call the NOC Hotline at 877-706-5662.
Northern California Calling Issues
We were notified this morning of issues dialing numbers in Northern California on our Fusion Voice service. The issue was traced to one of our upstream phone providers in this area. Calls are now routed out redundant capacity and the issue is resolved. There was no impact to 911 service.
-Nathan
Legacy ATM Network Outage
Today, October 30 at approximately 5:43 PM, we began tracking an issue with one of our ATM switches that serves DSL to customers in the greater Bay Area. This issue seems to be dropping all traffic across that switch. We are working to remedy the problem as quickly as possible.
-Jared
Update: The problem appeared to be a general failure of the ATM switch’s control interface. We rebooted the ATM switch and the problem has been cleared. All affected customers should be back online at this time. We will continue to monitor the switch for further issues.
Telecom Backbone Maintenance part 2
Tonight, Thursday October 26 at 11:01 PM, we will be performing a software upgrade on four core backbone routers in our CLEC network. This upgrade should not have any impact on customer traffic as we will route traffic away from these routers before the invasive work. Each router should be down for 10-15 minutes for the software upgrade.
-Jared and Nathan
Update: After some struggles with a bad RAM upgrade for one of the core routers, all work tonight has been completed. There should have been no customer impact.
Telecom Backbone Maintenance
Tonight, Thursday October 26 at 12:01 AM, we will be performing a software upgrade on three of our core backbone routers in our CLEC network. This upgrade should not have any impact on customer traffic as we will route traffic away from these routers before the invasive work. Each router should be down for 5-10 minutes for the software upgrade.
-Jared
Update: The software upgrade of the three core routers is completed. There should have been no impact to customer traffic.
Wireless Backbone Issues
Tonight at approximately 6:35 one of the backbone links in our wireless network failed, causing poor performance and connectivity for most of our wireless Internet customers. We have a technician en-route now to investigate the problem and will report as we make progress.
-Jared and Adam
Update: As of 7:35, we have restored service. One of the radio transmitters failed and had to be rebooted. We will be scheduling a replacement of this unit in the near future.
Invasive Telecom Network Maintenance
This Saturday, October 22 at 12:01 AM, we will be performing a software upgrade on one of our customer-serving routers in Santa Rosa. This will cause a 5-10 minute interruption of Internet and phone service for customers served by this router.
-Jared and Nathan
Partial CRAC Failure in Santa Rosa
We’ve had a major failure in our redundant CRAC (Computer Room AC) that handles cooling for our Santa Rosa Data Center a Colocation Facilities. The redundant systems have taken over and we believe that the situation is under control. Our vendors are en route, more information forthcoming.
Update: 14:15:14 PDT. The redundant system is functioning properly and the room temperature has nearly stabilized. We believe the failure was caused by a refrigerant leak due to the failure of a pressure release safety valve. The faulty valve has been isolated and technicians are en route with refrigerant to recharge the system.
Update: 17:45:30 PDT. The failed system has been recharged (with over 800 lbs of refrigerant) and restored to service and appears to be functioning correctly.