Tonight, Tuesday November 29 at 12:01 AM, we will be performing a maintenance reload on several legacy DSL aggregation routers that serve some of our customers in the Bay Area and Los Angeles areas. Affected customers will experience 5-10 minutes of downtime as the routers reboot.
-Jared
Update: The maintenance reloads have been completed and all affected customers should be back online.
This evening, our support team began tracking an issue causing poor DSL performance for legacy DSL subscribers served by a certain SMS. Upon further investigation, we found a port on our ATM switch to be failing, and causing loss for traffic going across it. We moved the SMS to a different port on the ATM switch, and the performance problem should be resolved now.
-Jared and John
Earlier this afternoon, we made some changes to the authentication service used by our VPN Concentrator, which resulted in an inability to establish new connections to the VPN Concentrator. We tracked down and fixed the configuration problem that caused this issue, and all VPN sessions should be able to connect at this time. We apologize for any inconvenience this caused.
-Jared and Jasper
At approximately 3:09PM, while performing routine cable maintenance, a telco technician accidentally pulled an incorrect fiber jumper causing a brief outage. This would have affected legacy DSL service for customers in the Sacramento area. The mistake was immediately corrected and service was restored less than a minute later. We apologize for any inconvenience this may have caused.
-Tim and Nathan
The raised floor bracing and cabinet mounting systems at our Santa Rosa datacenter are currently undergoing a seismic retrofit. The work is being handled by a contractor that specializes in raised floor environments and will not cause an interruption in any of our services. Colo customers should note that some cabinet doors must be opened in order to provide sufficient clearance to perform the work. Please do not be alarmed at an unexpected door open event during regular business hours over the next few days.
All Fusion Voice subscribers are able to access 3-1-1 services where available. Please note that 3-1-1 services are available only in a limited set of cities. For more information about 3-1-1, and the list of cities where 3-1-1 services are available, please see our wiki.
Tonight, Friday November 4 at 12:01 AM, we will be reconfiguring approximately 50 Flexlink Ethernet customers to smooth the way for future network upgrades. This work will disrupt connectivity to those customers for less than 5 minutes apiece. No work or changes will be necessary for these customers to restore service.
-Jared, Tomoc, Robbie, and Phil
Update: All customer reconfigurations are complete. No customers should have been offline for more than 5 minutes. If you are experiencing any problems with your Flexlink Ethernet service, please call the NOC Hotline at 877-706-5662.
We were notified this morning of issues dialing numbers in Northern California on our Fusion Voice service. The issue was traced to one of our upstream phone providers in this area. Calls are now routed out redundant capacity and the issue is resolved. There was no impact to 911 service.
-Nathan
Today, October 30 at approximately 5:43 PM, we began tracking an issue with one of our ATM switches that serves DSL to customers in the greater Bay Area. This issue seems to be dropping all traffic across that switch. We are working to remedy the problem as quickly as possible.
-Jared
Update: The problem appeared to be a general failure of the ATM switch’s control interface. We rebooted the ATM switch and the problem has been cleared. All affected customers should be back online at this time. We will continue to monitor the switch for further issues.
Tonight, Thursday October 26 at 11:01 PM, we will be performing a software upgrade on four core backbone routers in our CLEC network. This upgrade should not have any impact on customer traffic as we will route traffic away from these routers before the invasive work. Each router should be down for 10-15 minutes for the software upgrade.
-Jared and Nathan
Update: After some struggles with a bad RAM upgrade for one of the core routers, all work tonight has been completed. There should have been no customer impact.