Month: October 2011

Today, October 30 at approximately 5:43 PM, we began tracking an issue with one of our ATM switches that serves DSL to customers in the greater Bay Area. This issue seems to be dropping all traffic across that switch. We are working to remedy the problem as quickly as possible.

-Jared

Update: The problem appeared to be a general failure of the ATM switch’s control interface. We rebooted the ATM switch and the problem has been cleared. All affected customers should be back online at this time. We will continue to monitor the switch for further issues.

Tonight, Thursday October 26 at 11:01 PM, we will be performing a software upgrade on four core backbone routers in our CLEC network. This upgrade should not have any impact on customer traffic as we will route traffic away from these routers before the invasive work. Each router should be down for 10-15 minutes for the software upgrade.

-Jared and Nathan

Update: After some struggles with a bad RAM upgrade for one of the core routers, all work tonight has been completed. There should have been no customer impact.

Tonight, Thursday October 26 at 12:01 AM, we will be performing a software upgrade on three of our core backbone routers in our CLEC network. This upgrade should not have any impact on customer traffic as we will route traffic away from these routers before the invasive work. Each router should be down for 5-10 minutes for the software upgrade.

-Jared

Update: The software upgrade of the three core routers is completed. There should have been no impact to customer traffic.

Tonight at approximately 6:35 one of the backbone links in our wireless network failed, causing poor performance and connectivity for most of our wireless Internet customers. We have a technician en-route now to investigate the problem and will report as we make progress.

-Jared and Adam

Update: As of 7:35, we have restored service. One of the radio transmitters failed and had to be rebooted. We will be scheduling a replacement of this unit in the near future.

This Saturday, October 22 at 12:01 AM, we will be performing a software upgrade on one of our customer-serving routers in Santa Rosa. This will cause a 5-10 minute interruption of Internet and phone service for customers served by this router.

-Jared and Nathan

We’ve had a major failure in our redundant CRAC (Computer Room AC) that handles cooling for our Santa Rosa Data Center a Colocation Facilities.  The redundant systems have taken over and we believe that the situation is under control.  Our vendors are en route, more information forthcoming.

Update: 14:15:14 PDT.  The redundant system is functioning properly and the room temperature has nearly stabilized.  We believe the failure was caused by a refrigerant leak due to the failure of a pressure release safety valve.  The faulty valve has been isolated and technicians are en route with refrigerant to recharge the system.

Update: 17:45:30 PDT.  The failed system has been recharged (with over 800 lbs of refrigerant) and restored to service and appears to be functioning correctly.

This Friday, October 21 at 12:01 AM, we will be migrating all of our legacy DSL subscribers in the Chico area to a new, higher-capacity circuit. This migration will take 20-30 minutes, and affected customers will lose Internet connectivity while the migration is being done.

-Jared and Nathan

Update: The migration has been completed. Due to a DHCP server misconfiguration, dynamic IP DSL customers may have experienced instability in their Internet connection until about 1AM. At this time all migrated customers should be up and running with no problems.

This Thursday October 20 at 12:01 AM we will be performing an equipment upgrade on our backbone at our San Jose POP. We will be routing customer traffic away from the point of maintenance, so there should be no customer impact.

-Jared, Nathan and Matt

Tonight, Wednesday Oct 12 at 12:01 AM we will be performing maintenance on one of our core backbone routers in the telecom network. This maintenance should take 10-15 minutes, and no there should be no customer impact.

-Jared

This maintenance is now completed. The core router is back in service, and there should have been no interruption of service.