This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers in one portion of San Francisco. Affected customers should only experience 5 minutes of downtime.
-Jared
Month: February 2011
Emergency CRAC Maintenance
The primary compressor on the B-side of our Core4 CRAC (Computer Room Air Conditioning) system that cools our Santa Rosa datacenter is being taken out of service and electrically swapped with it’s backup compressor. The primary compressor experienced loss of oil pressure and while it is apparently still functional it has clearly suffered significant internal damage. Once the primary compressor has been replaced, the backup compressor will be returned to its normal standby operation.
The Core4 CRAC system is 2N redundant so we can loose the entire A or B side of the system and still satisfy the cooling requirements of our datacenter. Additionally, each primary compressor on each side is paired with a backup compressor so should this damaged compressor have actually ceased to function, the backup compressor would have taken over automatically to meet the cooling demand.
The swap was completed in the time it took to draft this notice. Both sides of the CRAC are currently fully operational.
ATM Customer Aggregation Router Reload
This Thursday, February 24 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.
-Jared
Emergency Router Maintenence
At 9:40AM this morning we will be performing an emergency router reload on one of our ATM customer aggregation routers. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload.
-Tim and the NOC
Santa Rosa ADSL1 outage
One of the remote terminals serving a large number of customers in Santa Rosa is currently off line resulting in a sync no surf situation. AT&T reports that they have a technician dispatched to repair but we do not have an estimated time of repair.
CLEC Intrusive Maintenance
This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers in Windsor. Affected customers may experience up to 30 minutes of downtime while the work is performed.
-Nathan, Jared, and Tim
Colocation UPS Maintenance
This Friday, starting at 9AM, one of the three UPSes that serves our Santa Rosa datacenter will be undergoing its annual preventative maintenance. The maintenance is fully scripted with our vendor and no interruption of services are expected. During the maintenance window our standby genset will be kept running should our PG&E service fail. -Kelsey and Russ
Update: Fri Feb 18 19:49:15 PST 2011, the schedule maintenance has been completed with some complication. One of of the bypass contactors jammed preventing the UPS from going back online. While the contactor eventually actuated and the UPS was brought online, it’s replacement will be scheduled as soon as possible. -Kelsey and Russ.
LATA1 ADSL1 Outage
Tonight shortly before 7:30pm we lost connectivity on one of the backhaul circuits serving ADSL1 customers in the Bay Area and North Coast. This has resulted in many customers experiencing a sync-no-surf outage, where the DSL modem indicates it has a connection to the local DSLAM, but no traffic passes through. Â Out network engineers have been working on the problem since immediately after it happened, but we have no ETR yet. Fusion, wireless, and dialup service, as well as ADSL1 service outside the Bay Area and North Coast are unaffected. Â — John F and the NOC staff
Update: Our network engineers have been working counterparts at AT&T and have narrowed the problem down to what appears to be a faulty switch on their end. Still no ETR. — John F and the NOC staff
Update: As of 11:03 PM tonight, the ATM backhaul circuit was restored to service by ATT. All affected customers should be back online at this time. If you are still experiencing a problem with your DSL connection, please reboot your DSL modem.
-Jared