Month: June 2010

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

Tonight, Wednesday June 30 between 1 AM and 2 AM, we will be performing a maintenance reload of one of our ATM switches to resolve a diagnostic bug we have found. Affected customers will experience about 5 minutes of downtime as the switch reloads onto new code.

-Jared

Update: The ATM switch reload has been completed without incident. Total downtime for affected customers was less than 10 minutes.

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

This evening, Tuesday June 29, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from one of our ATM OC-12 circuits to a less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes.

-Jared and Nathan

Update: This migration has been postponed to Wednesday June 30, at 12:01 AM.

Update: This migration has been completed successfully. All affected customers should be back online at this time.

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

This evening, Friday June 25, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from one of our ATM OC-12 circuits to less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes.

-Jared and Nathan

Update: The subscriber migration has been completed without any undue impact. All affected customers should be back online at this time.

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

Since the cluster failure last month we had been running our rercursive name servers in a sub optimal (but still redundant) configuration.  Last Friday, we completed the re-deployment and once again have a multiple layers of redundancy with both our primary and secondary recursive name servers announced into our network in Santa Rosa, San Francisco and Los Angeles.  The old Alteon hardware load balancers that used to handle 208.201.224.11 in Santa Rosa and San Francisco have been replaced with Linux LVS/Quagga BGP servers that do not suffer from the same limitations that caused the original cluster failure.  While each of the real servers could handle many times our total DNS load on its own, neither of the Alteons that were responsible for directing traffic for 208.201.224.11 could handle it after some routine network configuration shifted traffic inside of our network.  Apart from a small hiccup due to a BGP misconfiguration Friday morning, the new clusters have performed perfectly. –Kelsey, Nathan and Jared

This Tuesday, June 22, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers in Sausalito, San Mateo, San Jose, and Millbrae. Affected customers should expect less than 30 minutes of downtime.

-Tim, Nathan, and Jacob