Network Instability

At 12:55pm today it was brought to our attention customers on our network were experiencing intermittent reachability to parts of the internet.  We determined the source of this to be a routing issue with one of our upstream providers. At approximately 1:08 we routed traffic away from the affected upstream, thus mitigating the issue.

The upstream provider has since fixed their routing issue, but we are performing further tests on the link before returning network traffic back to it.

-Matt and Jared

Edit: Fixed incorrectly stated times in the above message. -Matt

Web Cluster Issues

Update: Our web cluster was the target of a large DDoS; the targeted host has been isolated and the cluster has been stabilized.

We are currently experiencing issues with our web server cluster. We are investigating the problem and will provide an update shortly.

-Tim, Nathan, and William

DSL Subscriber Migration

This evening, Thursday July 15, at 12:01AM, we will be migrating all DSL subscribers in the Sacramento area to a new OC-12 circuit. Affected customers may experience up to 10 minutes of downtime while the migration is performed.

-Tim, Jared, and Nathan

Update: This migration has been completed and all affected customers should be back online at this time.

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

ATM Switch Maintenance

Tonight, Wednesday June 30 between 1 AM and 2 AM, we will be performing a maintenance reload of one of our ATM switches to resolve a diagnostic bug we have found. Affected customers will experience about 5 minutes of downtime as the switch reloads onto new code.

-Jared

Update: The ATM switch reload has been completed without incident. Total downtime for affected customers was less than 10 minutes.

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

DSL Subscriber Migration

This evening, Tuesday June 29, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from one of our ATM OC-12 circuits to a less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes.

-Jared and Nathan

Update: This migration has been postponed to Wednesday June 30, at 12:01 AM.

Update: This migration has been completed successfully. All affected customers should be back online at this time.

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

DSL Subscriber Migration

This evening, Friday June 25, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from one of our ATM OC-12 circuits to less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes.

-Jared and Nathan

Update: The subscriber migration has been completed without any undue impact. All affected customers should be back online at this time.

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob