Month: May 2009

Customer Router Hardware Maintenance

This evening, May 28 at 11:00 pm, we will be inserting 3 new interface cards into one of our aggregation routers. This router provides Flexlink coverage for the Sebastopol area. This work is not expected to cause an interruption of service, but a router reload may be required to complete this operation. If a reload is required, customers served by the affected router will experience 5-10 minutes of service interruption.

-Clay and Jared

Broadlink Wireless outage

Broadlink Wireless outage: About 80 customers on Broadlink’s wireless network are experiencing an outage currently. services are unaffected. Broadlink is aware of the situation, however, there is no estimated time of repair.
Affected customers are welcome to call Support if they need assistance setting up a dial-up connection in the interim.
– Eli, Support

Update: Faulty hardware on the affected tower has been replaced, and service has been restored.

CLEC DSLAM Maintenance

This Wednesday, May 27 at 12:01 AM, we will be replacing a card in our DSLAM at SNFCCA12, which serves Fusion to customers in northern San Francisco. This maintenance will cause about 10 minutes of downtime for affected customers while the card is replaced.

-Jared and Clay

Update: The card replacement was completed with no problems. Service should be fully restored. If your Fusion service is experiencing any problems, please reboot your DSL modem and any routers or PCs connected to it to reset any potentially lingering DHCP information.

Brief News Server Outage

One of the switches that supports our diablo news server cluster stoped forwarding traffic until it was reset.  This is the second time the switch has failed recently so it has been slated for replacement shortly.  While the switch was offline the loadbalanced virtual server was unavailable but all NNTP traffic was still flowing inside the network.  -Kelsey and Augie

Unexpected Brief Web Resolution Interruption

Beginning at 11:51 AM we began experiencing problems with queries to and some of our name based hosted customers. During this time, customers may have experienced intermittent connection errors. We have determined the problem to be a hardware problem with one of our load balancers. Because of the nature and obscurity of the problem, the period in which there was delay or question refusals is estimated at 20-24 minutes. We apologize for this unexpected downtime. Service has since been restored to full normal operation. –Don, Augie, and the SOC

Disk quota usage error

This morning a number of notifications regarding disk quota usage being exceeded went out to customers in error. The usage mentioned in the messages was twice the actual usage. You can check your real disk quota usage using our Current disk usage reporting tool.

Our apologies for any confusion these messages caused.

Support closing early this Friday night

Our technical support department will be closing at 9pm this Friday instead of the usual 11pm time.  We are sorry for any inconvenience this may cause.  The 24 hour network operations center will as always remain available for business dedicated and colocation customers. Support will re-open as usual on Saturday at 8am.  -Dane

ATM Switch Maintenance

This Friday May 8 at 12:01 AM, we will be performing a card swap on one of our ATM switches. This card swap will expand our ATM aggregation capacity. A subset Business-T and FRATM customers in the North Bay area will experience 5-10 minutes of downtime as the card that serves them is replaced.

-Jared and Matt

Update: The card swap went smoothly with no unexpected issues. Total downtime for affected customers was less than 1 minute.

Large DDoS Attack

At 7:17 PM today, a large distributed denial of service attack was aimed at one of our customers. The DDoS attack reached over 1 million pps at our network edge. The primary impact of the DDoS was to destabilize the FRATM/Business-T router that it was aimed at, though all users may have noticed some sluggishness while browsing, due to the impact it had on some of our upstream transit links. The DDoS was blocked at our edge within 10 minutes, and its impact has been mitigated.