Month: December 2009

DSL Aggregation Router Lockup

Tonight at approximately 1:20 AM, one of our DSL aggregation routers that serves the greater Bay Area had it’s network uplink interface lock up. This problem was resolved by reloading the affected router. At this time, the router is back up and fully operational, and all affected customers should be back online. Total downtime was approximately 15 minutes for affected customers

-Jared

Infrastructure maintenance

Wednesday (09, Dec.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.

-Augie, Aaron

Los Angeles Area Outage

We are currently experiencing an outage affecting our Los Angeles area aggregation equipment. This is affecting all DSL and ATM-based Business-class services. We are working on restoring service and will provide an update shortly.

-Jared and Tim

Update: The outage was caused by a crashed management card in our core ATM switch in Los Angeles. The ATM switch does have a redundant management card, but that card did not load its configuration properly. The ATM switch had to be completely power failed to restore it to service. At this time the switch is up and functioning properly, and all connectivity should be restored. We have also isolated and fixed what caused the backup management card to not load its configuration properly.

Sonic.net Office Phone System Outage

This morning our phone lines in to the Sonic.net office are reporting all circuits busy.  Due to this fact our support, sales, and billing departments are currently unreachable by telephone.  We have reported this outage to our vendor who has found a problem between their network and AT&T. They have dispatched a technician to fix the problem, but have not yet provided us with an estimated time of repair.

This outage affects inbound and outbound calls through our office phone system. No other services are affected by this outage.

Thank you for your patience while we work with our vendor to resolve this issue.

Update — Our phone provider is working on the issue which has resulted in intermittent availability.  Customers that have gotten through to support have been dropped at times due to the work being performed.  We appologize for the inconvenience, and appreciate your patience while we get this issue resolved.

Update — The problem was diagnosed as an overheating Fiber Mux card at the carrier’s central office.  The card has been replaced and service has been restored.