Unexpected DSL Aggregation Router Reboot

At approximately 3:00 AM today, one of our DSL aggregation routers serving customers in the Bay Area suffered what appears to be a software crash. This software crash kept the router out of service until 3:14 AM, at which time it was booted up and handling customer traffic normally.

The exact cause of the software crash is being investigated with the vendor at this time. We are monitoring the router, and all affected customers appear to be back online at this time.

-Jared

Update: The router has unexpectedly crashed again. We are in the process of moving all customers served by this router to a hot spare. Connectivity should be restored shortly.

Update: All affected customers have been moved to the hot spare successfully. All affected customers should be online again at this time.

DSL Outage in Stockton area

At this time we are tracking an ATM backhaul problem affecting LATA9. This LATA covers DSL service for the Stockton/Modesto areas. We believe the problem to be on AT&T’s backhaul side, and are working with them currently to diagnose and repair.

-Jared

Update: We have narrowed the issue down to a specific line card in an ATM device in our network. Due to the nature of the problem, we had to perform a reload of the entire device. This briefly affected customers in LATAs 1, 2, and 6 as well. We apologize for affecting previously unaffected customers while reloading this device.

CLEC DSLAM Maintenance

This evening, at 12:01AM, we will be performing a software upgrade on the DSLAM that serves Fusion and FlexLink customers in the downtown Santa Rosa area. Expected downtime is less than 30 minutes while the DSLAM is rebooted onto the new software release.

-Tim and Nathan

DSL Aggregation Router Reload

This evening, Saturday, February 6 at 12:01 AM we will be performing maintenance reloads of two Redbacks that terminate traditional DSL service. This will affect all Chico DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.

-Jared

Update: The reloads have been completed without incident. All affected customers should be back online at this time. Total downtime was less than 5 minutes.

CLEC Intrusive Maintenance

This evening, between 12:01AM and 1:00AM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink ADSL2+ customers in downtown Santa Rosa. Affected customers should experience less than 15 minutes of downtime.

-Tim

Webmail Service Interuption

Both SquirrelMail and NutsMail we’re unavailable from approximately 11:45 to 12:00 today. AtMail was not affected.  We’re sorry for the interruption and will review the procedures that led to the failure.

-Sonic Operations

CLEC Network Issues

We are currently experiencing issues with one of the AT&T back haul circuits that carries traffic to and from the Sonic Telecom network. We are working with AT&T on resolving the issue and have isolated the trouble to a specific card in a central office. We will provide further updates as soon as possible.

-Tim and Nathan

Update: The issue has been resolved. The circuit has remained stable for the past 45 minutes. We will continue to monitor closely for any further problems.

CLEC DSLAM Maintenance

This evening, at 12:01AM, we will be performing a software upgrade on the DSLAM that serves Fusion customers in the Rincon Valley area of Santa Rosa. Expected downtime is less than 30 minutes while the DSLAM is rebooted onto the new software release.

-Tim and Nathan

MySQL scheduled maintenance tonight

Tonight at 10:00PM I will be performing some needed maintenance on two of our public facing MySQL servers. This will most likely cause a small window of about 3-5 minutes where queries directed at these two systems may timeout. Every effort will be taken to make this window as very small as possible. –Don

Update: This maintenance window has been moved to 12 midnight from 10PM.
Update: Maintenance completed successfully – Less than 60 seconds of possible interruption window.

Authoritative Name Server Outage.

From about 6:30pm to 7:30pm we lost connectivity to one of our geographically diverse redundant Authoritative Name Servers (c.auth-ns.sonic.net) as the co-location facility that it is hosted at recovered from a major network outage — customers’ websites may have been slow to load during this time as DNS requests were routed to the other Name Servers.

–Augie