Category: Uncategorized

We have identified the single point of failure that caused our ftp servers to stop accepting connections yesterday morning.

We apologize for the downtime, and are working to ensure this won’t happen again.

Thank you,

-Scott

This Tuesday, October 20 at 12:01 AM we will be performing maintenance reloads of three Redbacks that terminate traditional DSL service. This will affect all Los Angeles DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.

-Jared

Update: The maintenance reloads have been completed without incident. Downtime for affected customers was under 5 minutes.

We have upgraded PHP5 on our webcluster to version 5.3.0. If you experience any issues with this new version please let us know.
-William

Updated: Due do some unexpected incompatibility issues we’ve rolled back to the previous version of PHP5.  We’re sorry for any inconvenience this may have caused.  -William and Kelsey

Update (0243 16, Oct.) : Maintenance finished; customers shouldn’t have noticed much if any down time. – Augie and Kelsey.

This Thursday (15, Oct.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.

-Augie, Aaron, Don, and Kelsey.Update :

Early this morning four of our DHCP servers started having issues responding quickly to DCHP requests.  These simultaneous failures overwhelmed our ability to migrate load to our hot-standby servers but we were able to put several work-arounds in place to mitigate the issues.  These failures were initially believed to be consistent with disk pre-failure scenarios where a single disk’s performance is impacted and the RAID system has yet to fail the disk.  However, upon further investigation it was revealed that these failures were triggered by scheduled SMART tests.  Ironically, the SMART tests were recently enabled to help us detect and replace failing disks before their failure triggered a service impacting event.

At this time, all DHCP services have been returned to normal.

-Nathan, Don, and Kelsey

This upcoming Thursday, Oct 1 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Jared

Update: The router reloads have been completed without incident. All Business-T and FRATM customers should be back up at this time. Total downtime was less than 10 minutes.

Starting at approximately 9:50 this evening, some areas of Santa Rosa began experiencing intermittent DSL connectivity.   We are working with ATT to resolve the issue.    As a result of this outage, the telephone wait for Technical Support is unusually high.  We expect resolution shortly, but do not have an ETR .

-Kim,  Support

Update: AT&T has diagnosed the problem as a failing card at an ATM switch in Northern California.  This outage persists. We currently have no estimated time of repair. AT&T has reset the card which seems to have temporarily restored service to some customers while they work toward a permanent solution.

Update:  Service should now be fully restored to all customers affected by this outage.  If you are still having connectivity issues please contact Sonic.net technical support.

Tonight, this Tuesday, Sep 15 at 12:01 AM, we will be performing a software upgrade of the linecards that provide ADSL2+ service out of our Downtown Santa Rosa CO. This will cause a brief disruption of service to ADSL2+ customers in that area. Affected customers will experience approximately 5 minutes of downtime as the cards reboot onto the new software.

-Jared

Update: The ADSL2+ cards have all been upgraded to the new software and are up and functioning normally.

This evening, Saturday September 12, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from a heavily loaded DSL aggregation router to a newer, less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes. We apologize for the short lead time on this announcement, but we feel it is better to address the load issue before the weekend. Thank you for your understanding.

-Jared
Update: The migration has been completed without incident and all affected customers appear to be up and running normally at this time.

This afternoon at 12:15 PM a huge DoS attack was aimed at a customer on our Sonic Telecom CLEC network. The DoS attack was large enough to destabilize routing in our CLEC network, so customers may have seen 10-15 minutes of spotty connectivity as we tracked down and blocked the DoS. At this time, the attack has been blocked and all services are functioning normally.

-Jared and Nathan