Infrastructure maintenance.

Update : (0123 20, Nov.) Maintenance complete; everything is fully operational again.

Tonight (19, Nov.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.

-Augie, Aaron

Large DDoS Attack

This morning at 11AM, a large DDoS was aimed at a server in our colocation facility in Santa Rosa. The DoS was large enough (1.5 million pps) to disrupt connectivity to our Santa Rosa datacenter, which would have affected access to mail and web services hosted by Sonic. The DoS was blocked by 11:07 AM, and no further ill effects should have been felt after that point.

-Jared

POP3 problems

We experienced a brief issue with our POP3 (email) performance today due to a problem with our load balancing servers. This issue has been resolved and all POP3 services should be working normally at this time.

DSL Performance Problems in LATA1

This morning at about 9:30 AM we began tracking a problem that was causing some DSL customers in the Bay Area to experience poor speeds and performance. We have tracked the cause of the problem back to a failing ATM card in one of our core routers, and have removed that card from service. At this time, all DSL should be functioning normally, but we continue to monitor the situation.

-Jared and Nathan

Dial-up Trunk Migration

This Monday, November 2 at 11 AM, we will be migrating dial-up numbers from one of our trunks to another. The migration is expected to take 5-10 minutes, during which time calls to the migrating numbers may fail. If you are disconnected during this maintenance, please re-try your connection attempt a few minutes later. Once the migration is completed, dial-up connections should be functioning exactly as they were prior to the move.

-Jared

Update: This migration was completed at 11:50 AM today. All dial-up numbers should be functioning normally at this time. Please contact support at 707-547-3400 if you are experiencing any dial-up troubles.

php5 upgrade take 2

PHP 5 has been upgraded to version 5.2.11

We rolled back our PHP 5.3 deployment to the older PHP 5.2.8 because many applications installed today are incompatible with PHP 5.3.  In order to make things easier for customers we have rolled out two new versions of PHP 5.  PHP 5.2 has been upgraded to the latest release, 5.2.11 and PHP 5.3 has been deployed at /opt/php5.3/bin/php.  Customers are advised to upgrade or patch their applications to be compatible with PHP 5.3 as PHP 5.2 is nearing end of life.  -William and The SOC

ftp server trouble last weekend

We have identified the single point of failure that caused our ftp servers to stop accepting connections yesterday morning.

We apologize for the downtime, and are working to ensure this won’t happen again.

Thank you,

-Scott

DSL Aggregation Router Reload

This Tuesday, October 20 at 12:01 AM we will be performing maintenance reloads of three Redbacks that terminate traditional DSL service. This will affect all Los Angeles DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.

-Jared

Update: The maintenance reloads have been completed without incident. Downtime for affected customers was under 5 minutes.

PHP5 upgrade

We have upgraded PHP5 on our webcluster to version 5.3.0. If you experience any issues with this new version please let us know.
-William

Updated: Due do some unexpected incompatibility issues we’ve rolled back to the previous version of PHP5.  We’re sorry for any inconvenience this may have caused.  -William and Kelsey

Planned infrastructure maintenance.

Update (0243 16, Oct.) : Maintenance finished; customers shouldn’t have noticed much if any down time. – Augie and Kelsey.

This Thursday (15, Oct.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.

-Augie, Aaron, Don, and Kelsey.Update :