Month: October 2009

This Monday, November 2 at 11 AM, we will be migrating dial-up numbers from one of our trunks to another. The migration is expected to take 5-10 minutes, during which time calls to the migrating numbers may fail. If you are disconnected during this maintenance, please re-try your connection attempt a few minutes later. Once the migration is completed, dial-up connections should be functioning exactly as they were prior to the move.

-Jared

Update: This migration was completed at 11:50 AM today. All dial-up numbers should be functioning normally at this time. Please contact support at 707-547-3400 if you are experiencing any dial-up troubles.

PHP 5 has been upgraded to version 5.2.11

We rolled back our PHP 5.3 deployment to the older PHP 5.2.8 because many applications installed today are incompatible with PHP 5.3.  In order to make things easier for customers we have rolled out two new versions of PHP 5.  PHP 5.2 has been upgraded to the latest release, 5.2.11 and PHP 5.3 has been deployed at /opt/php5.3/bin/php.  Customers are advised to upgrade or patch their applications to be compatible with PHP 5.3 as PHP 5.2 is nearing end of life.  -William and The SOC

We have identified the single point of failure that caused our ftp servers to stop accepting connections yesterday morning.

We apologize for the downtime, and are working to ensure this won’t happen again.

Thank you,

-Scott

This Tuesday, October 20 at 12:01 AM we will be performing maintenance reloads of three Redbacks that terminate traditional DSL service. This will affect all Los Angeles DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.

-Jared

Update: The maintenance reloads have been completed without incident. Downtime for affected customers was under 5 minutes.

We have upgraded PHP5 on our webcluster to version 5.3.0. If you experience any issues with this new version please let us know.
-William

Updated: Due do some unexpected incompatibility issues we’ve rolled back to the previous version of PHP5.  We’re sorry for any inconvenience this may have caused.  -William and Kelsey

Update (0243 16, Oct.) : Maintenance finished; customers shouldn’t have noticed much if any down time. – Augie and Kelsey.

This Thursday (15, Oct.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.

-Augie, Aaron, Don, and Kelsey.Update :

Early this morning four of our DHCP servers started having issues responding quickly to DCHP requests.  These simultaneous failures overwhelmed our ability to migrate load to our hot-standby servers but we were able to put several work-arounds in place to mitigate the issues.  These failures were initially believed to be consistent with disk pre-failure scenarios where a single disk’s performance is impacted and the RAID system has yet to fail the disk.  However, upon further investigation it was revealed that these failures were triggered by scheduled SMART tests.  Ironically, the SMART tests were recently enabled to help us detect and replace failing disks before their failure triggered a service impacting event.

At this time, all DHCP services have been returned to normal.

-Nathan, Don, and Kelsey