Month: October 2009

Dial-up Trunk Migration

This Monday, November 2 at 11 AM, we will be migrating dial-up numbers from one of our trunks to another. The migration is expected to take 5-10 minutes, during which time calls to the migrating numbers may fail. If you are disconnected during this maintenance, please re-try your connection attempt a few minutes later. Once the migration is completed, dial-up connections should be functioning exactly as they were prior to the move.

-Jared

Update: This migration was completed at 11:50 AM today. All dial-up numbers should be functioning normally at this time. Please contact support at 707-547-3400 if you are experiencing any dial-up troubles.

php5 upgrade take 2

PHP 5 has been upgraded to version 5.2.11

We rolled back our PHP 5.3 deployment to the older PHP 5.2.8 because many applications installed today are incompatible with PHP 5.3.  In order to make things easier for customers we have rolled out two new versions of PHP 5.  PHP 5.2 has been upgraded to the latest release, 5.2.11 and PHP 5.3 has been deployed at /opt/php5.3/bin/php.  Customers are advised to upgrade or patch their applications to be compatible with PHP 5.3 as PHP 5.2 is nearing end of life.  -William and The SOC

ftp server trouble last weekend

We have identified the single point of failure that caused our ftp servers to stop accepting connections yesterday morning.

We apologize for the downtime, and are working to ensure this won’t happen again.

Thank you,

-Scott

DSL Aggregation Router Reload

This Tuesday, October 20 at 12:01 AM we will be performing maintenance reloads of three Redbacks that terminate traditional DSL service. This will affect all Los Angeles DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.

-Jared

Update: The maintenance reloads have been completed without incident. Downtime for affected customers was under 5 minutes.

PHP5 upgrade

We have upgraded PHP5 on our webcluster to version 5.3.0. If you experience any issues with this new version please let us know.
-William

Updated: Due do some unexpected incompatibility issues we’ve rolled back to the previous version of PHP5.  We’re sorry for any inconvenience this may have caused.  -William and Kelsey

Planned infrastructure maintenance.

Update (0243 16, Oct.) : Maintenance finished; customers shouldn’t have noticed much if any down time. – Augie and Kelsey.

This Thursday (15, Oct.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.

-Augie, Aaron, Don, and Kelsey.Update :

Pogowave finishes network address translation migration

Today is the day that the wireless department will turn off network address translation on our last Pogowave station; The Golden Apple Ranch on Fitzpatrick Lane.

This is our largest and most diverse station, featuring three low band access points.

We think that this can be performed in less than 30 minutes of down time.  Thank you for your patience during this transition that will allow for improved customer support for all Sonic.net Pogowave customers.

DSL DHCP Server Issues

Early this morning four of our DHCP servers started having issues responding quickly to DCHP requests.  These simultaneous failures overwhelmed our ability to migrate load to our hot-standby servers but we were able to put several work-arounds in place to mitigate the issues.  These failures were initially believed to be consistent with disk pre-failure scenarios where a single disk’s performance is impacted and the RAID system has yet to fail the disk.  However, upon further investigation it was revealed that these failures were triggered by scheduled SMART tests.  Ironically, the SMART tests were recently enabled to help us detect and replace failing disks before their failure triggered a service impacting event.

At this time, all DHCP services have been returned to normal.

-Nathan, Don, and Kelsey