This morning at about 9:30 AM we began tracking a problem that was causing some DSL customers in the Bay Area to experience poor speeds and performance. We have tracked the cause of the problem back to a failing ATM card in one of our core routers, and have removed that card from service. At this time, all DSL should be functioning normally, but we continue to monitor the situation.
-Jared and Nathan
This Monday, November 2 at 11 AM, we will be migrating dial-up numbers from one of our trunks to another. The migration is expected to take 5-10 minutes, during which time calls to the migrating numbers may fail. If you are disconnected during this maintenance, please re-try your connection attempt a few minutes later. Once the migration is completed, dial-up connections should be functioning exactly as they were prior to the move.
-Jared
Update: This migration was completed at 11:50 AM today. All dial-up numbers should be functioning normally at this time. Please contact support at 707-547-3400 if you are experiencing any dial-up troubles.
PHP 5 has been upgraded to version 5.2.11
We rolled back our PHP 5.3 deployment to the older PHP 5.2.8 because many applications installed today are incompatible with PHP 5.3. In order to make things easier for customers we have rolled out two new versions of PHP 5. PHP 5.2 has been upgraded to the latest release, 5.2.11 and PHP 5.3 has been deployed at /opt/php5.3/bin/php. Customers are advised to upgrade or patch their applications to be compatible with PHP 5.3 as PHP 5.2 is nearing end of life. -William and The SOC
We have identified the single point of failure that caused our ftp servers to stop accepting connections yesterday morning.
We apologize for the downtime, and are working to ensure this won’t happen again.
Thank you,
-Scott
This Tuesday, October 20 at 12:01 AM we will be performing maintenance reloads of three Redbacks that terminate traditional DSL service. This will affect all Los Angeles DSL subscribers, and some of our Bay Area DSL subscribers. Expected downtime is 5 minutes.
-Jared
Update: The maintenance reloads have been completed without incident. Downtime for affected customers was under 5 minutes.
We have upgraded PHP5 on our webcluster to version 5.3.0. If you experience any issues with this new version please let us know.
-William
Updated: Due do some unexpected incompatibility issues we’ve rolled back to the previous version of PHP5. We’re sorry for any inconvenience this may have caused. -William and Kelsey
Update (0243 16, Oct.) : Maintenance finished; customers shouldn’t have noticed much if any down time. – Augie and Kelsey.
This Thursday (15, Oct.) we will be performing maintenance on some of the internal networking and NetApp storage filers that many of our servers depend on (including e-mail and web); from 11pm to 1am the following morning you may see a slow down in performance when retrieving or sending e-mail or accessing your web content as we perform our maintenance.
-Augie, Aaron, Don, and Kelsey.Update :
Today is the day that the wireless department will turn off network address translation on our last Pogowave station; The Golden Apple Ranch on Fitzpatrick Lane.
This is our largest and most diverse station, featuring three low band access points.
We think that this can be performed in less than 30 minutes of down time. Thank you for your patience during this transition that will allow for improved customer support for all Sonic.net Pogowave customers.
Early this morning four of our DHCP servers started having issues responding quickly to DCHP requests. These simultaneous failures overwhelmed our ability to migrate load to our hot-standby servers but we were able to put several work-arounds in place to mitigate the issues. These failures were initially believed to be consistent with disk pre-failure scenarios where a single disk’s performance is impacted and the RAID system has yet to fail the disk. However, upon further investigation it was revealed that these failures were triggered by scheduled SMART tests. Ironically, the SMART tests were recently enabled to help us detect and replace failing disks before their failure triggered a service impacting event.
At this time, all DHCP services have been returned to normal.
-Nathan, Don, and Kelsey
This upcoming Thursday, Oct 1 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.
-Jared
Update: The router reloads have been completed without incident. All Business-T and FRATM customers should be back up at this time. Total downtime was less than 10 minutes.