Legacy DSL Aggregation Router Reconfiguration

This Thursday, December 29 at 12:01 AM we will be performing core configuration changes on the routers at the head end of our Legacy DSL aggregation network. We will be taking all possible precautions to prevent any outages, but when doing this type of deep configuration change, the possibility of a problem is always present. The reconfiguration should take between 30 and 60 minutes, and we will be able to roll back quickly in the event of an issue.

-Jared and Nathan

Update: The reconfiguration has been completed. There were a couple brief, unexpected hiccups which disrupted connectivity to a small subset of our legacy DSL customers. We apologize for any inconvenience this may have caused.

ATM Customer Aggregation Router Reload

This Wednesday, December 28 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Jared

ssl.sonic.net outage

The server that handles ssl.sonic.net (our legacy shared https web hosting server) suffered a double disk raid failure (in a mirror) this afternoon and was offline for approximately 23 minutes while we restored to replacement disks from our backups.  -Kelsey and William

Closure for Staff Party

Sonic.net will be closed today from noon to five pm for our annual staff holiday party. Thanks everyone for a great 2011, it has been our best year yet. Our rapidly growing team (135 of us now!) will be raising a glass to toast our customers this afternoon.

Technical support will re-open 5pm-11pm, and we will be returning calls to anyone who has left a message during the closure. Our 24×7 network operations team will be available on call on the NOC hotline for our Enterprise dedicated circuit and colocation customers.

Network Solutions Outage.

Starting around 1pm pst Network Solution’s name servers became unreachable. This problem appears to be isolated to the west coast. Network Solutions has been made aware of the problem, and are  working on a fix.

MySQL 4.1 custsql migration

Tonight, Wednesday December 7th at 12:10 AM we will be migrating all databases currently on our MySQL 4.1 database server to our new MySQL 5.5 server. We are anticipating approximately 2 hours of downtime. -William and the SOC

Update: All services have been migrated to the new server.

Non-Intrusive Network Maintenance

Tonight, Tuesday December 6 at 12:01 AM we will be performing non-intrusive network maintenance on one of our Santa Rosa POPs. This maintenance should last only 5 minutes and there should not be any customer impact.

-Jared

Update: The maintenance has been completed. No impact to customer traffic was detected.

Customer SQL Server Outage

This morning, at 8:32 AM, one of our customer MySQL servers crashed due to an apparent bug in MySQL.  Additionally, after the crash it was failing to restart.  The trouble was isolated to a specific user’s database, which when manually removed, allowed the MySQL server to start normally.  The server appears to be functioning correctly at this time.

San Jose DSL outage

We have confirmed with AT&T that there is currently a DSL outage in the San Jose area. They are working to resolve the issue but we do not currently have an ETR. Symptoms are that customers have sync but are unable to get on line.

 

Update: The outage should be clear. If your connection is still not working please try rebooting your DSL modem.

DSL Aggregation Router Reload

Tonight, Tuesday November 29 at 12:01 AM, we will be performing a maintenance reload on several legacy DSL aggregation routers that serve some of our customers in the Bay Area and Los Angeles areas. Affected customers will experience 5-10 minutes of downtime as the routers reboot.

-Jared

Update: The maintenance reloads have been completed and all affected customers should be back online.