Internal Database Server Upgrades

Tonight, after Support closes, we’re going to begin an invasive upgrade on our primary internal database servers.  The upgrade necessitates a complete dump and restore to new hardware and is expected to take several hours to complete.  The new servers are substantially more powerful and many of our internal systems should see a marked performance improvement.  While the data is being migrated and imported, all signup and member tools will be unavailable but all other services will otherwise remain unaffected.  — Kelsey and William

Update:  The upgrade appears to have gone according to plan and all internal systems are back online at this time.

Changes to our DNS servers ns1/ns2.sonic.net

Starting today ( 8, Feb ) the name servers we hand out to our customers for DNS resolution ( ns1.sonic.net and ns2.sonic.net ) are going to be updated to exclude authoritative data, and be recursive only name servers.

This means that our name servers will behave more like other name servers on the Internet, prior to this change you may have received confusing results when asking our name server for a domain name, and asking Google’s open recursive DNS servers ( for example ) — now you will receive the same results.

Note: customers do not need to update their DNS resolver settings.

–Augie

Intrusive Network Maintenance

Tonight,  Wednesday 8 at 12:01 AM we will be performing some intrusive network maintenance on our backhaul circuits to the Sacramento area. Customers in that area may experience 5-10 minutes of downtime.

-Jared and Nathan

Legacy DSL Outage

Tonight, February 5 at approximately 12:25 AM, one of the ATM switches that serves legacy DSL to the greater bay and Chico areas suffered a card software failure. The switch was rebooted to clear the problem, and all affected customers should be back online. We will be investigating this issue with the equipment vendor.

-Jared

Emergency Legacy DSL Maintenance

We will be performing immediate emergency maintenance on equipment serving a number of legacy DSL customers in the LATA1 area. This will impact affected customers for 5 to 10 minutes while the maintenance is performed.

-Tim and Jared

Update: The source of the problem was a bad port on a single DSL aggregation router in our ATM network. We have moved the affected connection to a new port and re-routed all traffic to the new port. Affected customers should be back online at this time.

Colocation UPS Maintenance

This Thursday, starting at 9AM, one of the three UPSes that serves our Santa Rosa datacenter will have all of it’s batteries replaced.  This is not a service impacting event and the UPS will be online for the duration of  work.  If necessary, the battery replacement may continue Friday morning.   -Kelsey and Russ

Santa Rosa Datacenter Router Replacement, Part 2

Tonight, Thursday, January 26 at 11:01 PM, we will be replacing the second core router in our datacenter. The switchover should only take a few minutes, and traffic will be routed around the affected router. If there is any disruption, it should not last more than 5-10 minutes at worst.

-Jared

Update: The router replacement was completed without incident. There should have been no impact to customer traffic.

Santa Rosa Datacenter Router Replacement

Tonight, Tuesday January 24 at 12:01 AM, we will be replacing the core router in our datacenter which failed earlier this morning. The switchover should only take a few minutes, and traffic will be routed around the affected router. If there is any disruption, as routing updates, it should not last more than 5-10 minutes at worst.

-Jared

Update: The router replacement has been completed without incident. There should have been no impact to customer traffic. In a few days we will schedule a replacement of the other in the pair of redundant routers in our Santa Rosa datacenter.

Santa Rosa Datacenter Routing Hiccup

At 5:32 AM today, one of our core routers in our Santa Rosa datacenter unexpectedly restarted its routing processes. This caused 5-10 minutes of impacted connectivity to our datacenter as the network recovered from the event. This would have affected access to mail, Sonic.net hosted websites, and colocated servers.

Ironically, we had already scheduled retirement and replacement of our core routers in the Santa Rosa datacenter for this upcoming week. We will accelerate our plans due to this event, and apologize for any impact it may have caused.

-Jared

Member tool authentication improvement

Today we have updated our member tool authentication code. This is a transparent change in our back end code and does not require any end user changes. We believe this will resolves an intermittent bug  that was causing users to occasionally have to re-authenticate their session and also results in a slight performance increase while using the tools.

-William