Month: February 2012

Sacramento Network Reconfiguration

Tonight, Tuesday, Frebruary 28 at 12:01 AM we will be performing some potentially intrusive network configuration changes to our Telecom network in the Sacramento area. We do not expect any downtime, but there may be some brief outage of network connectivity for Fusion and Flexlink customers in the Sacramento area.

-Jared

ADSL Outage

This afternoon we are experiencing an outage on our ADSL network that manifests as end-users retaining sync but getting either extremely-slow page loads or a complete loss of the ability to surf. Our network engineering staff is currently investigating.  Fusion, dial-up, and other services should be unaffected. — John F

Update: this outage was caused by a card failure. We have switched over to replacement equipment and all affected circuits should be fixed. — Nathan and John F

Customer Database Server Upgrades

Tonight, February 10th at 12:01 AM, the shared customer database servers will undergo routine maintenance which requires a reboot of the servers.  Expected downtime for customer websites that use the shared database service is less than 10 minutes.  This work is in addition to the internal database server upgrades scheduled tonight, but is unrelated.

-Kevan

Update:  All upgrades were completed without event and customer database services are working correctly.

Internal Database Server Upgrades

Tonight, after Support closes, we’re going to begin an invasive upgrade on our primary internal database servers.  The upgrade necessitates a complete dump and restore to new hardware and is expected to take several hours to complete.  The new servers are substantially more powerful and many of our internal systems should see a marked performance improvement.  While the data is being migrated and imported, all signup and member tools will be unavailable but all other services will otherwise remain unaffected.  — Kelsey and William

Update:  The upgrade appears to have gone according to plan and all internal systems are back online at this time.

Changes to our DNS servers ns1/ns2.sonic.net

Starting today ( 8, Feb ) the name servers we hand out to our customers for DNS resolution ( ns1.sonic.net and ns2.sonic.net ) are going to be updated to exclude authoritative data, and be recursive only name servers.

This means that our name servers will behave more like other name servers on the Internet, prior to this change you may have received confusing results when asking our name server for a domain name, and asking Google’s open recursive DNS servers ( for example ) — now you will receive the same results.

Note: customers do not need to update their DNS resolver settings.

–Augie

Intrusive Network Maintenance

Tonight,  Wednesday 8 at 12:01 AM we will be performing some intrusive network maintenance on our backhaul circuits to the Sacramento area. Customers in that area may experience 5-10 minutes of downtime.

-Jared and Nathan

Legacy DSL Outage

Tonight, February 5 at approximately 12:25 AM, one of the ATM switches that serves legacy DSL to the greater bay and Chico areas suffered a card software failure. The switch was rebooted to clear the problem, and all affected customers should be back online. We will be investigating this issue with the equipment vendor.

-Jared

Emergency Legacy DSL Maintenance

We will be performing immediate emergency maintenance on equipment serving a number of legacy DSL customers in the LATA1 area. This will impact affected customers for 5 to 10 minutes while the maintenance is performed.

-Tim and Jared

Update: The source of the problem was a bad port on a single DSL aggregation router in our ATM network. We have moved the affected connection to a new port and re-routed all traffic to the new port. Affected customers should be back online at this time.

Colocation UPS Maintenance

This Thursday, starting at 9AM, one of the three UPSes that serves our Santa Rosa datacenter will have all of it’s batteries replaced.  This is not a service impacting event and the UPS will be online for the duration of  work.  If necessary, the battery replacement may continue Friday morning.   -Kelsey and Russ