Category: Maintenance

Santa Rosa Datacenter Router Replacement

Tonight, Tuesday January 24 at 12:01 AM, we will be replacing the core router in our datacenter which failed earlier this morning. The switchover should only take a few minutes, and traffic will be routed around the affected router. If there is any disruption, as routing updates, it should not last more than 5-10 minutes at worst.

-Jared

Update: The router replacement has been completed without incident. There should have been no impact to customer traffic. In a few days we will schedule a replacement of the other in the pair of redundant routers in our Santa Rosa datacenter.

Legacy DSL Aggregation Router Reconfiguration

This Thursday, December 29 at 12:01 AM we will be performing core configuration changes on the routers at the head end of our Legacy DSL aggregation network. We will be taking all possible precautions to prevent any outages, but when doing this type of deep configuration change, the possibility of a problem is always present. The reconfiguration should take between 30 and 60 minutes, and we will be able to roll back quickly in the event of an issue.

-Jared and Nathan

Update: The reconfiguration has been completed. There were a couple brief, unexpected hiccups which disrupted connectivity to a small subset of our legacy DSL customers. We apologize for any inconvenience this may have caused.

Closure for Staff Party

Sonic.net will be closed today from noon to five pm for our annual staff holiday party. Thanks everyone for a great 2011, it has been our best year yet. Our rapidly growing team (135 of us now!) will be raising a glass to toast our customers this afternoon.

Technical support will re-open 5pm-11pm, and we will be returning calls to anyone who has left a message during the closure. Our 24×7 network operations team will be available on call on the NOC hotline for our Enterprise dedicated circuit and colocation customers.

Non-disruptive NetApp Filer Maintenance.

Tonight ( 29, August ) at 11:00 PM we will be performing a set of software upgrades across our NetApp Filers.

These Filers store customers’ e-mail, web files, and other data.

There will be no impact on customers ability to access their information during this maintenance, and we expect it to take no more than an hour to complete.

–Augie

Update: Maintenance completed at 12:30 AM Tuesday morning. Not all of the software updates were applied during the maintenance window, another maintenance window is scheduled for tonight ( 30, August ) at 11:00 PM to complete the software updates.

Update: 31, August The final round of software upgrades will take place tonight at 11:00 PM.

Update: 1, September Maintenance continues tonight at 11:00 PM as last night’s upgrade did not proceed as planned, tonight will hopefully be the final round of upgrades, again there will be no customer impact during the maintenance. –Augie

FINAL UPDATE: 2, September 00:01 Maintenance complete all NetApp filers have been updated.

Scheduled mail storage infrastructure maintenance.

07:40 — Some customers may have noticed a problem when checking their e-mail via imap.sonic.net this AM; the new NetApp filers’ time was off by 30 seconds, which was enough for the IMAP software to think there was a significant problem and not allow those people whose e-mail is stored on the newer cluster to check their e-mail. A configuration error on the new filers was to blame and has been corrected. –Augie

01:49 — Maintenance complete; no major hiccups, please contact us immediately if you notice any problems. –Augie

Beginning early this Friday morning ( 15, July ) at 12:30AM we will be performing maintenance on the cluster of NetApp filers that store customers’ e-mail messages.

We will be adding an additional storage cluster which will improve performance and reliability for all customers’ sending and receiving of e-mail.

E-mail may be un-available or delayed during the maintenance window, which is estimated at one hour.

–Augie

ATM Customer Aggregation Router Reload

This Tuesday, June 28 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Jared

UPS Maintenance

Next Tuesday, 9/14, the UPS’s serving our Santa Rosa data center will be be undergoing their semi-annual non-intrusive maintenance and inspection.  This is a fully scripted event coordinated with our service vendors and no interruption of power services should occur.  -Kelsey and Russ

CLEC Intrusive Maintenance – SNFCCA21

At 12:01AM on 9/2/10 we will be performing intrusive maintenance on the DSLAM serving the (San Francisco) SNFCCA21 POP. All Flexlink Ethernet and Fusion customers served from this POP should experience between 5-15 minutes of downtime while the work is performed.

-Matt, Tim and Jared

Update: Maintenance work has been completed. Downtime for affected customers was less than 5 minutes.

Scheduled server and power maintenance.

Tomorrow evening (24, June) starting at 10pm Pacific Time we will be changing the power configuration on the NetApp storage filers that store customer e-mail; these filers are redundantly powered and we expect no down time for them during the maintenance window. During this time we will also perform server maintenance that may cause brief service interruption to our customer facing Member Tools.

–Augie

Non-Intrusive Santa Rosa T1 Aggregation Router Maintenance

This evening at 12:01Am we will be performing a card swap on one of the aggregation routers serving legacy T1s out of our Santa Rosa office.  This should not be customer impacting, but like most maintenance there is a small potential chance of downtime.

-Matt Tim and Jared

Update: This router maintenance has completed without incident.