Month: February 2009

Router Software Upgrade

This Tuesday, March 3 at 12:01 AM, we will be performing software upgrades on two of our T1 and WBA customer aggregation routers in Santa Rosa. The software upgrade will require a reboot of the routers, resulting in approximately 5 minutes of downtime for customers served by these routers. These routers serve T1 customers in the Santa Rosa area, and a small subset of our WBA customers.

-Jared

The router software upgrades have been completed. Of the two routers being upgraded, one went smoothly, but the other had some issues properly loading the new software. This router was offline for approximately 20 minutes across 3 intervals while we worked to get the new software loaded. At this time, both routers are fully upgraded and appear to be operating normally. We apologize for the extra downtime incurred during this operation.

Web and MySQL failure

This morning around 6:30 AM one of the servers in our web cluster locked up and became unavailable. Whilst en route to investigate the problem two more servers in the cluster locked up and became unavailable. Customers may have noticed decreased performance with the cluster but should not have suffered any actual down time. Unfortunately due to human error, during the restoration of service we also had one of our MySQL servers go down because someone unplugged the wrong server. 🙁

It went something like this 🙂 . We apologize for any inconvenience and are trying to determine the cause to prevent future failures of this nature.

-William and the SOC

Webmail interface database maintenance

At 12:01 this morning, I will be performing the second and final phase of maintenance items on our databases that serve the Sonic.net Webmail interfaces During this time, there will be a slight interruption in the ability to connect to these interfaces. In no way does this effect the flow of mail, inbound/outbound, or IMAP/POP services. –Don

Update: Maintenance completed. All systems restored.

Sonic Telecom Santa Rosa CO Retrofit

This coming Tuesday, Feb 17, beginning at 12:01 AM, we will be performing a retrofit of our network build supporting our Sonic Telecom CLEC installation at the main Santa Rosa central office. The maintenance window is expected to be up to an hour, but customer impact should be minor, no more than 2 10-minute outages during that time frame. This maintenance will only affect customers who recieve Sonic Telecom services in the downtown Santa Rosa area.

-Jared and Clay

Update: The retrofit has been completed. All operations went according to plan, and customers should have seen at most 10 minutes of service interruption. All customers served from this CO appear to be fully functional again.

Emergency Router Maintenance

At 3:45AM this afternoon we will be performing an emergency router reload
on one of our ATM customer aggregation routers. All connected Business-T
and FRATM customers will experience approximately 5 minutes of downtime
during the reload.
-Jared and Matt

Data backup storage increase.

Due to the overwhelming interest in our Sonic.net Data Backup service we have had to greatly expand the storage capacity of our backup servers. Our data backup service will be unavailable this morning for a few hours while we migrate data to our new servers. Any pending backup jobs will continue once data migration is complete and the backup servers are back online. For more information about Sonic.net Data Backup please see our sales page.

-William and the SOC

DHCP Server Failure

One of the DHCP servers that serves our Bay Area DSL subscribers had a disk begin failing, which caused the server to not properly handle all DHCP requests sent to it. This caused spotty DHCP connectivity for some customers until we were able to diagnose the silently failing disk. We have removed the failing server from service, and DHCP service should be fully restored to the affected customers.

-Jared and Nathan

DSL Aggregation Router Maintenance Reload

This coming Friday, Feb 6 at 12:01 AM, we will be performing maintenance reloads of two of our DSL aggregation routers. These routers serve DSL to customers in the Bay Area. Expected impact is less than 5 minutes of loss of connectivity for DSL customers served by these routers.

-Jared

Update: The maintenance reloads have been completed without incident. Total downtime for affected customers was less than 5 minutes.