Web and MySQL failure

This morning around 6:30 AM one of the servers in our web cluster locked up and became unavailable. Whilst en route to investigate the problem two more servers in the cluster locked up and became unavailable. Customers may have noticed decreased performance with the cluster but should not have suffered any actual down time. Unfortunately due to human error, during the restoration of service we also had one of our MySQL servers go down because someone unplugged the wrong server. šŸ™

It went something like this šŸ™‚ . We apologize for any inconvenience and are trying to determine the cause to prevent future failures of this nature.

-William and the SOC

Webmail interface database maintenance

At 12:01 this morning, I will be performing the second and final phase of maintenance items on our databases that serve the Sonic.net Webmail interfaces During this time, there will be a slight interruption in the ability to connect to these interfaces. In no way does this effect the flow of mail, inbound/outbound, or IMAP/POP services. –Don

Update: Maintenance completed. All systems restored.

Sonic Telecom Santa Rosa CO Retrofit

This coming Tuesday, Feb 17, beginning at 12:01 AM, we will be performing a retrofit of our network build supporting our Sonic Telecom CLEC installation at the main Santa Rosa central office. The maintenance window is expected to be up to an hour, but customer impact should be minor, no more than 2 10-minute outages during that time frame. This maintenance will only affect customers who recieve Sonic Telecom services in the downtown Santa Rosa area.

-Jared and Clay

Update: The retrofit has been completed. All operations went according to plan, and customers should have seen at most 10 minutes of service interruption. All customers served from this CO appear to be fully functional again.

Emergency Router Maintenance

At 3:45AM this afternoon we will be performing an emergency router reload
on one of our ATM customer aggregation routers. All connected Business-T
and FRATM customers will experience approximately 5 minutes of downtime
during the reload.
-Jared and Matt

Data backup storage increase.

Due to the overwhelming interest in our Sonic.net Data Backup service we have had to greatly expand the storage capacity of our backup servers. Our data backup service will be unavailable this morning for a few hours while we migrate data to our new servers. Any pending backup jobs will continue once data migration is complete and the backup servers are back online. For more information about Sonic.net Data Backup please see our sales page.

-William and the SOC

DHCP Server Failure

One of the DHCP servers that serves our Bay Area DSL subscribers had a disk begin failing, which caused the server to not properly handle all DHCP requests sent to it. This caused spotty DHCP connectivity for some customers until we were able to diagnose the silently failing disk. We have removed the failing server from service, and DHCP service should be fully restored to the affected customers.

-Jared and Nathan

DSL Aggregation Router Maintenance Reload

This coming Friday, Feb 6 at 12:01 AM, we will be performing maintenance reloads of two of our DSL aggregation routers. These routers serve DSL to customers in the Bay Area. Expected impact is less than 5 minutes of loss of connectivity for DSL customers served by these routers.

-Jared

Update: The maintenance reloads have been completed without incident. Total downtime for affected customers was less than 5 minutes.

Unexpected Customer Router Reload

At 6:07 PM today, one of our T1 aggregation routers in Santa Rosa reloaded unexpectedly. The router is back up and serving normally at this time. We will be investigating the cause of this router reload, and apologize for any interruption of service this may have caused.

-Jared and Nathan

Emergency Provider Circuit Maintenance

The backhaul provider of our circuit that provides DSL service to the Chico area must perform card maintenance that will disrupt our service to that area. The expected downtime is between 1 and 15 minutes as they replace the card, and is scheduled to occur tomorrow, Friday, Jan 30, between 5AM and 6AM. We apologize for this necessary interruption of service.

-Jared and Nathan

SMS Uplink Maintenance

One of the uplinks on our SMS that terminates DSL for subscribers in Sacramento is taking errors, causing slight packet loss for all subscribers on that SMS. At 11:45 today, we will be moving that uplink to a new port to resolve the problem. This move will cause 5 minutes of loss of connectivity for all subscribers on this SMS.

-Jared and Nathan

Update: This uplink move has been completed with no complications. Service from this SMS is now trouble-free.