Database Server Upgrade

*Update* All work is complete – customer databases should now be back up and functional.

This Thursday, Aug 22, from 12am to 1am, System Operations will be performing a memory upgrade to the customer SQL server. During the maintenance window, customer databases will not be reachable.

— Joe & the SOC

Customer MySQL Database Servers Maintenance

Tonight at Midnight, our customer database server will be taken off-line to replace two failed disks as well as repair some suspected bad cabling on the raid controller.  While the server is off-line any hosting accounts requiring database access will be off-line as well.  It is expected to take less that 30 minutes to complete the replacement and repair and have the system back in on-line and in service.

Services were restored on time just before 00:30.

-Kelsey

Support Closed Sunday

Our team will be taking time off this coming Sunday to celebrate Sonic.net’s 19th birthday.  Customer Support will be closed on 8/11/2013 from 11am to 6pm.  Support will re-open at 6pm on Sunday and remain available until 10pm.

Our normal 24x7x365 schedule is still applicable for our on-call network operations team for dedicated business and colocated customers.

-Adam and Support

FlexLink Equipment Maintenance

On Thursday the 8th, beginning at 12:00AM, we will be performing intrusive maintenance on equipment serving FlexLink Long Range customers in the Burlingame area. Expected downtime for affected customers is less than 15 minutes.

-Tim and Monroe

Update: We will also be performing maintenance on equipment serving a very small number of Private Ethernet Transport customers in the San Francisco area. Expected downtime for this operation is less than 10 minutes.

Customer Database Load

Due to excessive load on our customer database servers,  databases connection may intermittently time out. We are working to fix the problem and prevent further interruption.

Update: Currently both customer mysql server instances are off-line, we’re investigating what appears to be a hardware related issue and hope to have services restores shortly.

Update: It appears that these problems were caused by an issue with the raid controller not properly handling IO at anywhere near its expected performance levels.  A reboot seems to have restored performance back to normal.  We will continue to monitor the system closely.

 

 

Custom Hosting updates

Update: Reboot was complete as of 12:05 am.

 

Tonight, at 11:59 pm, we will be rebooting our custom hosting platform. Any sites hosted on that platform will be unreachable during this time. Expect service to be available at 12:15 am.

Grant and SOC

Database Latency

UPDATE: Customer databases and sites are back to normal.

Operations staff is currently acting on reports of slow database access on our customers’ websites. An update will be posted when access times are back to normal.

— Joe and the SOC

NOC hotline unreachable

The NOC hotline for business customer support is presently experiencing call completion issues. We are working to diagnose and resolve this problem, but in the mean time please contact support at 707-547-3400 and request to be transferred to the NOC if necessary.  An update will be posted when this service interruption has been restored.

-Sonic.net NOC

Update: NOC hotline reachablity should now be restored. We are still working to determine the root cause of the issue.

Server Maintenance

Tonight, July 25, from 11:30 – 11:59pm, System Operations will be performing server updates. Customers with Custom Hosting will experience a short period of downtime. All other upgrades are internal and should not cause service interruption.

— Joe, Grant, and the SOC