Category: Uncategorized

Fusion/FlexLink Equipment Maintenance

This evening, beginning at 12:00AM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink Ethernet customers in the Forestville area. Expected downtime for affected customers is less than 15 minutes.

-Tim

Customer Service Phone Interruption

Starting at 2:54pm today, inbound calls to our customer service phone lines in Santa Rosa (serving our sales, billing, and technical support representatives) lost connectivity. We are currently working to resolve this issue. We have no ETR. Please note that in this specific outage, our Callback Assist system continued to make outbound connections. Customer voice and data connections to everywhere else should be unaffected. — John and Adam

Update: service was restored moments after this status message was posted. Agents are standing by ready to assist again. — John

Fiber DIA Intrusive Maintenance

On Friday, 8/30/2013, between the hours of midnight and 6AM, the telco will be performing intrusive maintenance on equipment serving some Fiber DIA customers in Northern California. Actual down time is expected to be less than three hours for affected customers.

-Tim

Update: The planned maintenance work was not performed last night due to unforeseen circumstances. A new date will be provided once the work has been rescheduled.

Customer MySQL Database Servers Maintenance

Tonight at Midnight, our customer database server will be taken off-line to replace two failed disks as well as repair some suspected bad cabling on the raid controller.  While the server is off-line any hosting accounts requiring database access will be off-line as well.  It is expected to take less that 30 minutes to complete the replacement and repair and have the system back in on-line and in service.

Services were restored on time just before 00:30.

-Kelsey

Support Closed Sunday

Our team will be taking time off this coming Sunday to celebrate Sonic.net’s 19th birthday.  Customer Support will be closed on 8/11/2013 from 11am to 6pm.  Support will re-open at 6pm on Sunday and remain available until 10pm.

Our normal 24x7x365 schedule is still applicable for our on-call network operations team for dedicated business and colocated customers.

-Adam and Support

FlexLink Equipment Maintenance

On Thursday the 8th, beginning at 12:00AM, we will be performing intrusive maintenance on equipment serving FlexLink Long Range customers in the Burlingame area. Expected downtime for affected customers is less than 15 minutes.

-Tim and Monroe

Update: We will also be performing maintenance on equipment serving a very small number of Private Ethernet Transport customers in the San Francisco area. Expected downtime for this operation is less than 10 minutes.

Customer Database Load

Due to excessive load on our customer database servers,  databases connection may intermittently time out. We are working to fix the problem and prevent further interruption.

Update: Currently both customer mysql server instances are off-line, we’re investigating what appears to be a hardware related issue and hope to have services restores shortly.

Update: It appears that these problems were caused by an issue with the raid controller not properly handling IO at anywhere near its expected performance levels.  A reboot seems to have restored performance back to normal.  We will continue to monitor the system closely.

 

 

NOC hotline unreachable

The NOC hotline for business customer support is presently experiencing call completion issues. We are working to diagnose and resolve this problem, but in the mean time please contact support at 707-547-3400 and request to be transferred to the NOC if necessary.  An update will be posted when this service interruption has been restored.

-Sonic.net NOC

Update: NOC hotline reachablity should now be restored. We are still working to determine the root cause of the issue.

Fusion/FlexLink Equipment Maintenance

This evening, beginning at 12:00AM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink Ethernet customers in the Windsor and Healdsburg areas. Expected downtime for affected customers is less than 15 minutes.

-Tim

Update: It has been determined that the software update done during this maintenance window has caused significant stability issues for some of the affected Fusion customers. We will be rolling back the software update this evening at 12:00AM and will be working further with our equipment vendor to determine the root cause of the stability issues.

Emergency Network Maintenance

Today, starting at around 9am, a software bug caused intermittent connectivity issues to telecom customers in the San Pedro area. An emergency reload of this equipment has stabilized all connections, and we will be working with our hardware vendors to ensure this does not happen again.

-Tomoc