Category: Uncategorized

Sonic will be closing early today, 12/11/2015 from 6pm to 10pm for our annual staff holiday party. We will return Saturday to our normal working hours. Our 24×7 network operations team will still be on call for our Enterprise, dedicated circuit and colocation customers.

One of the two pairs of storage clusters we use for email storage had a cluster takeover event at 11:09 this AM.  One of the heads lost communication with all of its disks and its partner successfully took over all of its services without any interruption.  However, while tracing the failure of the first head to a failed FCAL optics package – which was replaced – the cluster interconnect adapter in the second head locked up and triggered a panic.  This panic lead to a brief interruption if POP/IMAP services at approximately 12:00.  The second filer rebooted successfully still in partner takeover.

Unfortunately, since the second filer is still in takeover mode and can’t see the cluster interconnect adapter, the most conservative resolution requires that we halt the second filer and replace the failed adapter.  We’ve tentatively scheduled this for tomorrow after midnight provided that we receive the replacement adapter from our vendor in time.  POP/IMAP services will be offline for the duration of the maintenance which should take less than an hour to complete.

-Kelsey and William

Update 01:00: The cluster interconnect adapter has been replaced and all services have been fully restored.

-Kelsey and William

Tonight, November 25, at 1:00am, we will be performing a maintenance reload of core network equipment serving our Santa Rosa data center. No service interruption is expected, however in the worst case scenario customers may experience a brief period of routing instability towards colocation customers and Sonic services such as email and DNS.

-Tomoc

Ops has observed and fixed a problem with ftp.sonic.net that would have caused problems for users from 5:45pm to 8:50pm this evening. A networking issue between our VM cluster and the storage backend was to blame, and we are doing what we can to prevent an outage of this nature in the future.

– SOC

This evening, November 18, starting around 5:00pm, Amazon began blackholing traffic towards the Sonic network. We attempted to route through a different upstream provider, but it appears the routing issue was too deep into the Amazon network. We believe they have fixed the routing issue, and all traffic has been restored as of 5:25pm.

Update: This issue appears to have re-surfaced. We are reaching out to Amazon to determine the cause of the outage and do everything we can to ensure it does not happen again.

-Tomoc and the NOC

Today, Oct 28th from about 3:45pm to 3:55pm we had a voice routing problem that affected inbound and outbound calling. The issue has been resolved.

-Brandon

Customers that have Sonic’s Business Virtual Fax service are experiencing an outage.  We are working with our vendor to figure out what happened and to restore service as soon as possible.  We will update you as things progress.  Sorry for the inconvenience.

-Brandon

Update (2:09pm)  We are currently working to restore virtual faxing and should have the issue resolved within the next 30-45 mins.

Update(5:17pm) There were issues programming numbers that has been since fixed.  We have made successful faxes to test lines.  We are currently adding and testing all fax lines that previously existed.  We will update when all lines have confirmed to be working as expected. ETR ~1hr

Update(6:16) All Virtual Fax lines are programmed in, and tested.

We are experiencing a connectivity problem that is affecting Fusion customers in the Pittsburg area. We are dispatching technicians to troubleshoot the problem with an ETA of 10:00am

-Michael

Update:  The router encountered a software defect which caused the router to stop responding.  Our technicians restarted the router and service has been restored.

-Brandon

At 8:30 am we started experiencing a large DDOS attack, causing most of our systems to become unaccessable. Our Network Operations Center is currently working to resolve the issue.

Update:9:40 am

We have blocked the large DDOS attack to our Santa Rosa Data Center. All services are back up.

Update: 10:45 AM

We have experienced some further trouble from this attack and have blocked the malicious traffic at this point.

NOC

Today at 8:35 there was a network configuration error that caused a brief interruption of service for some of our Flexlink Long Range customers served out of SNRSCA01. The interruption lasted for about 10 minutes.

–Michael