Category: Uncategorized

Santa Rosa Datacenter UPS Maintenance

The UPSes for our Santa Rosa Datacenter are scheduled for their regular maintenance on Tuesday, Feb. 14th starting at 10:00AM.  Maintenance is fully scripted with our vendor and should not impact any services.  However, the maintenance does involve bypassing the UPS and running loads directly on utility power.  The backup generator will be kept running for the duration of the maintenance to protect against utility power failure.

– Sonic Facilities and System Operations

San Francisco Fusion Fiber Outage

Today, starting at 12:08am, equipment serving a subset of Fusion Fiber customers in the San Francisco area appears to have suffered a hardware failure. We are working to replace the equipment and restore service as quickly as possible.

Update(2:40am): ETR 2 hours

Update(4:59am): Affected equipment has been replaced and we believe service has been restored to all affected customers

-Tomoc

System Maintenance

Update: Maintenance complete.

Tonight at 11:59pm, System Operations will be running maintenance updates on several customer facing systems. The following services may experience brief interruptions:

  • Customer hosted websites

The maintenance period is expected to last 1 hour.

-SOC

Authoritative Name Server Migration

Sonic maintains three geographically diverse and redundant authoritative domain name servers, in NY, CA and TX.  These servers are used for our own domains as well as customers who have domains hosted on our network.  Early this morning, c.auth-ns (NY), had a complete hardware failure and we took the opportunity to move services to a different colocation provider in NJ later today.  The failure and subsequent migration should have had no noticeable impact on any of our services.

-Dexter and Kelsey

 

Phone Switch Maintenance

Update: 3:10  Maintenance is complete.

 

We will be performing scheduled maintenance on our phone switch tonight at midnight. We do not expect any service impact by this maintenance.

 

Network Engineering

Enterprise SSE Fiber circuit

Update: 1/23/2017 – 4:05pm – A bug has been identified in the vendors network that is related to the CPE that is put on site and the router in the vendors core network.  Our vendor has been issuing upgrades to the CPE which has the potential expose this bug on the core router, causing traffic to loop.  When this occurs, the traffic is perceived as BUM traffic and is policed causing speeds to be capped at 2 Mbps.  We are working with our vendor to create solutions to help identify and act quickly when this bug happens.

Update: 1/23/2017 – 9:58am – We are continuing to actively work with our vendor’s advanced Engineering and Management teams to resolve the remaining affected customers. Please report outstanding SSE service impacting issues to NOC for us to investigate.

Update: 1/22/2017 – 3:30pm We are receiving some more reports this afternoon regarding this issue being unresolved for a portion of our customers and have escalated with our vendor.

Update: 1/21/2017 – 8:30pm We believe this issue has been located and fixed at this point. Please contact the NOC if you are experiencing any further trouble.

Update: 1/21/2017 – 5:40pm We are continuing to escalate and work with our vendor on this issue.

Update: 1/21/2017 – 11:22am We are still working with high level escalation with our vendor.  Currently they are tracing aggressively through their network in an attempt to locate any issues or anomalies.

Update: 1/21/2017 – 6:46am We are working direct with upper management and 2 different teams.  We will update soon when they get some more data to us.

Update: 10:38pm – Escalation has moved to a higher level department and our issue is currently being worked.  We are awaiting the results of their testing.  No ETR.

Update: 8:11pm – We are still on working with our vendors and have escalated the issue again. There is currently no ETR.

Update: 4:45pm – Our issue is currently being worked by our vendors Engineering staff, they suspect an issue in the core of their network.  No ETR yet.

Update: 2:09pm – We are still working with our vendor and have assisted in narrowing down the issue. There is currently no ETR.

Update: 10:39am – We are working with our fiber vendor to isolate and resolve this issue. There is currently no ETR. As soon as we have one or know more, this post will be updated.

~ Network Operations

We are investigating reported packet loss and high latency spikes on our enterprise SSE fiber circuits.  We will update this once we know more.

~ Network Operations

Santa Rosa Data Center – Outage

Beginning tonight at around 4:50PM we experienced a large DDoS attack towards a colocation customer of ours. During this attack you may have noticed little or no connectivity to Sonic members services such as email or members tools and an inability to phone into our support department. The attack is still ongoing but we believe services have been fully restored as of about 7:00PM.

Network Engineering

Santa Rosa Data Center – Outage

Around 8:04pm Sonic experienced an NTP attack to our data center – taking down connectivity to services hosted within.  We have put in place measures to prevent the attack from happening further and service stability has been restored.

-Network Engineering

Voice Equipment Maintenance

Update (12:59AM): Maintenance is complete. Due to some unexpected complications, customers may have experienced interruptions in voice service between 12:08 and 12:13AM.

Today at 4:26PM we experienced an equipment failure upstream of our core voice network. This device is redundant so customer impact for this event should have been minimal. Tonight beginning at midnight we will be replacing this hardware. We do not expect this to cause any downtime.

-Network Engineering