Category: DNS

Northern California Network Core Maintenance

Update(1:10AM): Maintenance successfully completed.

Tomorrow (9/17/20) at 11:59PM we will be performing maintenance on core network hardware in our Santa Rosa data-center.  We expect this maintenance to be non-intrusive, but has the potential to impact a wide range of enterprise and Sonic hosted services.  The maintenance window will be 4 hours.

– Network Engineering

Sonic Data Center Core Maintenance – Night 2 of 2

Update(1:50AM) Maintenance ran long due to unforeseen hardware issues, but all work has been been successfully completed.

Wednesday, January 8, starting at 11:59pm, we will be performing maintenance on core equipment in our Santa Rosa HQ Data Center. Minimal customer impact is expected, with brief periods of routing instability towards some Sonic services, such as web-hosting, email and Member Tools. This maintenance window is 4 hours.

-Network Engineering

Sonic Data Center Core Maintenance – Night 1 of 2

Update(1:20AM) All work for tonight has successfully concluded.

Tuesday, January 7, starting at 11:59pm, we will be performing maintenance on core equipment in our Santa Rosa HQ Data Center. Minimal customer impact is expected, with brief periods of routing instability towards some Sonic services, such as web-hosting, email and Member Tools. This maintenance window is 4 hours.

-Network Engineering

San Francisco Collocation Power Maintenance

This Saturday (6/9) starting at 5AM, one of the San Francisco data centers we collocate equipment, will be performing a planned power maintenance. Although we have been assured by the colocation provider this work will be non-impactful, we have taken additional preparations to verify redundancy on all critical hardware, and will have a network engineer on site as a fail-safe.  The maintenance window officially extends until 6PM, and we will provide update when all work is completed.

– Network Engineering

(Update 8:12am) – We had a Power supply fail in a piece of our equipment during the first phase of the datacenter power maintenance.  We quickly swapped out the cabling to restore the unit.  Legacy DSL customers would have been impacted but service should now be restored.

 

Update(1:03pm) Power maintenance complete.

intermittent dns failure

Between 5:00 pm yesterday and 9:00 am today, customers may have experienced intermittent DNS failures or slower than normal name resolution. At 9:00 am this morning we noticed a configuration failure on one of our name server clusters. We immediately disabled the cluster which allowed traffic to flow over to our other redundant cluster. We have since addressed the issue and restored the cluster to working service. We are currently investigating our monitoring procedures to identify why this issue wasn’t detected earlier and to make sure it doesn’t happen again. We apologize for any inconvenience this may have caused.

– William & Kelsey

Updates to Our Recursive DNS Servers

On the morning of  Tuesday the 22nd, we will block access to our recursive DNS servers from off of our network.  This will help reduce the use of our DNS servers as part of the ongoing DNS Amplification Attacks as well as protect their quality of service for all of our customers.  This will, however, cause issues for any roaming device that has our DNS servers hard coded.  These devices should be configured to use server assigned DNS settings instead.  For more information please see our forums.

-Kelsey and William

 

Slow DNS responses this morning

This morning approximately between 6:30 AM and 7:00 AM, users may have noticed slow or dropped responses to our recursive dns servers. Further investigation discovered this interruption was caused by a small reflective DOS attack. We are actively working on a solution to fix these DOS attacks that affect our dns servers and hope to have a solution soon. We apologize for not getting this notice out sooner when the problem occurred.

-William and the SOC

Unreachable Sonic.net Services

This morning, from around 2:10AM to 2:25AM, a large DDoS attack targeting an IP within our Santa Rosa datacenter caused reachability problems to some Sonic.net member and hosting services. The attack has been blocked and all reachability restored.

– Robbie

DNS Service interruption.

DNS service was interrupted to our ns1.sonic.net name server cluster at 5pm PST for approximately 10 minutes.

All service has been restored and we are investigating the root cause of the problem.

–Augie

Update: Another brief service interruption today ( 21 March ) at 4:31pm PST for approximately 7 minutes as we were updating our name servers. Service was restored at 4:38pm PST.

DNS resolution problem.

Today at 3:45pm PST we suffered a DNS outage to one of our two DNS clusters, service was restored to the cluster at 3:51pm.

All services are functioning as expected at this time, and we are investigating the cause of the outage.

–Augie