We’ve been working on several improvements to our recursive DNS cluster configs to improve performance across the board and better support network growth in new regions beyond our existing service foot print in Northern and Southern California and have rolled out several config changes to the DNS proxies that handle ns1 and ns2.sonic.net over the past week. What we believed was to be the last of those changes was pushed out this afternoon to the entire fleet after having cooked properly on a few systems at 3:15PM. After that change was pushed, a significant portion of IPv6 DNS requests appeared to be black holed by some of the servers. The issues continued until about 3:46PM. We are still unclear on the root cause of this but all services are currently stabilized and running as expected at this time. We will continue to investigate in the hope that we can identify the cause, it seems possible it could be a bug in the dns specific load balancing software itself.
It is worth noting that our expectation was that most clients would have both v6 and v4 servers configured but it is evident that is not the case and it is likely that the majority of v6 enabled clients on our network with no fail over to v4 requests. If you have static configured name servers, we’d suggest you list both the v6 and v4 address listed below.
2001:5a8::11
2001:5a8::33
50.0.1.1
50.0.2.2
-Kelsey, William and the rest of Systems.
I had a service interruption at about that time, but I don’t think I use IPv6. I couldn’t connect to NYTimes.com, user.well.com, or to a crossword puzzle page. It resolved after I restarted my router.