This evening at about 5:20 PM, a network reconfiguration caused our DNS resolver load to be shifted onto only one of our multiple load-balanced DNS servers. The server was unable to cope with the full load alone, and was providing sluggish DNS responses, causing Internet connections to appear down or sluggish. We resolved the issue at about 6:45 by re-balancing the DNS traffic. We apologize for this disturbance and will be revisiting our DNS server architecture soon.
-Jared, Kelsey and Nathan
Out of curiosity, how many DNS servers do you have? Do they hit their load limits on bandwidth or some hardware limit? (I’m a sysadmin… I am curious about these things 🙂
I have a list of ten name servers that I use to check to make sure things are consistent on our end. My understanding is that this problem was a CPU limit on a piece of equipment that was subsequently bypassed.