Fri Mar 16 16:56:35 PST 2001 — Unexpected Network Outage. We have been working with Extreme Networks to troubleshoot and identify some problems that we’ve had with VLANs in their switches not behaving as expected under certain circumstances. We were gathering a packet dump to help their engineers resolve the problem when we unexpectedly caused a switch loop between giga and ape (our two core switches.) We were able to resolve the loop in a matter of seconds but our Alteon load balancing switch decided to down it’s interfaces into ape making all web and email servers unavailable. It took a few minutes to coax the Alteon to bring it’s interfaces back up and then a few minutes more for the server network to converge.
All locally served web sites and email services were offline for five minutes. No other services were affected. -Russ, Nathan and Kelsey