Month: November 1998

Starting around 8pm or so, our mail server…

Thu Nov 19 21:42:15 PST 1998 — Starting around 8pm or so, our mail server was severely impacted by a tremendous influx of email. Someone spammed AOL from a forged domain name, and unfortunately, we host that domain. This means that the tens of thousands of bounce messages were returned to the domain owner’s mailbox. [As of Fri Nov 20 00:31:16, I’ve counted 23,000 messages so far. /sd]

That wouldn’t have been a problem — except, AOL’s mail server farm was all cooperating to deliver these messages. In other words: thundering herds of AOL mail servers stampeded Sub, our mail server. We resolved that by turning off connectivity between AOL’s server farm’s networks and us, and then slowly bringing up connectivity, allowing traffic to clear one network at a time. I’ll post more about that on news:sonic.general. -Scott

UUNet experienced a major outage across their

Sun Nov 8 17:45:57 PST 1998 — UUNet experienced a major outage across their backbone from ~9am until ~3pm today. Discussion on a network operators’ mailing list intimates that this may have been due to software bugs in two vendors’ routers — bugs which reenforced each other destructively, causing massive instability on the Internet’s global routing mesh. There are reports of instability on other backbones, but from our neck of the Internet woods, UUNet seemed hardest hit. We were able to shut down our connection with UUNet, and balance traffic between our Sprint link and CW (formerly MCI) link. If memory serves, the last Internet outage of this magnitude occured in May of 1997, when a new exchange point on the East coast created a routing loop in the global routing mesh. (They had passed BGP routes between borders using RIP, with exciting results: their router announced that it was easiest to reach a large chunk of the Internet through their exchange point.) We here at Sonic hope that the effects of today’s outage didn’t cause too much frustration; we will be evaluating the outage to determine how we can prevent those effects in future. -Scott & Dane