Thu Apr 29 12:41:50 PDT 1999 — We’ve had a quite few server issues that have affected customers in the past 24 hours. Here’s an overview:
Storm.sonic.net, aka www.sonic.net crashed with a kernel panic last night around 9pm last night. Dustin was on the premises and rebooted it, and it was down for a total of about 30 minutes.
At 4am last night the FTP server got backlogged with lots of people in Japan downloading MP3 files from user ftp space, and the ftp server was failing to answer due to the large load. We’ve removed or changed permissions on the pirated music, and have sent warnings to the customers who were publishing this copywritten material. Please don’t do that, we prefer to sleep at night.
At 4:30am news.sonic.net’s INN server exited, and Scott and I restarted it within 10 minutes – if you were reading news at that hour, you would have noticed, but most of you were sleeping.
At around 6am, a dialup user at another ISP tried to brute force guess the email password of one of our customers, and the resulting logged failures filled up our mail server’s disk drive to a point where new mail couldn’t be received. We have modified our disk use monitoring script so that the tool will page us before we reach the user full threshold – we actually had space, but only in the root reserved percentage, so new mail was backlogged. This modification to the paging system will prevent this problem in the future.
The mail server disk space issue has caused inbound email to be backlogged – we’re receiving a large volume of email now which was sent between 6am and 10am this morning. Sorry for the delay in receipt of these emails!
Lots of little issues – we’re on the job, and keeping things running here. -Dane, Scott, Steve and Eli