Thu Aug 24 13:20:49 PDT 2000 — We have had a system failure that led to lost customer email. Beginning at 9:28:53 AM Thu Aug 24 2000 and ending at 12:15:00 PM, three out of our redundant array of five email servers were misconfigured, and silently discarded customer email. Lost messages were not bounced back to the sender, and cannot be recovered. We are shocked by this event, and have eliminated the configuration which caused this. We will be informing all senders and all recipients of each lost email message with full information including the date, time, sender and recipient of each message so that they can be re-sent.
We sincerely apologize for the error, and the inconvenience that it causes.
In addition to elimination of the configuration setup which caused this, we’re committed to deployment of a testing engine which pro-actively sends and receives email through each of the mail servers individually on an ongoing basis. By testing each and every server independently every few minutes, both locally at Sonic.net and from off site, we can stay better informed about any possible loss or delay which might affect our customers email delivery. This is similar to our existing ‘ckhosts’ and ‘ckdisk’ tools which check for availability of web, mail, ftp, Usenet news services and disk space currently.
These tools page our staff of system administrators with text messages anytime a server becomes unresponsive or unavailable, and this allows us to deliver high availability for your Internet services. We will followup here in the MOTD once this testing engine is deployed.
If you suspect that you may have lost email, please stand by for a notification. -Dane, Scott and Kelsey