Thu Oct 23 09:43:39 PDT 2003 — Mail Delays: Freezer, the NetApp filer that handles among other things, ‘/home’, had a disk failure this morning. The filer behaved properly, failed the disk and began to rebuild onto a spare. However, the failed disk continued to cause problems for the filer, but triggering repeated bus resets. Meanwhile, the SpamAssassin cluster, which relies heavily on ‘/home’ suffered terribly. After removing the failed disk from freezer things are starting to get better but it will still take some time before mail delivery is returned to normal and all queued mail is delivered. At this time, we have disabled SpamAssassin in order to allow mail delivery to resume. -Kelsey
Update: 12:15:32 PDT — All services have been restored: Freezer has finished rebuilding it’s raid, the SpamAssassin servers are back online and all of the back-logged mail queues have been processed. -Kelsey, Nathan, Jared and Russ