One of four NFS filers that supports our backend mail spool storage suffered from a broken FCAL loop and went off-line at 23:24. While the other three remained in service the mail cluster doesn’t take kindly to this situation and users may have noticed delays, timeouts or other errors while checking their mail. All services were completely restored by and operational by 23:56. The loop was most likely broken by a disk that had failed earlier this evening that was awaiting removal from the system in the morning. This is one of the rare failure scenarios that our clustered filers are not able to handle without manual intervention. -Kelsey and Don