Tue Mar 18 13:28:43 PST 2003 — Mail server load problems. The failure of the switch this morning triggered a cyclical failure of our already taxed mail cluster. We’ve been busy all morning doing what we can to assure smooth mail delivery and have, at this point, restored most services. Customers may experience some delays in delivery of email, however no email will or has been lost. Customers may also still encounter some difficulties sending or retrieving mail while we continue to work on the servers to improve the situation.
In order to increase NFS server performance we’ve added three more disks to the Netapp volume that handles mail. These additional spindles have already helped reduce mail server load. We’ve also added a secondary MX server to handle mail for sonic.net when our cluster is off line; this keeps the mail under our control and ensures more rapid delivery of delayed mail. -Kelsey, Nathan, Zeke, et al.