Public MySQL downtime: Our public MySQL…

Mon Dec 9 15:25:41 PST 2002 — Public MySQL downtime: Our public MySQL crashed under some normal maintenance. The databases are currently undergoing consistency checking. The server will be brought back up as soon as it is finished. This also affects both of our web-based mail clients, TWIG and Squirrel Mail. -Kelsey

SSL problem.

Sat Dec 7 12:17:44 PST 2002 — SSL problem. Earlier this morning, our SSL web server stopped answering requests. Restarting the web server program cleared up the problem. The server now restarts daily at 4am to keep it healthy. -Scott and John

Spamassassin hiccup.

Fri Dec 6 19:57:42 PST 2002 — Spamassassin hiccup. One of our two spamassassin servers had a problem, and spam was making it through for spamassassin-sql-enabled customers. This did not affect file-based spamassassin users. We believe the problem is corrected, and are monitoring the situation. -Scott and Dane

DB replacing NIS.

Fri Dec 6 16:26:34 PST 2002 — DB replacing NIS. We have begun deploying Berkeley DB hashes across systems that normally use NIS for their system databases. Currently the mail servers and the spamassassin servers are using DB, with NIS as a backup. This change means improved performance for services like popmail and sendmail, as well as more robust performance when the servers are under high load. -Scott and Kelsey

Broadlink tower down.

Thu Dec 5 22:01:44 PST 2002 — Broadlink tower down. It appears we have a tower down at Broadlink. We have contacted them, and hopefully we will have a speedy resolution. -Scott and Dane

Update: Power was lost to the top three floors of the tower’s building. ETR is 9am this morning. -Scott, and Tim from Broadlink

Update: Power was restored at 9am, but the Broadlink equipment didn’t come back up. After a few hours of troubleshooting, the tower is back up. Scott, Dane, and Tim and Jason from Broadlink

Intermittent problems with mail servers.

Thu Dec 5 08:36:23 PST 2002 — Intermittent problems with mail servers. Our mail server cluster was occasionally returning “connection refused”. Two senior engineers are working on the problem. We have implemented a solution and the mail servers seem to have calmed down. We are still evaluating the cause of the trouble. -Scott and Russ

707-522-1003 was giving intermittent busy…

Tue Dec 3 19:04:37 PST 2002 — 707-522-1003 was giving intermittent busy signals this evening due to the failure of a modem card. The card was rebooted and is now working correctly.

We are implementing a system to monitor the modems and automatically reboot them if needed in order to avoid giving busy signals. -Russ

Shell Server Downtime: Our shell server…

Mon Dec 2 00:10:39 PST 2002 — Shell Server Downtime: Our shell server mysteriously ‘turned off’ 11:11 PM tonight. This is the second time that we’ve seen this happen and, although it has an excellent dual hot swap power supply, we believe that it may be faulty and need replacing. We’ll be keeping a close eye on the server and if it fails again, we’ll swap out the power supply with a spare unit. -Kelsey