Author: admin

Public MySQL downtime: Our public MySQL…

Mon Dec 9 15:25:41 PST 2002 — Public MySQL downtime: Our public MySQL crashed under some normal maintenance. The databases are currently undergoing consistency checking. The server will be brought back up as soon as it is finished. This also affects both of our web-based mail clients, TWIG and Squirrel Mail. -Kelsey

SSL problem.

Sat Dec 7 12:17:44 PST 2002 — SSL problem. Earlier this morning, our SSL web server stopped answering requests. Restarting the web server program cleared up the problem. The server now restarts daily at 4am to keep it healthy. -Scott and John

Spamassassin hiccup.

Fri Dec 6 19:57:42 PST 2002 — Spamassassin hiccup. One of our two spamassassin servers had a problem, and spam was making it through for spamassassin-sql-enabled customers. This did not affect file-based spamassassin users. We believe the problem is corrected, and are monitoring the situation. -Scott and Dane

DB replacing NIS.

Fri Dec 6 16:26:34 PST 2002 — DB replacing NIS. We have begun deploying Berkeley DB hashes across systems that normally use NIS for their system databases. Currently the mail servers and the spamassassin servers are using DB, with NIS as a backup. This change means improved performance for services like popmail and sendmail, as well as more robust performance when the servers are under high load. -Scott and Kelsey

Broadlink tower down.

Thu Dec 5 22:01:44 PST 2002 — Broadlink tower down. It appears we have a tower down at Broadlink. We have contacted them, and hopefully we will have a speedy resolution. -Scott and Dane

Update: Power was lost to the top three floors of the tower’s building. ETR is 9am this morning. -Scott, and Tim from Broadlink

Update: Power was restored at 9am, but the Broadlink equipment didn’t come back up. After a few hours of troubleshooting, the tower is back up. Scott, Dane, and Tim and Jason from Broadlink

Intermittent problems with mail servers.

Thu Dec 5 08:36:23 PST 2002 — Intermittent problems with mail servers. Our mail server cluster was occasionally returning “connection refused”. Two senior engineers are working on the problem. We have implemented a solution and the mail servers seem to have calmed down. We are still evaluating the cause of the trouble. -Scott and Russ

707-522-1003 was giving intermittent busy…

Tue Dec 3 19:04:37 PST 2002 — 707-522-1003 was giving intermittent busy signals this evening due to the failure of a modem card. The card was rebooted and is now working correctly.

We are implementing a system to monitor the modems and automatically reboot them if needed in order to avoid giving busy signals. -Russ

Shell Server Downtime: Our shell server…

Mon Dec 2 00:10:39 PST 2002 — Shell Server Downtime: Our shell server mysteriously ‘turned off’ 11:11 PM tonight. This is the second time that we’ve seen this happen and, although it has an excellent dual hot swap power supply, we believe that it may be faulty and need replacing. We’ll be keeping a close eye on the server and if it fails again, we’ll swap out the power supply with a spare unit. -Kelsey

Sebastopol POP trouble.

Sun Dec 1 09:25:07 PST 2002 — Sebastopol POP trouble. We are experiencing problems with our Sebastopol POP. It appears to be a Pac Bell outage as T1’s and PRI in the area are down. If you use a dial-up number ending in 8812, please try an alternate number. We provide multiple dial-up numbers in most areas and they can be found at www.sonic.net/cgi-bin/pops.pl -Matt

Update: Sun Dec 1 10:31:09 PST 2002 — Connectivity restored. Pac Bell traced the trouble to a power outage in one of their facilities and fixed the problem. All service has been restored. -Matt