Mail Cluster Update.

Sun Apr 4 17:41:46 PDT 2004 — Mail Cluster Update. We’ve fixed the DoS ‘attack’ that was responsible for the recent instability problems with our mail cluster. Astute users may see the connection between the last two MOTDs. In order to restore stability to our mail cluster we finally broke down and disabled anti-virus filtering – with filtering on our POP servers, customers were still protected. This allowed the customer created mail bomb to pass through our systems, eventually filling up /home where the customer delivers their mail. Although a mail loop was expected they can be quite hard to track down. We will also be enforcing very high hard quotas to prevent a single user from filling all available space. The messages created by the mail loop contained deeply recursive MIME parts. These messages take a great deal of memory to scan since each part must be disassembled, decoded and scanned for viruses. This morning we modified our anti virus-software to reject deeply recursive MIME encoded messages before scanning them for viruses and reenabled anti-virus filtering on our MTAs. We’ve also been in contact with the developers of both projects that we use in regards to this problem.

Shell Users: The Operations department urges caution when forwarding mail, especially when this is done with procmail. If you are currently forwarding mail in procmail without loop detection please man procmailex to see examples on how to properly forward mail with procmail. Procmail is a very powerful tool and can do alot of damage to our systems. In an extreme case, misuse of procmail will be interpreted as a violation of our AUP. -Kelsey, Nathan and Scott

/home filled up.

Sun Apr 4 13:58:05 PDT 2004 — /home filled up. Someone managed to consume 25 Gigabytes of storage on our /home partition this morning. You may have seen “no space left on device” if you use our shell server. The problem was cleared up at 7:44am. -Scott

Mail Cluster Hiccup.

Sat Apr 3 19:01:11 PST 2004 — Mail Cluster Hiccup. The mail cluster that servers outbound email for mail.sonic.net failed this evening. Both dual Xeon servers needed to be rebooted to restore services. It’s believed that our anti-virus software took both servers out. -Operations

Mail alias changes.

Thu Apr 1 09:11:28 PST 2004 — Mail alias changes. We are making changes to email delivery to certain aliases. These aliases are of the form “username@pop.sonic.net”, “username@mail.sonic.net”, and “username@bolt.sonic.net”.

You may have received an email from us regarding this. The mailing was not based on actual use of these addresses, but rather, existing mail aliases in our system. Since almost all shell users have a “username@bolt.sonic.net” alias, customers who use shell.sonic.net (Bolt) would have received a notice.

The notice can safely be discarded if you aren’t using a “pop.sonic.net”, “mail.sonic.net”, or “bolt.sonic.net” address to receive email. Almost nobody is using these addresses anymore, but we thought we’d be safe and try to notify folks that these types of addresses would stop working. (This is yet another measure to reduce the influx of spam.) -Scott and Kevan

DNS Disturbance.

Wed Mar 31 15:16:44 PST 2004 — DNS Disturbance. For a short period of time at least one of our recursive name servers started to intermittently fail to properly resolve hostnames. The name server processes were restarted in turn on all of the name servers which seemed to clear up the problem. The cause of this failure is unknown. -Kelsey

Web server update.

Thu Mar 25 08:37:00 PST 2004 — Web server update. Thunder had a bad motherboard, which has been replaced and thunder also got a ram upgrade. Web services were not interrupted. -Geoff

Delayed email delivery.

Wed Mar 24 10:36:37 PST 2004 — Delayed email delivery. This morning one of our incoming mail servers developed a problem and stopped delivering mail. The server has been restored and mail service is back to normal, but some mail will be delayed. We expect to have locally queued email delivered shortly, and expect mail that was queued remotely to arrive within the next few hours. No email was lost during this brief event. -Operations

Bandwidth stats currently unavailable.

Wed Mar 24 10:18:44 PST 2004 — Bandwidth stats currently unavailable. Hedgehog, the machine which collects bandwidth usage and web traffic data and generates reports, has developed problems and is offline. Reports are currently not available. Data was not lost, and is still being collected, but reporting is down until further notice. Our operations staff is working to restore the machine; there is no ETR at this time. -Operations

Night Ops.

Wed Mar 24 21:42:48 PST 2004 — Night Ops. Tonight, at Midnight, we are going to be relocating a number of switches that serve our core ISP services to make room for additional equipment. Due to the redundant meshed configuration we should be able to do this with little to no service interruptions. Our shell server, bolt, however, is single homed off of one of the switches and will be briefly unavailable when it’s switch is moved. -Kelsey and Nathan

Web server update.

Tue Mar 23 22:16:34 PST 2004 — Web server update. One of our 3 redundant webservers, storm, is still out of service. It had severe disk problems and had to be rebuilt. This took all day and was just finished. It will be tested and placed back into service on Wednesday. -Kelsey, Nathan and Geoff