Sat Apr 10 20:13:16 PDT 2004 — Multicast and IPv6 Outage. The router that serves both multicast and IPv6 services locked up approximately 15 minutes ago. A reboot brought the device back up. -Nathan and Justin

Thu Apr 8 18:30:59 PDT 2004 — Updated Email Filtering. We have employed a multi layered system to fight spam for quite some time. While SpamAssassin and Virus filtering have gotten all of the recent attention, we have also used the MAPS RBLs, or real-time blacklists, to reject mail sent from known spam sources.

In response to the ever increasing volume of spam that has been reaching our customers inboxes we enabled use of two additional blacklists earlier today. The SBL and XBL from spamhaus.org have reduced the total volume of email we accept for customers by nearly 50%.

We have also been hard at work improving our own local blacklists to target many Cable and DSL provider’s dynamic clients – the biggest source of spam today. There was a brief error with this list earlier today where a number of large blocks were incorrectly added resulting in the bouncing of some legitimate senders.

In addition to these steps, we’ve also launched a new member tool which users can use to choose additional RBLs to be used to filter their email or opt entirely out of our MTA level filtering. Please use caution with this tool, some of the RBLs provided are very aggressive and will block legitimate Email. Unless you are familiar with the RBL’s listing policy and understand the ramifications it’s best to leave the settings at default. The tool is available at sonic.sonic.net/membertools/spamlist.pl

Please see news://news.sonic.net/sonic.antispam for a complete discussion of the new filters and tool. -Sonic Operations

Mon Apr 5 14:40:07 PDT 2004 — Hard Disk Quotas Enabled. We’ve enabled hard disk quotas on all major service directories for user to eliminate the ability for a single user to consume all available disk space. We’ve configured the quotas with a high upper limit of 2G per tree. If you require more space than this the quota can be overridden for an agreed upon fee. -Sonic Operations

Sun Apr 4 17:41:46 PDT 2004 — Mail Cluster Update. We’ve fixed the DoS ‘attack’ that was responsible for the recent instability problems with our mail cluster. Astute users may see the connection between the last two MOTDs. In order to restore stability to our mail cluster we finally broke down and disabled anti-virus filtering – with filtering on our POP servers, customers were still protected. This allowed the customer created mail bomb to pass through our systems, eventually filling up /home where the customer delivers their mail. Although a mail loop was expected they can be quite hard to track down. We will also be enforcing very high hard quotas to prevent a single user from filling all available space. The messages created by the mail loop contained deeply recursive MIME parts. These messages take a great deal of memory to scan since each part must be disassembled, decoded and scanned for viruses. This morning we modified our anti virus-software to reject deeply recursive MIME encoded messages before scanning them for viruses and reenabled anti-virus filtering on our MTAs. We’ve also been in contact with the developers of both projects that we use in regards to this problem.

Shell Users: The Operations department urges caution when forwarding mail, especially when this is done with procmail. If you are currently forwarding mail in procmail without loop detection please man procmailex to see examples on how to properly forward mail with procmail. Procmail is a very powerful tool and can do alot of damage to our systems. In an extreme case, misuse of procmail will be interpreted as a violation of our AUP. -Kelsey, Nathan and Scott

Sun Apr 4 13:58:05 PDT 2004 — /home filled up. Someone managed to consume 25 Gigabytes of storage on our /home partition this morning. You may have seen “no space left on device” if you use our shell server. The problem was cleared up at 7:44am. -Scott

Sat Apr 3 19:01:11 PST 2004 — Mail Cluster Hiccup. The mail cluster that servers outbound email for mail.sonic.net failed this evening. Both dual Xeon servers needed to be rebooted to restore services. It’s believed that our anti-virus software took both servers out. -Operations

Thu Apr 1 09:11:28 PST 2004 — Mail alias changes. We are making changes to email delivery to certain aliases. These aliases are of the form “username@pop.sonic.net”, “username@mail.sonic.net”, and “username@bolt.sonic.net”.

You may have received an email from us regarding this. The mailing was not based on actual use of these addresses, but rather, existing mail aliases in our system. Since almost all shell users have a “username@bolt.sonic.net” alias, customers who use shell.sonic.net (Bolt) would have received a notice.

The notice can safely be discarded if you aren’t using a “pop.sonic.net”, “mail.sonic.net”, or “bolt.sonic.net” address to receive email. Almost nobody is using these addresses anymore, but we thought we’d be safe and try to notify folks that these types of addresses would stop working. (This is yet another measure to reduce the influx of spam.) -Scott and Kevan

Wed Mar 31 15:16:44 PST 2004 — DNS Disturbance. For a short period of time at least one of our recursive name servers started to intermittently fail to properly resolve hostnames. The name server processes were restarted in turn on all of the name servers which seemed to clear up the problem. The cause of this failure is unknown. -Kelsey

Thu Mar 25 08:37:00 PST 2004 — Web server update. Thunder had a bad motherboard, which has been replaced and thunder also got a ram upgrade. Web services were not interrupted. -Geoff

Wed Mar 24 10:36:37 PST 2004 — Delayed email delivery. This morning one of our incoming mail servers developed a problem and stopped delivering mail. The server has been restored and mail service is back to normal, but some mail will be delayed. We expect to have locally queued email delivered shortly, and expect mail that was queued remotely to arrive within the next few hours. No email was lost during this brief event. -Operations