Tue Apr 20 16:27:05 PDT 2004 — Spam Filtering Updates. The new default RBL set that we launched in conjunction with per-user MTA filtering controls have had a huge impact on the volume of spam that we accept for delivery to customers with little, or no, collateral damage. We’ve seen our mail flow to customers reduced from ~850 msg/min to less than 400 msg/min. This has significantly reduced load throughout our mail server complex. We didn’t stop here and are continuing to focus on additional methods to reduce the flood of spam.

On the 14th, we enabled a reflexive filter in the MTA that is rejecting between 110,000 to 180,000 known spams a day based off of signatures built off our own spam collection system in real-time. Due to limitations in the SMTP there is no way to make this filter optional. However, it is not needed as it is virtually impossible for a false positive hit to occur.

We are working to add some additional RBLs that users have requested, including some of the country based blackholes.us lists, and have a few other rules undergoing testing. Hopefully these will all be ready in the next few days.

We are also going to be enabling outbound email bulk detection in the next couple of days, possibly as early as tomorrow. The outbound bulk detection will begin to reject mail sent out through our servers once a unique or highly similar mail has been sent to more than a configured number of destination addresses. The bulk threshold is going to be initially set at 1000 and may change as need dictates. The outbound bulk detection goes a long way to ensure that our own outbound mail flow stays spam free and our mail servers stay out of blacklists when a customer’s computer or CGI is exploited to send spam through our network.

We are aware that our outbound bulk detection will block some legitimate uses of our mail servers. We strongly encourage customers who are running broadcast newsletters to set up and use a Sonic.net provided free mailing list which will not be so limited. Some users may be pleased to note that a mailman server is in the works to replace our majordomo sever. If using a free mailing list is not practical or possible, we can exempt customers from the bulk detection. For more information please contact support@sonic.net or visit news://news.sonic.net/sonic.antispam -Sonic.net Operations

Mon Apr 19 16:16:56 PDT 2004 — Occidental wireless service down. The Occidental Rooftop wireless service is offline. The tower is responding however, customers are not reachable. We are dispatching a technition with a new head-end unit to troubleshoot the problem. ETR is at least 1 hour. -Matt and Bryan

Sun Apr 18 22:04:54 PDT 2004 — SpamCop listing. For some time today both of our outbound mail servers were listed in bl.spamcop.net. Any mail sent by our servers to a remote server (including our own, where a user had bl.spamcop.net enabled) would have been bounced. The listing was in error and was related to an issue with the way SpamCop handles our known spam feed. SpamCop has assured us that this should not happen again. Unfortunately, due to DNS caching, even though our servers have been removed from the list it may be a day or two before all remote servers have forgotten about it. -Kelsey

Tue Apr 13 18:07:02 PDT 2004 — UUNet Scheduled Network Maintenance. On April 27th at 3AM MCI will be upgrading the software on our UUNet upstream router. Customers may notice loss of connectivity for approximately 30 seconds while BGP re-converges and traffic is routed via alternate paths. -Network Operations

Tue Apr 13 16:11:00 PDT 2004 — Microsoft Security Updates for April 2004. Microsoft has released a number of new critical updates for Windows systems, and Sonic.net customers who use Windows should run Windows update to keep their systems secure. For full information on the current updates, see:

go.microsoft.com/?LinkID=466770

To update your system, select “Tools / Windows Update” in Internet Explorer, or visit the following URL: windowsupdate.microsoft.com/

If you have not already done so, turning on the critical update notification facility is a very good idea. Automatic updates can also be set up, and your system will be updated and secured automaticly. -Dane

Mon Apr 12 21:01:29 PDT 2004 — NFS disk failure. Today marks icebox, our NFS filer’s, 365 days uptime. In honor of the occasion, icebox failed a disk this evening. From 8:19 to 8:35 services that depend on icebox were down. This includes web hosting, SSL, and our internal SQL. Icebox auto-recovered and is rebuilding onto the hot spare disk. – Operations

Sat Apr 10 20:13:16 PDT 2004 — Multicast and IPv6 Outage. The router that serves both multicast and IPv6 services locked up approximately 15 minutes ago. A reboot brought the device back up. -Nathan and Justin

Thu Apr 8 18:30:59 PDT 2004 — Updated Email Filtering. We have employed a multi layered system to fight spam for quite some time. While SpamAssassin and Virus filtering have gotten all of the recent attention, we have also used the MAPS RBLs, or real-time blacklists, to reject mail sent from known spam sources.

In response to the ever increasing volume of spam that has been reaching our customers inboxes we enabled use of two additional blacklists earlier today. The SBL and XBL from spamhaus.org have reduced the total volume of email we accept for customers by nearly 50%.

We have also been hard at work improving our own local blacklists to target many Cable and DSL provider’s dynamic clients – the biggest source of spam today. There was a brief error with this list earlier today where a number of large blocks were incorrectly added resulting in the bouncing of some legitimate senders.

In addition to these steps, we’ve also launched a new member tool which users can use to choose additional RBLs to be used to filter their email or opt entirely out of our MTA level filtering. Please use caution with this tool, some of the RBLs provided are very aggressive and will block legitimate Email. Unless you are familiar with the RBL’s listing policy and understand the ramifications it’s best to leave the settings at default. The tool is available at sonic.sonic.net/membertools/spamlist.pl

Please see news://news.sonic.net/sonic.antispam for a complete discussion of the new filters and tool. -Sonic Operations

Mon Apr 5 14:40:07 PDT 2004 — Hard Disk Quotas Enabled. We’ve enabled hard disk quotas on all major service directories for user to eliminate the ability for a single user to consume all available disk space. We’ve configured the quotas with a high upper limit of 2G per tree. If you require more space than this the quota can be overridden for an agreed upon fee. -Sonic Operations

Sun Apr 4 17:41:46 PDT 2004 — Mail Cluster Update. We’ve fixed the DoS ‘attack’ that was responsible for the recent instability problems with our mail cluster. Astute users may see the connection between the last two MOTDs. In order to restore stability to our mail cluster we finally broke down and disabled anti-virus filtering – with filtering on our POP servers, customers were still protected. This allowed the customer created mail bomb to pass through our systems, eventually filling up /home where the customer delivers their mail. Although a mail loop was expected they can be quite hard to track down. We will also be enforcing very high hard quotas to prevent a single user from filling all available space. The messages created by the mail loop contained deeply recursive MIME parts. These messages take a great deal of memory to scan since each part must be disassembled, decoded and scanned for viruses. This morning we modified our anti virus-software to reject deeply recursive MIME encoded messages before scanning them for viruses and reenabled anti-virus filtering on our MTAs. We’ve also been in contact with the developers of both projects that we use in regards to this problem.

Shell Users: The Operations department urges caution when forwarding mail, especially when this is done with procmail. If you are currently forwarding mail in procmail without loop detection please man procmailex to see examples on how to properly forward mail with procmail. Procmail is a very powerful tool and can do alot of damage to our systems. In an extreme case, misuse of procmail will be interpreted as a violation of our AUP. -Kelsey, Nathan and Scott