Thu Dec 4 17:40:30 PST 2003 — Focal Router Maintenance: Tonight at Midnight we are going to working on the router at Focal. Users that are not directly connected to the focal router may notice degraded performance for the duration of the maintenance, while those that dial into the Focal pop will be isolated from the rest of our network. If everything goes as planned, the maintenance window should last for less than 10 minutes. -Nathan, John and Kelsey.

Wed Dec 3 16:25:28 PST 2003 — Upgrades and Changes to custmx: We upgrade the RAM and disks in our customer secondary MX server today to better handle the load on the box. We also enabled use of mail-abuse.org’s RBL+, a combination of the RBL, DUL, RSS and OPS, to help reduce the spam relayed by the server. These changes only affect users that run their own mail server and use our server as a backup MX handler. -Kelsey

Tue Dec 2 07:54:35 PST 2003 — Problems With the new Kernels: Our web cluster has been having stability problems with the new kernel. www.sonic.net and all of our hosted domains web sites have been having reachability issues since early this morning. We’re working to resolve the problems to assure the stability and security of the web cluster as quickly as possible. -Matt and Kelsey

Update: Tue Dec 2 22:48:52 PST 2003 — Most of the problems were solved by 9am, though some kernel issues emerged throughout the day. It appears that Kelsey and Nathan have the problem licked, but we are still monitoring the situation. -Scott

Mon Dec 1 21:50:04 PST 2003 — Kernel Upgrades: Due to a new exploit that’s been seen in the wild, we are upgrading all of the kernels on our servers. All machines which customers have access to have already been patched and rebooted. We’re continuing to work down the list of servers, eliminating any potential problems. We have run into one issue so far: the SQL server used by the SpamAssassin cluster didn’t take it’s new kernel and is currently off-line. We hope to have it back up shortly after some hands on work is performed. Until its service is restored, SpamAssassin will filter mail based on its default settings. -Kelsey and Nathan

Sun Nov 30 16:45:22 PST 2003 — Usenet Services Restored: news.sonic.net is fully functional again. As it turns out, our NNTP feed server had a corrupted spool. That spool is dedicated to all non-binary messages. Until the problem was tracked down and resolved it would not transit non-binary posts. It appears that all locally posted articles during this time have been handed out to our peers and that missing posts are being filled in. It’s unlikely that 100% of the old inbound messages will reach us. -Kelsey

Sat Nov 29 21:16:29 PST 2003 — Something strange with Usenet. Folks have been reporting a lower Usenet volume than usual — so far, we haven’t found a definitive cause for this. However, one change on our news reader box might have solved the problem.

We apologize for any Usenet strangeness you might encounter, and we continue to investigate. -Dane, Kelsey, and Scott

Sat Nov 29 16:28:52 PST 2003 — Sebastopol busy signals: Equipment failure and problems with the hunt sequence on our Sebastopol dial-up system caused busy signals on 707-823-8812. The problem has been repaired and the number is no longer giving busy signals. This system will soon be replaced with a new, more fault tolerant, system which will prevent this type of problem. The new equipment has already been delivered and the new lines are slated for activation next week. Transition to the new system will take place after a week or two of testing. Our other dial-up numbers have already be upgraded to fault tolerant systems – Russ

Wed Nov 26 10:31:23 PST 2003 — Web, SSL and FTP service issues: For a yet undetermined cause, the tool which controls users bandwidth allocation and that locks out access to a user’s files if that user has enabled quota protection, locked out the root directories of our public web, ssl and ftp servers. We’ve taken steps to assure that this will not happen again, and are continuing to investigate why the tool failed. -Kelsey

Wed Nov 26 08:44:27 PST 2003 — SSL and Web Mail issue: We are experiencing an issue with our SSL server that drives our web mail application. Currently, the tool is off-line while we make repairs. -Matt

Tue Nov 25 10:23:47 PST 2003 — SpamAssassin Changes: In order improve the overall quality and performance of our primary anti-spam filters we’ve disabled one of it’s features; The AWL, or auto-whitelist, keeps track of all addresses that mail has been received from along with statistics about that senders mail profile. This information is used to shift a given message’s score to the senders mean. This can be an effective tool in helping reduce false positives, but, as implemented currently, it has a number of flaws. The most significant flaw is run away growth which eats into users’ disk quotas; it never deletes records from the database.

One of the side effects of the ever-increasing AWL database size is increased server IO load. This morning, the IO load generated by the AWL on freezer, the NetApp filer that they are stored on, was approaching 90% of the filer’s total capacity. As freezer slows down under these high load conditions it cascades to high load on the SpamAssassin servers, mail servers and results in unfiltered mail.

We are working on a major SpamAssassin upgrade that we expect to launch around the New Year. Hopefully, if we feel that the AWL is still a valuable feature of SpamAssassin, we’ll be able to improve and re-enable it at this time. -Kelsey