Thu Dec 23 14:51:57 PST 2004 — DSL Packet Loss on Santa Rosa SMS. We’ve identified what appears to be a bad ethernet interface in the SMS thats causing as much as 10% packet loss for DSL customers terminated in Santa Rosa. We are currently investigation solutions to resolve this problem that do not involve replacing the card which would require that we reboot equipment potentially leading to an extended complete outage. -Kelsey, Nathan and John.

Fri Dec 17 11:34:46 PST 2004 — Sonic.net is closed today from 12 noon until 3pm for our staff year-end recognition party. We’re sorry for any inconvenience this may cause – please phone us after 3pm today if you need anything! Happy Holidays! -Dane

Fri Dec 17 11:07:54 PST 2004 — problems with dial-up numbers ending in 9606. Equipment failure caused connectivity problems on dial-up numbers ending with 9606. The equipment was rebooted and is now working properly. – Russ

Fri Dec 17 17:00:54 PST 2004 — DSL Problems in Stockton. Our network provider in Stockton where our DSL termination equipment is colocated is currently experiencing significant packet loss. They have been informed and will have the trouble resolved shortly. -Kelsey, Nathan

Thu Dec 16 10:15:44 PST 2004 — Internal Packet Loss. Last night we discovered some rather severe packet loss in our server backend SAN network that affected a number of hosts. In particular, two of our SpamAssassin servers and both FTP servers were affected by this loss. This may have exhibited in problems using our FTP servers and some mail messages not being filtered by SpamAssassin. The other hosts affected by the problem do not support public services. We resolved the problem this morning when we found one of the SAN switches was flopping back and forth between it’s primary and secondary uplinks to the other switches. -Kelsey and Nathan

Wed Dec 15 17:21:31 PST 2004 — Kernel Security Updates. We are upgrading the Linux kernels on all of our production servers to resolve two recently announced kernel exploits. While each server is rebooted some customer may notice small server outages while the clusters failover. -Kelsey, Scott, and Kevan.

Wed Dec 15 16:24:49 PST 2004 — Mail Issues. For a short time yesterday a small percentage of mail was incorrectly rejected due to being on the SBL spamhaus.org RBL. We disabled the use of this RBL in our MTA filters so we could track down the problem which appears to either be a bug in our custom software or our internal DNS servers. It was not due to hosts being incorrectly listed in the SBL. This only affected people using the SBL to filter their inbound mail – part of our default filter set – and did not affect anyone who had specifically turned it off. We’re going to re-enable the SBL once we’ve added some additional checks to our code. -Kelsey, Russ and Kevan

Wed Dec 15 15:28:40 PST 2004 — PHP upgrade on new web cluster. The new web cluster has been upgraded to PHP 4.3.10 which addresses some security issues present in earlier versions. The upgrade went smoothly and there was no interruption in service. -Kevan

Wed Nov 24 15:26:59 PST 2004 — More Kernel Updates. We’ve identified a bug in the 2.4.28 linux kernel’s arp subsystem that causes some significant problems during the booting of our servers. While we continue to track down this bug we’ve back ported the security updates to 2.4.27. Unfortunately this means we have to reboot a large number of servers again. -Kelsey, Augie, Kevan and Nathan.

Tue Nov 23 17:35:43 PST 2004 — Kernel Security Updates. We are in the process of upgrading the Linux kernels on most of our servers to resolve a potential security problem. Thanks to clustering of our major services these upgrades usually go unnoticed; however, we’ve had a increased call volume in response to some minor issues associated with the reboots. –Kelsey and Augie