Wed Feb 25 10:27:33 PST 2004 — Petaluma Wireless outage. The tower site serving Petaluma wireless has suffered an extended power outage as a result of the weather. We are working with the tower site to restore power. -Matt and Bryan

Wed Feb 25 09:54:33 PST 2004 — BroadLink service outage. At 4:20am one of BroadLink’s primary backhaul radios stopped operating. The cause is unknown but presumably weather related. Resetting the unit brought the radio link back up which partially restored service (9:20am). There are still about 9 customers offline in a fairly tight cluster, we are investigating. -Jason K. (BroadLink)

Wed Feb 25 16:20:08 PST 2004 — DSL Speed Update: Tonight at 1am we are going to upgrade the software on the RedBack SMS in Santa Rosa. We’ve believe that we’ve identified the cause performance problems and a newer release of the OS should allow us to resolve the problem. If the upgrade goes as planned downtime should only be a few minutes. If the upgrade resolves the problems we will upgrade the other RedBack SMS’s in San Francisco and Stockton shortly. -Kelsey, Nathan and John

Tue Feb 24 19:33:48 PST 2004 — DSL Speed Issues: We have identified a limitation in SBC’s Remote Terminal (RT) hardware that prevents DSL customers on RTs from receiving the full performance of their link. After applying the recommended workaround to our Redback Networks DSL termination hardware, we found that all customers started to experience light (.5-2%) packet loss on their circuits. This loss has persisted even after backing out the changes.

Resolving this issue is our highest priority, and we have been given a list of potential fixes by Redback’s support department. Unfortunately, these fixes will require one or more reloads of each Redback over the next few days. The first of these reboots is scheduled for 1am on Feb. 25th, and will affect only customers that connect to our Santa Rosa Redback. Each event should cause under 5 minutes worth of downtime.

We’ll post updates to this MOTD entry as we learn more and apologize in advance for the service interruptions. -Nathan, Kelsey, John and Dane

Wed Feb 18 14:43:52 PST 2004 — Mass Kernel Upgrade: All servers have received new Linux kernels in response to a recently announced local exploit. (FYI, a “local exploit” means that local shell users could potentially exploit a system. Sonic.net has not had any exploit issues, this is a preemptive fix) If you run Linux, it’s time to upgrade your kernels to 2.4.25 or newer. Many Linux distros will back-port the patches to earlier kernel revisions. -Kelsey

Wed Feb 11 16:39:17 PST 2004 — Mutt Updated on bolt: Mutt has been upgraded to the latest stable release on bolt to fix a buffer overflow. It’s an obscure bug, but be sure to update your own unix systems as well. Additionally, support for POP, IMAP, and SSL has been added. -Kelsey

Sun Feb 8 11:10:24 PST 2004 — news.sonic.net down. Our local news server failed last night. We just rebooted it and it is now serving news readers again. As this server is quite stable (over 8 months since the last reboot) we are investigating why this occurred. -Scott and John

Fri Feb 6 10:50:23 PST 2004 — Mail Delays: Last night and this morning two events contributed to delays in handling inbound email. At approximately 01:15 this morning, one of the 4 clustered filers ethernet locked up. This was resolved quickly, but led to some delays as mail queued for delivery as well as for 1/4 of our customers, the inability to check mail. Later this morning, instabilities in our SpamAssassin cluster caused some delays upon email delivery. At this time everything if functioning at 100% and we are almost caught up with our queues. -Kelsey, Nathan, Zeke and Jared

Wed Jan 28 04:24:12 PST 2004 — Focal connectivity issues. The T3 between Focal in San Francisco and our Santa Rosa headquarters bounced at 3:25:21am this morning. The bounce triggered a lockup bug in our Santa Rosa router’s T3 card, which caused the link to remain down even after it was able to pass data. A reboot of the card in Santa Rosa fixed the lockup. This T3 failure impacted the ability for dialup customers served out of Focal to authenticate, check e-mail and get to other Sonic services. The failure also partitioned our UUNet transit circuit from the rest of our network. However, excess transit capacity was able to pick up the slack and only Focal was impaired. Total downtime was approximately 45 minutes. Alternate dialup numbers were available during this outage. To find a list of Sonic dialups in your area, visit:

www.sonic.net/cgi-bin/pops.pl -Nathan

Tue Jan 27 07:06:57 PST 2004 — New “MyDoom” email virus trapped by Sonic.net anti-virus system. A new email virus is sweeping the Internet. Almost immediately after its release on the Net, our automatic anti-virus software began blocking propagation of the virus.

News on the virus can be found on CNN.com:

tinyurl.com/293tp

Virus inception and impact can be viewed on one of our statistics pages:

tinyurl.com/2lomv

We suggest Internet users exercise caution when opening questionable email attachments. In most cases, the “MyDoom” virus will appear to have been sent by someone who has emailed you before.

We are very pleased with the performance of the relatively new anti-virus software — thanks to Kelsey for getting it online. -Scott and Dane