Month: February 2004

Thu Feb 26 02:02:40 PST 2004 — DSL Maintenance Complete: We upgraded the SMS as scheduled tonight. Total downtime was approximately 15 minutes. In addition to the OS upgrade we also replaced both the FE and ATM2-OC3c with on-site spares due to a failed diagnostics. However, it’s our impression that the diagnostics failure is not related to the performance issues. At this time, performance has improved and we will continue to monitor. It’s very likely that we will have at least one additional maintenance window on the Santa Rosa SMS before the issues will be completely resolved. -Kelsey and Nathan

Wed Feb 25 10:27:33 PST 2004 — Petaluma Wireless outage. The tower site serving Petaluma wireless has suffered an extended power outage as a result of the weather. We are working with the tower site to restore power. -Matt and Bryan

Wed Feb 25 09:54:33 PST 2004 — BroadLink service outage. At 4:20am one of BroadLink’s primary backhaul radios stopped operating. The cause is unknown but presumably weather related. Resetting the unit brought the radio link back up which partially restored service (9:20am). There are still about 9 customers offline in a fairly tight cluster, we are investigating. -Jason K. (BroadLink)

Wed Feb 25 16:20:08 PST 2004 — DSL Speed Update: Tonight at 1am we are going to upgrade the software on the RedBack SMS in Santa Rosa. We’ve believe that we’ve identified the cause performance problems and a newer release of the OS should allow us to resolve the problem. If the upgrade goes as planned downtime should only be a few minutes. If the upgrade resolves the problems we will upgrade the other RedBack SMS’s in San Francisco and Stockton shortly. -Kelsey, Nathan and John

Tue Feb 24 19:33:48 PST 2004 — DSL Speed Issues: We have identified a limitation in SBC’s Remote Terminal (RT) hardware that prevents DSL customers on RTs from receiving the full performance of their link. After applying the recommended workaround to our Redback Networks DSL termination hardware, we found that all customers started to experience light (.5-2%) packet loss on their circuits. This loss has persisted even after backing out the changes.

Resolving this issue is our highest priority, and we have been given a list of potential fixes by Redback’s support department. Unfortunately, these fixes will require one or more reloads of each Redback over the next few days. The first of these reboots is scheduled for 1am on Feb. 25th, and will affect only customers that connect to our Santa Rosa Redback. Each event should cause under 5 minutes worth of downtime.

We’ll post updates to this MOTD entry as we learn more and apologize in advance for the service interruptions. -Nathan, Kelsey, John and Dane

Wed Feb 18 14:43:52 PST 2004 — Mass Kernel Upgrade: All servers have received new Linux kernels in response to a recently announced local exploit. (FYI, a “local exploit” means that local shell users could potentially exploit a system. Sonic.net has not had any exploit issues, this is a preemptive fix) If you run Linux, it’s time to upgrade your kernels to 2.4.25 or newer. Many Linux distros will back-port the patches to earlier kernel revisions. -Kelsey

Wed Feb 11 16:39:17 PST 2004 — Mutt Updated on bolt: Mutt has been upgraded to the latest stable release on bolt to fix a buffer overflow. It’s an obscure bug, but be sure to update your own unix systems as well. Additionally, support for POP, IMAP, and SSL has been added. -Kelsey

Sun Feb 8 11:10:24 PST 2004 — news.sonic.net down. Our local news server failed last night. We just rebooted it and it is now serving news readers again. As this server is quite stable (over 8 months since the last reboot) we are investigating why this occurred. -Scott and John

Fri Feb 6 10:50:23 PST 2004 — Mail Delays: Last night and this morning two events contributed to delays in handling inbound email. At approximately 01:15 this morning, one of the 4 clustered filers ethernet locked up. This was resolved quickly, but led to some delays as mail queued for delivery as well as for 1/4 of our customers, the inability to check mail. Later this morning, instabilities in our SpamAssassin cluster caused some delays upon email delivery. At this time everything if functioning at 100% and we are almost caught up with our queues. -Kelsey, Nathan, Zeke and Jared