Mon Mar 1 11:55:09 PST 2004 — SSL Key Change: We’ve just changed the SSL certificate for our POP3 server, the old one was expiring in a few days. The new certificate is installed and fully functional however, it will take some additional testing in our side to assure that the certificate signing chain is properly configured. At this time, some users may get errors stating that the client is unable to verify the certificate. It is perfectly acceptable to use the certificate as it stands, we should have this resolved shortly. -Kelsey

Mon Mar 1 11:08:30 PST 2004 — Shared webserver failure. This morning all of the servers that support the shared web services stopped responding. We restarted them and are investigating the failure. -John, Geoff and Kelsey

Mon Mar 1 22:18:21 PST 2004 — Update on DSL speed issues. Due to an inadvertent application of system-wide speed profiles this afternoon, we’re back to a situation where many customers are not getting optimal DSL performance.

To recap, we’ve run into buffering issues on the DSL equipment which is causing deliver of about 30% below maximum performance. The performance is still well above allowable minimums, but we’re committed to delivering full performance as quickly as possible. We’ll continue to work to resolve the situation! -Dane (for John, Nathan, Kelsey and Eli)

Mon Mar 1 18:57:27 PST 2004 — Brief Mail Troubles: One of four NFS filers that handle Email message stores had a Ethernet interface lockup. Before this was manually fixed, 1/4 of our users would have been unable to check their email. This glitch will also result in slightly delayed Email delivery. At this time, all systems are fully functional. Since this is the second time this interface has locked up on us, it’s slated it for replacement with an on-site spare. -Kelsey

Thu Feb 26 02:02:40 PST 2004 — DSL Maintenance Complete: We upgraded the SMS as scheduled tonight. Total downtime was approximately 15 minutes. In addition to the OS upgrade we also replaced both the FE and ATM2-OC3c with on-site spares due to a failed diagnostics. However, it’s our impression that the diagnostics failure is not related to the performance issues. At this time, performance has improved and we will continue to monitor. It’s very likely that we will have at least one additional maintenance window on the Santa Rosa SMS before the issues will be completely resolved. -Kelsey and Nathan

Wed Feb 25 10:27:33 PST 2004 — Petaluma Wireless outage. The tower site serving Petaluma wireless has suffered an extended power outage as a result of the weather. We are working with the tower site to restore power. -Matt and Bryan

Wed Feb 25 09:54:33 PST 2004 — BroadLink service outage. At 4:20am one of BroadLink’s primary backhaul radios stopped operating. The cause is unknown but presumably weather related. Resetting the unit brought the radio link back up which partially restored service (9:20am). There are still about 9 customers offline in a fairly tight cluster, we are investigating. -Jason K. (BroadLink)

Wed Feb 25 16:20:08 PST 2004 — DSL Speed Update: Tonight at 1am we are going to upgrade the software on the RedBack SMS in Santa Rosa. We’ve believe that we’ve identified the cause performance problems and a newer release of the OS should allow us to resolve the problem. If the upgrade goes as planned downtime should only be a few minutes. If the upgrade resolves the problems we will upgrade the other RedBack SMS’s in San Francisco and Stockton shortly. -Kelsey, Nathan and John

Tue Feb 24 19:33:48 PST 2004 — DSL Speed Issues: We have identified a limitation in SBC’s Remote Terminal (RT) hardware that prevents DSL customers on RTs from receiving the full performance of their link. After applying the recommended workaround to our Redback Networks DSL termination hardware, we found that all customers started to experience light (.5-2%) packet loss on their circuits. This loss has persisted even after backing out the changes.

Resolving this issue is our highest priority, and we have been given a list of potential fixes by Redback’s support department. Unfortunately, these fixes will require one or more reloads of each Redback over the next few days. The first of these reboots is scheduled for 1am on Feb. 25th, and will affect only customers that connect to our Santa Rosa Redback. Each event should cause under 5 minutes worth of downtime.

We’ll post updates to this MOTD entry as we learn more and apologize in advance for the service interruptions. -Nathan, Kelsey, John and Dane

Wed Feb 18 14:43:52 PST 2004 — Mass Kernel Upgrade: All servers have received new Linux kernels in response to a recently announced local exploit. (FYI, a “local exploit” means that local shell users could potentially exploit a system. Sonic.net has not had any exploit issues, this is a preemptive fix) If you run Linux, it’s time to upgrade your kernels to 2.4.25 or newer. Many Linux distros will back-port the patches to earlier kernel revisions. -Kelsey