Tue Mar 23 09:21:34 PST 2004 — Web server issues. We are experiencing intermittent problems with one of our web servers. Customers should not notice any problems, but we have had a few reports of errors when publishing Front Page documents. Operations is investigating. -Matt, Nathan and Geoff

Sun Mar 21 12:11:39 PST 2004 — Sonoma County Airport Broadlink tower is back up. ~ 80 customers were affected (~ 64 businesses). -Scott and Jason from Broadlink

Sun Mar 21 10:53:49 PST 2004 — Broadlink outage near Sonoma County Airport. The Broadlink sector supporting the area around Sonoma County Airport appears to be down. Broadlink has been notified, we are waiting for an ETR. -Scott and Dan

Update: Sun Mar 21 11:01:43 PST 2004: Broadlink reports this is probably the same problem they saw with their switch last month; their ETR is 25 minutes. -Scott, Kavan, and Dan; Tim and Jason from Broadlink

Thu Mar 18 12:39:05 PST 2004 — Customer MySQL Upgrade: Tonight at Midnight we will be swapping in the new hardware for custsql. If the operation goes as scripted downtime will be a few minutes as the current box shuts down and the new box starts up. There will be an additional small maintenance windows as data is migrated from the current NFS SQL store to the local raid, and then again as the MySQL software is upgraded to the latest release. -Kelsey

Thu Mar 18 11:43:54 PST 2004 — POP Server Update: We identified the problem with the POP3 anti-virus filtering as a memory leak in ClamAV. No, it’s resident size should not be 2.9GB. We’ve upgraded to a newer version of ClamAV and will be watching memory utilization closely for the next few days. At this time, all POP3 sessions are receiving the final anti-virus scan. It’s proven quite effective and has blocked nearly 26,000 copies of viruses that would have otherwise been delivered to customers since it’s inception. -Kelsey and Nathan

Wed Mar 17 19:04:01 PST 2004 — POP Server Issues: Due to what appears to be some instability in the anti-virus software that we integrated with our POP3 server, our POP3 server was causing timeouts and other problems for customers. Russ rolled back the POP3 server version and we are now running without the extra layer of anti-virus filtering. The system was designed and tested to fail open when problems with the anti-virus software was encountered. We will restore the POP3 anti-virus filtering tomorrow after we’ve had a chance to analyze the failure and correct what we find. This may have also had some affect on people using webmail. -Kelsey, Russ, Nathan and Chris

Tue Mar 16 09:56:41 PST 2004 — Broadlink Barham tower offline. Broadlink has dispatched personnel to their Barham tower site to effect repairs. We will post an ETR when received from Broadlink. -Scott, as well as Jason and Linda from Broadlink

Update: Tue Mar 16 10:17:24 PST 2004: Broadlink has identified this as a problem with their switch at Barham; ETR is approximately 1 hour. -Scott, Jason, and Linda

Update: Tue Mar 16 11:15:08 PST 2004: Broadlink reports that sector 3 of Barham is now online, and Jason is working to get sector 4 online. Apparently there were power problems at the Barham tower site. -Scott, Jason, and Linda Update: Tue Mar 16 12:17:03 PST 2004: Sector 4 of Barham is still not fully operational, with 66 customers still down. Broadlink thinks they may have to replace the radio for sector 4. No ETR for that replacement yet. -Scott, Jason, and Linda

Update: Tue Mar 16 12:46:55 PST 2004: Sector 4 of Barham still not fully operational — 21 customers still down. We will cease reporting until and unless there is a significant event, including sector 4 becoming fully operational. -Scott, Jason, and Linda

Mon Mar 15 11:56:32 PST 2004 — Customer MySQL Server Issues: As some users may be aware custsql.sonic.net has had some stability problems recently. At this time we believe the problems to be hardware related and have a replacement server on order. The new server is significantly faster than the current server: It’s a Dual Xeon with 3-Ware SATA RAID. The replacement server should be on line in a few days. -Kelsey

Mon Mar 15 11:53:30 PST 2004 — Stockton-area DSL problems solved: SBC corrected a malfunction with one of their ATM switches yesterday afternoon. Thus, Stockton-area DSL should now be operating normally. -Scott

Sat Mar 13 14:50:28 PST 2004 — Stockton-area DSL performance problems: DSL customers in the Stockton area are experiencing an abnormal amount of lost packets, resulting in greatly impacted performance. The ASI maintenance crew is working on this issue, though we have no ETR available. -John F.