Month: March 2004

Thu Mar 18 12:39:05 PST 2004 — Customer MySQL Upgrade: Tonight at Midnight we will be swapping in the new hardware for custsql. If the operation goes as scripted downtime will be a few minutes as the current box shuts down and the new box starts up. There will be an additional small maintenance windows as data is migrated from the current NFS SQL store to the local raid, and then again as the MySQL software is upgraded to the latest release. -Kelsey

Thu Mar 18 11:43:54 PST 2004 — POP Server Update: We identified the problem with the POP3 anti-virus filtering as a memory leak in ClamAV. No, it’s resident size should not be 2.9GB. We’ve upgraded to a newer version of ClamAV and will be watching memory utilization closely for the next few days. At this time, all POP3 sessions are receiving the final anti-virus scan. It’s proven quite effective and has blocked nearly 26,000 copies of viruses that would have otherwise been delivered to customers since it’s inception. -Kelsey and Nathan

Wed Mar 17 19:04:01 PST 2004 — POP Server Issues: Due to what appears to be some instability in the anti-virus software that we integrated with our POP3 server, our POP3 server was causing timeouts and other problems for customers. Russ rolled back the POP3 server version and we are now running without the extra layer of anti-virus filtering. The system was designed and tested to fail open when problems with the anti-virus software was encountered. We will restore the POP3 anti-virus filtering tomorrow after we’ve had a chance to analyze the failure and correct what we find. This may have also had some affect on people using webmail. -Kelsey, Russ, Nathan and Chris

Tue Mar 16 09:56:41 PST 2004 — Broadlink Barham tower offline. Broadlink has dispatched personnel to their Barham tower site to effect repairs. We will post an ETR when received from Broadlink. -Scott, as well as Jason and Linda from Broadlink

Update: Tue Mar 16 10:17:24 PST 2004: Broadlink has identified this as a problem with their switch at Barham; ETR is approximately 1 hour. -Scott, Jason, and Linda

Update: Tue Mar 16 11:15:08 PST 2004: Broadlink reports that sector 3 of Barham is now online, and Jason is working to get sector 4 online. Apparently there were power problems at the Barham tower site. -Scott, Jason, and Linda Update: Tue Mar 16 12:17:03 PST 2004: Sector 4 of Barham is still not fully operational, with 66 customers still down. Broadlink thinks they may have to replace the radio for sector 4. No ETR for that replacement yet. -Scott, Jason, and Linda

Update: Tue Mar 16 12:46:55 PST 2004: Sector 4 of Barham still not fully operational — 21 customers still down. We will cease reporting until and unless there is a significant event, including sector 4 becoming fully operational. -Scott, Jason, and Linda

Mon Mar 15 11:56:32 PST 2004 — Customer MySQL Server Issues: As some users may be aware custsql.sonic.net has had some stability problems recently. At this time we believe the problems to be hardware related and have a replacement server on order. The new server is significantly faster than the current server: It’s a Dual Xeon with 3-Ware SATA RAID. The replacement server should be on line in a few days. -Kelsey

Mon Mar 15 11:53:30 PST 2004 — Stockton-area DSL problems solved: SBC corrected a malfunction with one of their ATM switches yesterday afternoon. Thus, Stockton-area DSL should now be operating normally. -Scott

Sat Mar 13 14:50:28 PST 2004 — Stockton-area DSL performance problems: DSL customers in the Stockton area are experiencing an abnormal amount of lost packets, resulting in greatly impacted performance. The ASI maintenance crew is working on this issue, though we have no ETR available. -John F.

Tue Mar 9 15:24:12 PST 2004 — Mail Server Upgrades: Over the past few days we’ve completed alot of behind the scenes mail server upgrades. We’ve upgraded to the latest release of sendmail in order to gain some additional controls over our mail queues. The controls have allowed us to significantly reduce the queue size on some of our servers which increases the rate that mail can flow through out servers. We’ve also upgraded both Clamav, the open-source Anti-Virus engine that we use, and the glue that we use to hook it into our mail servers. These upgrades have strengthened our anti virus filtering and should result in fewer viruses going undetected. -Kelsey

Thu Mar 4 20:43:51 PST 2004 — SMS Update: We just completed additional profiling work on the Santa Rosa SMS to address performance issues identified after last night’s reboot. Throughput looks better, and we’re hopeful that the DSL speed issues have been resolved. -Nathan, Kelsey and Dane

Thu Mar 4 02:38:42 PST 2004 — SMS Reboot Complete: The reload went as expected. It’s too early to tell for sure, but we are hopeful that the performance problems have been resolved. -Nathan and Kelsey