Thu Mar 25 08:37:00 PST 2004 — Web server update. Thunder had a bad motherboard, which has been replaced and thunder also got a ram upgrade. Web services were not interrupted. -Geoff

Wed Mar 24 10:36:37 PST 2004 — Delayed email delivery. This morning one of our incoming mail servers developed a problem and stopped delivering mail. The server has been restored and mail service is back to normal, but some mail will be delayed. We expect to have locally queued email delivered shortly, and expect mail that was queued remotely to arrive within the next few hours. No email was lost during this brief event. -Operations

Wed Mar 24 10:18:44 PST 2004 — Bandwidth stats currently unavailable. Hedgehog, the machine which collects bandwidth usage and web traffic data and generates reports, has developed problems and is offline. Reports are currently not available. Data was not lost, and is still being collected, but reporting is down until further notice. Our operations staff is working to restore the machine; there is no ETR at this time. -Operations

Wed Mar 24 21:42:48 PST 2004 — Night Ops. Tonight, at Midnight, we are going to be relocating a number of switches that serve our core ISP services to make room for additional equipment. Due to the redundant meshed configuration we should be able to do this with little to no service interruptions. Our shell server, bolt, however, is single homed off of one of the switches and will be briefly unavailable when it’s switch is moved. -Kelsey and Nathan

Tue Mar 23 22:16:34 PST 2004 — Web server update. One of our 3 redundant webservers, storm, is still out of service. It had severe disk problems and had to be rebuilt. This took all day and was just finished. It will be tested and placed back into service on Wednesday. -Kelsey, Nathan and Geoff

Tue Mar 23 21:49:35 PST 2004 — News server issues. Our news reader server suffered disk corruption today. It took us several hours to recover and restart the system. Any articles that came in while the disk was corrupted are lost. We believe locally posted articles did go out OK. -Kelsey, John and Scott

Tue Mar 23 09:21:34 PST 2004 — Web server issues. We are experiencing intermittent problems with one of our web servers. Customers should not notice any problems, but we have had a few reports of errors when publishing Front Page documents. Operations is investigating. -Matt, Nathan and Geoff

Sun Mar 21 12:11:39 PST 2004 — Sonoma County Airport Broadlink tower is back up. ~ 80 customers were affected (~ 64 businesses). -Scott and Jason from Broadlink

Sun Mar 21 10:53:49 PST 2004 — Broadlink outage near Sonoma County Airport. The Broadlink sector supporting the area around Sonoma County Airport appears to be down. Broadlink has been notified, we are waiting for an ETR. -Scott and Dan

Update: Sun Mar 21 11:01:43 PST 2004: Broadlink reports this is probably the same problem they saw with their switch last month; their ETR is 25 minutes. -Scott, Kavan, and Dan; Tim and Jason from Broadlink

Thu Mar 18 12:39:05 PST 2004 — Customer MySQL Upgrade: Tonight at Midnight we will be swapping in the new hardware for custsql. If the operation goes as scripted downtime will be a few minutes as the current box shuts down and the new box starts up. There will be an additional small maintenance windows as data is migrated from the current NFS SQL store to the local raid, and then again as the MySQL software is upgraded to the latest release. -Kelsey