Month: March 2000

We got two major upgrades done today.

Tue Mar 28 16:25:38 PST 2000 — We got two major upgrades done today. First, we slid a brand new SQL server under our old SQL server. The new box has a whole lot more power than the old one, and you may notice the member tools speed up as a result. We sure noticed the difference with our internal support and accounting tools. Second, buzz, was rebuilt and setup with sendmail and put into the mail load balancing server group again. This gives us four snappy mail servers in the group and plenty of speed, redundancy (two servers can almost sustain the load,) and room to grow. -Kelsey and Eli

After extensive testing, our new dynamic…

Sat Mar 25 17:18:51 PST 2000 — After extensive testing, our new dynamic apache configuration is online. The system was tested by running it on port 81 for all multihomes — feedback was gathered in We also tested SSI and Frontpage after the switch, found some problems, and corrected them. The new configuration improves the stability and speed of apache, and allows us to continue with our load-balancing project. The load-balancing project uses our Altheon web director to direct traffic to multiple web servers — traffic is directed to the most responsive server, and should a web server go down, traffic is automatically directed to servers that are up. If you detect a problem with your multihome, please email or post a note in Thanks. -Scott, Eli, and Kelsey

As a side effect of reloading the router this

Wed Mar 22 12:54:58 PST 2000 — As a side effect of reloading the router this morning, some of our remote dial-up POP’s saw spotty connectivity to and This caused intermittent delays in mail retrieval for customers using these POP’s. We’ve resolved the issue, and apologize once more for the rocky start to the day! — Eli, Kelsey

We’ve noticed some packet loss at our gateway

Wed Mar 22 10:19:25 PST 2000 — We’ve noticed some packet loss at our gateway to UUNet, and this correleates with some unusual behavior we’ve noticed in our core router. Despite the 5 minute downtime, we elected to reboot the router. We will continue to monitor the situation, and apologize for the ‘bump’. – Eli, Kelsey, and Dave

Due to repeated request from our customers to

Tue Mar 21 11:52:58 PST 2000 — Due to repeated request from our customers to keep their mail boxes free from SPAM we have implemented four new anti-spam filters in our mail servers: RBL, RSS, DUL and’s Relay list. In the first hour since these rules have been in place we’ve already caught over 1000 pieces of SPAM! We’d like to hear some feedback to know if you find these new rules effective in stopping SPAM. -Kelsey and Eli

Some web sites reported problems with CGI…

Mon Mar 20 18:54:40 PST 2000 — Some web sites reported problems with CGI today, which appears to be related to the number of virtual hosts on Thunder. As an interim fix, we’ve moved 250 virtual hosts to Storm. Additionally, we are experimenting with a new web server configuration that uses dynamic virtual host configuration. If you have a multihome, please test the new configuration by visiting — please report your findings to Thanks. -Scott and Kelsey

Our Network Appliance NFS disk array ran out…

Wed Mar 15 08:59:44 PST 2000 — Our Network Appliance NFS disk array ran out of space this morning, and we had to do some quick management to clear up room. It’s got a live backup feature that’s taking up quite a bit of disk space, so we’ve dialed back the timing on the backups so that there are less of the. We’re working today to find more disks to put into this array, and also turning on additional monitoring so that we’ll have early warning.

We will be running disk quotas this morning, and if you’re over quota, you will find billing in your next invoice. -Dane, Kelsey and Scott

SSI problems.

Tue Mar 14 20:40:13 PST 2000 — SSI problems. Some sites are reporting problems with server-side-includes (SSI), which we have been investigating all day. An upgrade to Apache 1.3.12 only made the problem worse. We’re continuing to investigate the problem, and have logged a bug report with -Scott

We had (Typhoon) on our…

Sun Mar 12 18:57:59 PST 2000 — We had (Typhoon) on our 12-port 100MB switch. Unfortunately, the traffic bound for Typhoon saturated the 100MB link to our gigaswitch, causing instability in our switch fabric. We’ve moved Typhoon to our core Black Diamond switch, and rebooted all our switches, as well as a few hubs. This caused selective unreachability of about 30 seconds intermittently across our network. Also, one hub was down for 15 minutes before we caught it, which caused some authentication errors in the 1001 hunt group. The network is now fixed. -Scott, Eli, and Dane (the switch bootin’ fiend)