Wed Mar 15 08:59:44 PST 2000 — Our Network Appliance NFS disk array ran out of space this morning, and we had to do some quick management to clear up room. It’s got a live backup feature that’s taking up quite a bit of disk space, so we’ve dialed back the timing on the backups so that there are less of the. We’re working today to find more disks to put into this array, and also turning on additional monitoring so that we’ll have early warning.

We will be running disk quotas this morning, and if you’re over quota, you will find billing in your next invoice. -Dane, Kelsey and Scott

Tue Mar 14 20:40:13 PST 2000 — SSI problems. Some sites are reporting problems with server-side-includes (SSI), which we have been investigating all day. An upgrade to Apache 1.3.12 only made the problem worse. We’re continuing to investigate the problem, and have logged a bug report with apache.org. -Scott

Sun Mar 12 18:57:59 PST 2000 — We had news.sonic.net (Typhoon) on our 12-port 100MB switch. Unfortunately, the traffic bound for Typhoon saturated the 100MB link to our gigaswitch, causing instability in our switch fabric. We’ve moved Typhoon to our core Black Diamond switch, and rebooted all our switches, as well as a few hubs. This caused selective unreachability of about 30 seconds intermittently across our network. Also, one hub was down for 15 minutes before we caught it, which caused some authentication errors in the 1001 hunt group. The network is now fixed. -Scott, Eli, and Dane (the switch bootin’ fiend)

Sun Mar 12 17:43:05 PST 2000 — News.sonic.net is doing much better now after a motherboard swap. Sorry about the up and down news services today. We’ll be catching up on our feeds shortly. We will be rebooting it once more in a few minutes for a RAM upgrade, as we shorted it by a quarter gig in this new configuration. That will resolve some sluggish performance that we’re seeing. -Scott, Eli and Dane

Sun Mar 12 05:39:00 PST 2000 — The news server is still having problems, and we’ve seen it reboot once in the last hour. As we don’t have a conclusive diagnosis for the trouble at this time, we’re going to be replacing all of the hardware, probably tomorrow night. In the mean time, we expect that it may occasionally reboot and be unavailable for about ten minutes at a stretch. Downtime during the swap to new hardware will be about an hour. Sorry about the anticipated flaky behavior! -Dane and Scott

Sun Mar 12 05:12:07 PST 2000 — As you can see from the MOTD entries, we had problems with date/time on shell.sonic.net. For some reason, it couldn’t bind to the network time protocol master server and get a correct time. -Dane, Eli and Scott

Sun Mar 12 03:50:21 PST 2000 — During system startup, our news server had problems with it’s power supply handling the load of the 110+ gigs worth of high draw Seagate SCSI disks. Basically, as it brought up the news software, the large amount of disk activity would cause the power supply to go to a low current situation, and the system would reboot. We swapped the power supply, but the second supply wouldn’t even boot the machine. We ended up with a dual power supply configuration with one supply handling the three large disks, and the other supplying the root disk and motherboard. Total Usenet news downtime was about an hour and a half while we worked this all out. We’ll find a larger power supply for this machine this week. -Dane, Dave and Steve

Sun Mar 12 02:39:08 PST 2000 — Our power plant change is complete. In total, we had 13 minutes without power. This puts the phase one build-out section of our data-center on a new APC Symmetra UPS located in our new power room downstairs. On Monday, we’ll be moving the old Symmetra downstairs to power our phase two build-out.

Placing these units downstairs gives us the physical space and floor load capacity to add additional battery frames for extended runtime. A single Symmetra with one extra battery frame runs about 50 minutes at full load and weighs one ton. We’ll be adding more extended run battery frames to both units now that we’ve eliminated the concern that they’ll come crashing through the ceiling of technical support, crushing unsuspecting support tech Brandon Butler. With a total of up to four battery frames per Symmetra, we could build runtime of up to three hours after our complete phase one and two build-out is complete. Total weight of the two Symmetras will be 10,000 pounds. A third Symmetra will be added for our third section build-out in 12 to 24 months.

Sonic.net crew: Dane, Scott, Eli, Kelsey, Steve and Dave IX Labs crew: Brad and Scooter Href/Robmart: Rob Martin O’Rourke Electric: Argi and Dan O’Rourke

Sun Mar 12 13:05:37 PST 2000 — News.sonic.net (aka typhoon) is still having problems. We’ve tried to move it’s disks into another system to resolve the rebooting problems, but we had problems with this move. We’re currently in the process of building it back into the old chassis, but with the disks on two different SCSI busses. We’ll see how it goes, wish us luck! -Scott (who’s doing all the work) and Dane

Fri Mar 10 12:47:42 PST 2000 — Covad is having problems right now, specificly with IDSL. Many customers are offline, and Covad is working hard to resolve this. This is affecting all Covad partner ISPs, so you can be sure that they’re working fast! The latest update from Covad is that they’ve resolved most issues, but we’re seeing many customers still down. -John, Eli and Dane