Night Operations.

Tue Jan 23 14:50:47 PST 2001 — Night Operations. Tonight we are going to have a few brief service outages while we upgrade our core Extreme Networks BlackDiamond switch’s management modules. The upgrade affords us a higher port density and an increased feature set including L4 switching capabilities. While the SE from Extreme Networks is working on the switch we will also be taking Freezer and Icebox down to resolve a cluster configuration problem that we introduced when we added the second shelves last week. We will begin work shortly after Midnight and don’t anticipate more than a few minutes of downtime to complete both projects. Please note that while the switch is offline (about 90 seconds) Sonic.net will be off of the net. While the two NetApps are being rebooted (about 120 seconds), email, shell and web hosting services will pause. -Kelsey

While some in the Santa Rosa area have been…

Wed Jan 17 16:50:01 PST 2001 — While some in the Santa Rosa area have been subject to rolling blackouts, Sonic.net is located on a high-priority grid that is exempt from these measures. In the unlikely event that there is a blackout in this area, our UPS capacity and a 150Kva generator will keep our system up. We tested this procedure last week in a power drill, and ran Sonic.net on Diesel for nearly an hour. -The Sonic Operations Team

Night Operations: We are going to be…

Wed Jan 17 13:18:28 PST 2001 — Night Operations: We are going to be performing upgrades and maintenance on our network tonight. We will be starting work at 1:00AM Thursday morning. We have purchased a warm spare for lily the T3 MUX and are going to simulate a failure situation and swap to the spare. During the testing period the top half of the 522-1003 dial-up group will be unavailable and will also drop any active connections. Some of our remote pops are also connected through the MUX and will briefly be isolated from Sonic’s networks and the rest of the Internet.

We will also be installing two new disk shelves to our NetApp filers to give us the ability to add disks as need to increase capacity. We will have to shutdown both filers so unfortunately the clustering won’t help us here. While the NetApps are down, all services will be offline. They shouldn’t be down for more than 5 minutes.

PacBell will be migrating one of our T3’s into our OC12 SONET ring from the DS3 that it’s delivered on now.

We will also be installing an additional 150GB of spool space to the news server. (The additional spool space won’t be brought online until Thursday day.)

-The Sonic Operations Team

Denial Of Service attack.

Sun Jan 14 21:16:05 PST 2001 — Denial Of Service attack. Starting about 20 minutes ago, our network started getting flooded with traffic, much like a Smurf attack but with a slightly different signature. The attack has ended, and we are monitoring/investigating the situation. – Scott, Eli, Kelsey

Tsunami’s password database had become…

Fri Jan 12 11:38:18 PST 2001 — Tsunami’s password database had become corrupt and lost a little over 1000 user’s passwords. I was able to restore from our tape backup which completed around 7:00 AM this morning. As a result about 50 passwords have been permanently lost. If you still cannot access your mailbox, please give support a call and we will reset your password. We will also be contacting the users whose passwords have been lost. I’ve also installed some new backup routines so in the event that the password database becomes corrupt again, we will have a more recent backup to work from. -Kelsey