Month: December 1999

Thu Dec 9 17:30:03 PST 1999 — After quite a bit of diagnosis on our 1003 dial group, we’ve found that 10 PRIs, or 230 ports worth of capacity were not properly configured into the hunt group by PacBell when the lines were installed some months ago. As we’ve filled up the other 22 PRIs in this group, it appears that we’ve started to run into capacity problems which are not being manifest in the usual ‘busy signal’ way. PacBell this afternoon has corrected this, and we now have all 32 PRIs in service on this dial group. Hopefully this will resolve the poor performance in this dial group. -Dane

Thu Dec 9 14:21:13 PST 1999 — For the second night in a row, PacBell’s techs were unable to complete the card upgrade on our SMDS T3 circuit to UUNet. On Wednesday AM, the card was installed, but the PacBell staff didn’t know how to configure it. On Thursday AM, someone who knew how to configure the card came in for the upgrade, but had problems getting the config to work.

I’ve escalated this issue with our account team, and the techs at PacBell are being equipped with a full set of contacts so that if they have problems, they’ll be able to get the support they need to get this upgrade in place. Meanwhile, latency is much higher than it should be, and packet loss is at 7%-9%. We appreciate your patience as we escalate our harassment of PacBell’s Network Data Products Service Center until they get it right. The maintenance window for this fix is 3am until 4am on Friday morning, and I’ll be on the phone myself with PacBell to make sure there are no further excuses. -Dane

Wed Dec 8 12:33:27 PST 1999 — PacBell has scheduled maintenance on our SMDS T3 to UUNet for tonight at 3 AM PST. This card swap moves us to a duplexed card with more CPU power, and PacBell believes that this will address the latency and packet loss that we’re seeing on this primary circuit right now. We’ve been through a long process, and have tried many fixes, and we hope that this brings us to final resolution. Thanks for your patience during times of slow performance. -Dane

Wed Dec 8 12:22:57 PST 1999 — We have upgraded memory and swap in the log processing system to three quarters of a gigabyte. This upgrade was needed to handle log processing of www.sonic.net and multihomed logs. Looking at the complete stats, we’re serving over three million hits per day right now! Stats for multihomed sites and for www.sonic.net are available using the stats interface at www.sonic.net/stats/. To access stats for a multihome, you must be the owner, but all users can browse stats for www.sonic.net.

Tue Dec 7 12:39:35 PST 1999 — Update on T3 packet loss. PacBell has gotten support advice which is leading them toward a new high speed duplexed line card for our SMDS T3 circuit to UUNet. They are currently working on the engineering design for this change, and we’re hoping they’ll be able to shift us over tonight or tommorow night. Meanwhile, latency is high, and packet loss is present. Thanks for your patience as Sonic.net, UUNet and PacBell work through this trouble. -Dane

Tue Dec 7 15:24:09 PST 1999 — Sonic.net has partnered with Akamai Technologies to deliver content from the most popular websites on a ‘zero hop’ basis.

Akamai has colocated five new servers here in Sonic.net’s Santa Rosa data center which are dedicated to delivering content to our customers. Used by major sites including Yahoo! and CNN, Akamai’s FreeFlow technology enhances Web performance by accelerating the delivery of robust, secure content with animation, graphics, streaming media and interactivity.

For more information and a photo of the new servers, see:

www.sonic.net/expansion/akamai/

Mon Dec 6 17:56:30 PST 1999 — We’re seeing continued packet loss and latency on our T3 circuit to UUNet, which is our primary connection to the Internet. We’ve shifted some traffic toward our backup circuits, but this ongoing trouble is certainly impacting interactive performance. UUNet and PacBell are continuing their work on finding and eliminating this trouble, and we all appreciate your patience. -Dane

Sat Dec 4 10:19:04 PST 1999 — During our night operations, we moved most of our core systems from our old Extreme Networks Summit 48 to the huge new Black Diamond switch. All of the systems except one were correct in auto-negotiating their full or half duplex 100 megabit fast ethernet settings. mega.sonic.net, our core Cisco router did not properly negotiate it’s link speed with the new switch. Because these settings were out of sync, as traffic increased into the AM hours, CRC errors began occuring on that connection, causing packet loss. I’ve manually overridden the auto-negotiated setting of half duplex in the switch, and will persue this with the switch vendor to figure out why the systems are failing to negotiate. The impact here has been slower than usual Internet connections this AM, and this has been resolved. Sorry about any inconveniance this may have caused!

The new Black Diamond switch is huge, with capacity to support up to 256 connections to systems running at 100 megabit at full speed. It’s non-blocking architecture allows for up to 64 gigabits of traffic in total, or full wire speed on all circuits all at once. Gigabit ethernet on fiber is also supported for up to 48 ports which could be connected to other large switches or directly to huge servers with high bandwidth needs. This is the largest enterprise network switch that we’re aware of on the market, and it provides Sonic.net with a great new core structure. -Dane

Sat Dec 4 05:29:59 PST 1999 — We completed our night operations and systems maintenance here at around 5am. There were a few brief and isolated service outages of individual features between 1am and 4am as we moved machines and connections in our data center. These changes included a memory upgrade for thunder.sonic.net, a new power plant for the 1001 dial group and a migration of most core systems to our new Ethernet switch. G’night! -Dane, Scott, Dave, Kelsey, Tony and Evan

Fri Dec 3 09:44:44 PST 1999 — During an alarm system run, fridge.sonic.net rebooted itself causing downtime of email and web space for three minutes and 28 seconds. fridge.sonic.net is internally highly redundant, and we’ve never had an issue like this with it. We’re working with Network Appliance and our alarm vendor to see what might have caused this. We suspect some sort of power surge or RF trouble. -Dane