Mon Jul 15 22:54:14 PDT 2002 — On Tuesday during the day, we’ll be turning up our new 100Mbps fiber link to Equinix, and deploying a Cisco 7206 and other equipment there. If all goes well, we’ll be moving our Cable & Wireless link from downtown Santa Rosa to San Jose sometime shortly afterwards.

On Wednesday morning at around 3am, we will be moving Focal (9811) dialup lines from Santa Rosa to San Francisco. Expected downtime is ten to fifteen minutes. No, we don’t drive that fast, we’ll have extra equipment in place in San Francisco to allow for a quick transition. Matt, Steve and Russ will be doing this migration for us. -Dane

Update: Wed Jul 17 07:31:10 PDT 2002 — Night Operations complete. The Focal dial-up lines were successfully moved to our San Francisco POP. There was less than 5 minutes of downtime during the move.

Update: Wed Jul 17 16:22:53 PDT 2002 — As a part of the move we renumbered the xxx-9811 dial pools NAS servers. A minority of ISDN routers may have to have their gateway address changed in order to function properly. -Matt, Nathan, Russ and Steve

Mon Jul 15 00:18:40 PDT 2002 — ssl.sonic.net, our shared secure web server has had a system disk failure. We are in the process of working on the on the system now and should have services fully restored in a few hours. Only the core OS is affected by this disk failure, all customer data is stored on our redundant Netapp cluster. – Kelsey and Nathan

Update: ssl.sonic.net has been completely restored on a new disk and it’s data has been verified by our backup software. It’s been up on the new hardware for over an hour and appears to be working properly.

Sun Jul 14 02:53:44 PDT 2002 — There seems to have been a power outage in Northern Santa Rosa, and we’ve noticed a number of T1 and DSL customers went offline. The T1 and ATG DSL customers now seem to be back online, but we have not been able to verify that the PacBell DSL customers are back up. Please let us know ASAP if you have any problems with your connection due to this event. -Dane, Matt and Russell (Xponentia)

Sun Jul 14 02:47:37 PDT 2002 — Our move of primary core storage and servers has been delayed due to problems getting the tape backup completed, indexed and verified. The full backup started at 5am Saturday, and as of 2am Sunday, is complete but not yet fully indexed, so it cannot be tested. For this reason, we’ve postponed the move of the storage array and all associated servers until the same time next weekend.

Instead, we’ll be doing a number of clean-up activities this evening, many of which are slightly service affecting. Steve will be doing a bit of tidying and rearranging of the 1003 dial pool, and customers will be disconnected as he moves equipment around. Nathan is planning to swap a T3 port adapter in one of the Cisco 7507s here for a HSSI, and expects about 30 seconds of inability to reach sites connected via Cable & Wireless. -Dane, Matt, Steve, Nathan and Kelsey

Sat Jul 13 04:09:42 PDT 2002 — News Server Updates: We’ve just completed a software upgrade to bring us up to the latest stable version of Typhoon, the news server software we use on our news reader box. We’ll be keeping a close eye on the news server over the next few days to see if this resolves the problems that it has been having this past week. -Kelsey UPDATE: The problems appear to have been resolved by the new version of Typhoon. It’s ran problem free for more than 12 hours. -Kelsey

Fri Jul 12 01:24:53 PDT 2002 — Night Op – We’re planning to move our core disk storage architecture on Sunday morning, with downtime beginning at about 1am.

Our Network Appliance F740 network file system cluster is the basis of Sonic.net’s storage solution. All user data resides on the two NetApp filers, and they’re configured in a completely redundant configuration. The drives are dual-channel fiber arbitrated loops, and are served by redundant processor heads. The units are RAID level four, WAFL filesystems, and include redundant power and cabling to all disks and the network itself at Gigabit speeds.

The move of the disk shelves themselves will be a service affecting move, and during the time that they are in transit, local web and ftp hosting, email and shell will be unavailable. Dialup, DSL and web browsing will be unaffected. We expect the downtime to be between an hour and an hour and a half, beginning just after 1am on Sunday morning. Actual server moves of redundant systems will begin at 11pm Saturday night, but these changes should be transparent.

The move to the new datacenter is nearing completion for Sonic.net equipment, and it’s been a good opportunity to redesign a number of network elements. Downtime has been very brief, and we appreciate your patience with any interruptions noted in the MOTD. -Dane

Wed Jul 10 18:58:53 PDT 2002 — Night Op – Redback SMS and DSL customer move. Tonight we will be moving the ATM DS3 which terminates Pac Bell DSL and FRATM customers. It is scheduled to take place at 3am and should last about 30-40 minutes as we relocate the equipment to the new data center.

Update: The move of the RedBack SMS DSL router and PacBell DSL customers is complete. Downtime was about twenty minutes. Pacific Bell did a great job of moving this circuit quickly and efficiently, and we had a very smooth transition of the equipment. -Matt, Dane, Eli, Kelsey, Nathan and Mike(2) from PacBell

Wed Jul 10 16:41:12 PDT 2002 — News Server Issues: news.sonic.net, our NNTP reader server has been experiencing stability problems for the past few days. This instability results in periodic refused connections as the server process reinitializes. We have been unable to find the cause and are in the process of working with the software vendor to resolve this as soon as possible. -Kelsey

Update: We are still experiencing trouble with the news server. The vendor has recommended a version upgrade and is currently analyzing our cores. We will attempt the upgrade as soon as we reach an appropriate maintenance window. -Kelsey

Tue Jul 9 11:11:43 PDT 2002 — Web performance impacted. A denial-of-service attack is affecting the performance of one of our web servers; this may result in slower response times when loading pages hosted at Sonic.net. Our operations crew is resolving the issue and web performance should return to normal shortly. We maintain a pool of redundant, load-balanced web servers, which greatly reduces the severity of problems of this type. -Eli and Russ

Tue Jul 9 18:27:50 PDT 2002 — Tonight we will be moving BroadLink wireless customers from one ATM router to another in order to prep for the migration of Pacific Bell customers tomorrow night. In order to have as much flexibility as possible, we’re deploying a second RedBack SMS, which means we can keep BroadLink and PacBell customers separate and move them separately. This will reduce downtime and potential for problems during migration for both groups of customers. BroadLink customers can expect about ten minutes of downtime tonight at around 2am.

We will also be swapping one of our Cisco 7206 edge routers for a loaner unit (thanks, John Harkin!) to enable final deployment of our equipment at Equinix next week. This moves us closer to completion of our new network design, which includes four Cisco 7507 RSP4 routers in Santa Rosa (two edge, two customer attach) and two Cisco 7206 VXR-400 routers at Equinix and Focal in San Jose and San Francisco respectively. The loaner 7206 will be used to handle our connection to Cable & Wireless until that’s moved over to the equipment deployed at Equinix next week. -Dane, Kelsey, Nathan, Chris (BroadLink) and John Harkin

Update – We’ve moved all BroadLink customers to the new RedBack router in downtown Santa Rosa in prep for the move of our large SMS serving Pacific Bell customers tomorrow night. We have also replaced delta, the current C&W edge. All appears to be well. In a few minutes, we’ll be making a change to the sr2 link to address some performance issues that have recently affected the 1001 dial group. -Dane, Nathan and Kelsey