Author: admin

Wed Jul 10 16:41:12 PDT 2002 — News Server Issues: news.sonic.net, our NNTP reader server has been experiencing stability problems for the past few days. This instability results in periodic refused connections as the server process reinitializes. We have been unable to find the cause and are in the process of working with the software vendor to resolve this as soon as possible. -Kelsey

Update: We are still experiencing trouble with the news server. The vendor has recommended a version upgrade and is currently analyzing our cores. We will attempt the upgrade as soon as we reach an appropriate maintenance window. -Kelsey

Tue Jul 9 11:11:43 PDT 2002 — Web performance impacted. A denial-of-service attack is affecting the performance of one of our web servers; this may result in slower response times when loading pages hosted at Sonic.net. Our operations crew is resolving the issue and web performance should return to normal shortly. We maintain a pool of redundant, load-balanced web servers, which greatly reduces the severity of problems of this type. -Eli and Russ

Tue Jul 9 18:27:50 PDT 2002 — Tonight we will be moving BroadLink wireless customers from one ATM router to another in order to prep for the migration of Pacific Bell customers tomorrow night. In order to have as much flexibility as possible, we’re deploying a second RedBack SMS, which means we can keep BroadLink and PacBell customers separate and move them separately. This will reduce downtime and potential for problems during migration for both groups of customers. BroadLink customers can expect about ten minutes of downtime tonight at around 2am.

We will also be swapping one of our Cisco 7206 edge routers for a loaner unit (thanks, John Harkin!) to enable final deployment of our equipment at Equinix next week. This moves us closer to completion of our new network design, which includes four Cisco 7507 RSP4 routers in Santa Rosa (two edge, two customer attach) and two Cisco 7206 VXR-400 routers at Equinix and Focal in San Jose and San Francisco respectively. The loaner 7206 will be used to handle our connection to Cable & Wireless until that’s moved over to the equipment deployed at Equinix next week. -Dane, Kelsey, Nathan, Chris (BroadLink) and John Harkin

Update – We’ve moved all BroadLink customers to the new RedBack router in downtown Santa Rosa in prep for the move of our large SMS serving Pacific Bell customers tomorrow night. We have also replaced delta, the current C&W edge. All appears to be well. In a few minutes, we’ll be making a change to the sr2 link to address some performance issues that have recently affected the 1001 dial group. -Dane, Nathan and Kelsey

Tue Jul 9 00:30:18 PDT 2002 — This evening at 3am, we will be moving dialup equipment serving our primary Pacific Bell Santa Rosa number, “1003”. Downtime is expected to be between 15 and 30 minutes. Note that we have at least three other numbers that all Santa Rosa area customers can use. See the POP finder at the following URL to look up backup dialups for your area. Printing the list for future reference would be a great idea!

www.sonic.net/cgi-bin/pops.pl

Update: The move of the 3Com/USR Total Control Enterprise Hub and T3 mux equipment is complete, and we’ve moved 736 dialup lines of capacity. Downtime was about ten minutes. If you notice any new dialup issues on 1003 on Tuesday, please let support know.

-Dane, Steve, Nathan, Kelsey, Mike (PacBell) and various helpers

Wed Jul 3 06:19:18 PDT 2002 — Night Op Complete. We have successfully moved mega and the circuits that it terminates. The majority of the move was completed in 45 minutes with the exception of a single T1 that took longer. Thanks to Mike from Pac Bell for his good work!

-Matt, Nathan, Kelsey and Mike (Pac Bell)

Tue Jul 2 08:32:11 PDT 2002 — We are experiencing authentication errors on our Focal POP. We are investigating this now and should have it fixed shortly. Meanwhile we do offer alternate dialup numbers in all of our dialup areas. To find an alternate dialup number you can go to www.sonic.net/cgi-bin/pops.pl . – Steve

Update – Tue Jul 2 09:36:39 PDT 2002 — We found the problem with the authentication errors and it has been corrected. The affected pop is now back up and passing traffic. – Steve, Matt and Nathan.

Tue Jul 2 18:31:47 PDT 2002 — Our news server software, Typhoon, has exited three times in the past 24 hours, causing a brief interruption in the availability of the Usenet news server each time. We’re keeping an eye on it, and will do some investigation to work out what it’s not happy about. -Dane, Scott and Kelsey.

Mon Jul 1 18:19:57 PDT 2002 — Night operations: Tuesday morning at 3am we will be moving mega (a Cisco 7507) which terminates T1, T3 and some Frame-Relay customers. This requires us to coordinate with Pac Bell to move the circuits to Apollo as we un-rack mega and drive it to the new data center. We expect the interruption to last about 30-45 minutes and will only impact customers who terminate in that router. -Operations Staff

UPDATE – Tue Jul 2 00:23:35 PDT 2002 — During this evenings migration, dial-up POP’s will experience various outages as we move circuits. Specifically our Sebastopol (numbers ending in 8812) and Oakland POP’s (numbers ending in 0174) which terminate on a T3 in mega. Some customers may find backup numbers by using our POP Finder tool located at:

www.sonic.net/cgi-bin/pops.pl

UPDATE – Tue Jul 2 05:01:10 PDT 2002 — Night Op Postponed. Pac Bell missed our 3am cut time as work in the Rohnert Park CO was not complete. Bell discovered the problem about 3:30am and by 4:20am it was clear that it was going to take too long to get the work finished. The move has been rescheduled for 3am Wednesday and Bell promises to have their work complete prior to the cut time. There was no service interruption during this operation. -Matt, Dane, Nathan, Kelsey, Zeke, Jon H., Scott B. and Mike (Pac Bell)

Sat Jun 29 12:56:46 PDT 2002 — Over the next couple weeks, we’ll be moving a number of servers and routers between midnight and 3 AM.

Some are internal, and won’t be noticed by customers. The internal servers include eye (network monitoring), bedlam (staff admin), logger (log storage), adelie and boom (fax). We will also be splitting up the RADIUS servers, moving one at a time so that user authentication is not impacted.

Other server moves will cause some customer impact. In particular, the move we anticipate to be the most intrusive and which will take the longest is the relocation of our huge NetApp filer disk array deployment. The units are dual-headed and fully redundant, but cannot be split between sites well without downtime. We will need to move the NetApps and at the same time the load balancing switches and our array of web and mail servers.

This is likely to interrupt email and hosting for two to three hours, between around 2 AM and 5 AM one night next week. We’ll keep you posted as we solidify the schedule. We will strive to keep downtime as short as possible.

Various routers and dialup equipment will also be moved over the next few weeks. Most of the dedicated T1, T3 and Frame Relay customers will move first, this coming Tuesday at 3 AM. Dialup gear for our Focal circuits will move next, and this is tentatively scheduled for Friday the 5th. This will be followed by PacBell dialup and then ATM DS3 (DSL). The router and dialup equipment moves are anticipated to take 30 to 45 minutes each.

We will post updates here as we solidify schedules and move equipment, and we appreciate your patience with any brief service interruptions as we wrap up this migration to our new facility. -Staff

Thu Jun 27 18:37:52 PDT 2002 — For a special event, Sonic.net support will be closed on Friday from 4pm to 7pm, in addition to their normally scheduled Friday training time from 1pm to 2pm. They will be open from 7pm to 9pm as usual. In addition, our administrative office will close early, at 4pm instead of 5pm. -Dane