Month: July 2002

Tue Jul 9 18:27:50 PDT 2002 — Tonight we will be moving BroadLink wireless customers from one ATM router to another in order to prep for the migration of Pacific Bell customers tomorrow night. In order to have as much flexibility as possible, we’re deploying a second RedBack SMS, which means we can keep BroadLink and PacBell customers separate and move them separately. This will reduce downtime and potential for problems during migration for both groups of customers. BroadLink customers can expect about ten minutes of downtime tonight at around 2am.

We will also be swapping one of our Cisco 7206 edge routers for a loaner unit (thanks, John Harkin!) to enable final deployment of our equipment at Equinix next week. This moves us closer to completion of our new network design, which includes four Cisco 7507 RSP4 routers in Santa Rosa (two edge, two customer attach) and two Cisco 7206 VXR-400 routers at Equinix and Focal in San Jose and San Francisco respectively. The loaner 7206 will be used to handle our connection to Cable & Wireless until that’s moved over to the equipment deployed at Equinix next week. -Dane, Kelsey, Nathan, Chris (BroadLink) and John Harkin

Update – We’ve moved all BroadLink customers to the new RedBack router in downtown Santa Rosa in prep for the move of our large SMS serving Pacific Bell customers tomorrow night. We have also replaced delta, the current C&W edge. All appears to be well. In a few minutes, we’ll be making a change to the sr2 link to address some performance issues that have recently affected the 1001 dial group. -Dane, Nathan and Kelsey

Tue Jul 9 00:30:18 PDT 2002 — This evening at 3am, we will be moving dialup equipment serving our primary Pacific Bell Santa Rosa number, “1003”. Downtime is expected to be between 15 and 30 minutes. Note that we have at least three other numbers that all Santa Rosa area customers can use. See the POP finder at the following URL to look up backup dialups for your area. Printing the list for future reference would be a great idea!

www.sonic.net/cgi-bin/pops.pl

Update: The move of the 3Com/USR Total Control Enterprise Hub and T3 mux equipment is complete, and we’ve moved 736 dialup lines of capacity. Downtime was about ten minutes. If you notice any new dialup issues on 1003 on Tuesday, please let support know.

-Dane, Steve, Nathan, Kelsey, Mike (PacBell) and various helpers

Wed Jul 3 06:19:18 PDT 2002 — Night Op Complete. We have successfully moved mega and the circuits that it terminates. The majority of the move was completed in 45 minutes with the exception of a single T1 that took longer. Thanks to Mike from Pac Bell for his good work!

-Matt, Nathan, Kelsey and Mike (Pac Bell)

Tue Jul 2 08:32:11 PDT 2002 — We are experiencing authentication errors on our Focal POP. We are investigating this now and should have it fixed shortly. Meanwhile we do offer alternate dialup numbers in all of our dialup areas. To find an alternate dialup number you can go to www.sonic.net/cgi-bin/pops.pl . – Steve

Update – Tue Jul 2 09:36:39 PDT 2002 — We found the problem with the authentication errors and it has been corrected. The affected pop is now back up and passing traffic. – Steve, Matt and Nathan.

Tue Jul 2 18:31:47 PDT 2002 — Our news server software, Typhoon, has exited three times in the past 24 hours, causing a brief interruption in the availability of the Usenet news server each time. We’re keeping an eye on it, and will do some investigation to work out what it’s not happy about. -Dane, Scott and Kelsey.

Mon Jul 1 18:19:57 PDT 2002 — Night operations: Tuesday morning at 3am we will be moving mega (a Cisco 7507) which terminates T1, T3 and some Frame-Relay customers. This requires us to coordinate with Pac Bell to move the circuits to Apollo as we un-rack mega and drive it to the new data center. We expect the interruption to last about 30-45 minutes and will only impact customers who terminate in that router. -Operations Staff

UPDATE – Tue Jul 2 00:23:35 PDT 2002 — During this evenings migration, dial-up POP’s will experience various outages as we move circuits. Specifically our Sebastopol (numbers ending in 8812) and Oakland POP’s (numbers ending in 0174) which terminate on a T3 in mega. Some customers may find backup numbers by using our POP Finder tool located at:

www.sonic.net/cgi-bin/pops.pl

UPDATE – Tue Jul 2 05:01:10 PDT 2002 — Night Op Postponed. Pac Bell missed our 3am cut time as work in the Rohnert Park CO was not complete. Bell discovered the problem about 3:30am and by 4:20am it was clear that it was going to take too long to get the work finished. The move has been rescheduled for 3am Wednesday and Bell promises to have their work complete prior to the cut time. There was no service interruption during this operation. -Matt, Dane, Nathan, Kelsey, Zeke, Jon H., Scott B. and Mike (Pac Bell)