Month: June 2002

Sat Jun 29 12:56:46 PDT 2002 — Over the next couple weeks, we’ll be moving a number of servers and routers between midnight and 3 AM.

Some are internal, and won’t be noticed by customers. The internal servers include eye (network monitoring), bedlam (staff admin), logger (log storage), adelie and boom (fax). We will also be splitting up the RADIUS servers, moving one at a time so that user authentication is not impacted.

Other server moves will cause some customer impact. In particular, the move we anticipate to be the most intrusive and which will take the longest is the relocation of our huge NetApp filer disk array deployment. The units are dual-headed and fully redundant, but cannot be split between sites well without downtime. We will need to move the NetApps and at the same time the load balancing switches and our array of web and mail servers.

This is likely to interrupt email and hosting for two to three hours, between around 2 AM and 5 AM one night next week. We’ll keep you posted as we solidify the schedule. We will strive to keep downtime as short as possible.

Various routers and dialup equipment will also be moved over the next few weeks. Most of the dedicated T1, T3 and Frame Relay customers will move first, this coming Tuesday at 3 AM. Dialup gear for our Focal circuits will move next, and this is tentatively scheduled for Friday the 5th. This will be followed by PacBell dialup and then ATM DS3 (DSL). The router and dialup equipment moves are anticipated to take 30 to 45 minutes each.

We will post updates here as we solidify schedules and move equipment, and we appreciate your patience with any brief service interruptions as we wrap up this migration to our new facility. -Staff

Thu Jun 27 18:37:52 PDT 2002 — For a special event, Sonic.net support will be closed on Friday from 4pm to 7pm, in addition to their normally scheduled Friday training time from 1pm to 2pm. They will be open from 7pm to 9pm as usual. In addition, our administrative office will close early, at 4pm instead of 5pm. -Dane

Thu Jun 27 18:34:19 PDT 2002 — Night operations: We will be doing a low impact night operation which will not affect many services. Non service affecting moves will be made for time, ping and nameservers. We’ll also be moving flash (ftp) and timber (sql) after midnight, which will result in brief interruptions of those services. -Dane, Kelsey and Nathan

UPDATE: We moved six servers this evening: rs2, our caching name server, ping, ssl, timber, and b.smtp-out. They are all up and running in our new datacenter. -Kelsey, Nathan and Matt.

Wed Jun 26 22:38:02 PDT 2002 — BroadLink Communications submits voluntary Chapter 11 filing. BroadLink serves wireless Internet service for about 950 Sonic.net customers. We do not expect any service interruption.

On the company operations, Tim McAllister, Executive VP and co-founder of BroadLink said “Service to BroadLink customers in our markets will remain business as usual.”

We’ll keep Sonic.net customers posted on any developments with our business partner. -Dane

Wed Jun 26 09:36:32 PDT 2002 — DSL outage for certain customers. Currently a number of DSL customers on VP 1 are having a sync-no-surf type outage. This only affects certain Santa Rosa DSL customers. We are in contact with SBC/ASI to determine the nature of the problem and get it resolved- in the interim, please remember that all DSL accounts also have dial-up capability. -Aaron and Eli

Update: SBC/ASI found and fixed a DSLAM problem and normal DSL service has been restored.

Tue Jun 25 16:13:52 PDT 2002 — Wireless Rooftop Router Maintenance Sonic.net will be performing maintenance on Rooftop customers in the Rohnert Park ‘M’ Section today at 4:30pm. -Matt and Bryan

Tue Jun 25 14:09:14 PDT 2002 — Our Cidera satellite news feed has been restored. We experienced a number of hardware related difficulties restoring the service and had to wait for additional replacement hardware shipped from Cidera’s NOC on the East Coast. Our Cidera feed accounts for the majority of our binary news articles. Our 16 other news peers are capable of maintaining excellent small article completion rates and filling in the missing articles from Cidera. However, these feeds are not capable of offering us a complete feed in the event that Cidera is offline due to the amazing bandwidth requirements of a full news feed. We are continually improving our NNTP feed by seeking out peers with key NNTP hubs. -Kelsey

Mon Jun 24 22:02:49 PDT 2002 — Night Op – UUNet T3 Migration Wednesday morning at 1:00am we will be bringing up our new UUNet T3 at our Focal POP in San Francisco. The UUNet T3 will be backhauled to the data center at 2250 Apollo Way on a Pac Bell T3 and terminate in a new Cisco 7507.

We plan to shift all UUNet traffic to traverse this new path and this time we will have the CPU power on the Cisco to handle it. This new deployment is a major shift in our traffic flow but customers will not experience any service interruption. -Ops Staff

Update, Wed Jun 26 02:25:28 PDT 2002: UUNet T3 Migration complete The UUNet migration has been completed successfully. We are now connecting to UUNet via a T3 at our Focal POP in San Francisco. All UUNet traffic from 300 B Street now flows to the data center at Apollo and then down to SF. This allows us to begin moving servers and equipment to Apollo over the next few weeks. -Matt, Dane, Nathan, Kelsey and Zeke

Sun Jun 23 13:26:44 PDT 2002 — Sonic.net support and SBC-ASI are working together to diagnose ongoing issues with a small number of DSL customers who are served by two DSLAMs in the Santa Rosa main central office (CO). Aaron here and Patrick at the SBC-ASI DOC are currently working to rebuild some end-user configurations in attempt to isolate the trouble, which was first observed about six days ago.

The affected customers have PVCs in the 1/.. and 39/.. ranges. You can determine your PVC by viewing your services in the member tools section of our website. Affected customers are suffering high packet loss and latency. -Dane, Aaron and staff

Update, Mon Jun 24 17:45:42 PDT 2002 – After some re-working of the circuits by ASI, and extensive testing today, the issue appears to be resolved, and performance to the affected customers has been restored. – Eli, Aaron, and staff

Sun Jun 23 00:04:30 PDT 2002 — Night Op – UUNet T3 Migration We will be shifting traffic from our UUNet T3 at 300 B Street to a new T3 circuit located at our Focal POP in San Francisco tonight at 12:30am. We will keep both circuits up, but will reconfigure BGP to announce out the T3 at Focal. This will be a seamless operation, but is a major shift in our network traffic flow.

Once complete, UUNet traffic will traverse the link between 300 B Street and Apollo Way before being routed to Focal and out the new UUNet T3. This is Phase 1 in bringing connectivity to our new NOC. Next week we will begin Phase 2 of the migration. In this phase we move equipment and servers from 300 B to Apollo. This will take place in 3 separate operations next week. -Matt, Nathan and Kelsey

Update, Sun Jun 23 02:59:41 PDT 2002: We successfully completed the turn up and BGP configuration of the new UUNet circuit, but with a minor setback. The router which terminates the T3 bridge between 300 B and Apollo as well as the T3 to Focal was unable to handle the load the new configuration put on it. Nathan determined that bridging the cross-site T3 and terminating the Focal T3 in the same router was too CPU intensive for the Cisco 7507. We have another 7507 on the way and will put the bridge and T3 on separate routers to reduce the CPU load. We returned traffic to the UUNet circuit at 300 B Street and will wait for the new equipment before making these changes permanent. The was no downtime during this operation.