Tue Jun 25 16:13:52 PDT 2002 — Wireless Rooftop Router Maintenance Sonic.net will be performing maintenance on Rooftop customers in the Rohnert Park ‘M’ Section today at 4:30pm. -Matt and Bryan

Tue Jun 25 14:09:14 PDT 2002 — Our Cidera satellite news feed has been restored. We experienced a number of hardware related difficulties restoring the service and had to wait for additional replacement hardware shipped from Cidera’s NOC on the East Coast. Our Cidera feed accounts for the majority of our binary news articles. Our 16 other news peers are capable of maintaining excellent small article completion rates and filling in the missing articles from Cidera. However, these feeds are not capable of offering us a complete feed in the event that Cidera is offline due to the amazing bandwidth requirements of a full news feed. We are continually improving our NNTP feed by seeking out peers with key NNTP hubs. -Kelsey

Mon Jun 24 22:02:49 PDT 2002 — Night Op – UUNet T3 Migration Wednesday morning at 1:00am we will be bringing up our new UUNet T3 at our Focal POP in San Francisco. The UUNet T3 will be backhauled to the data center at 2250 Apollo Way on a Pac Bell T3 and terminate in a new Cisco 7507.

We plan to shift all UUNet traffic to traverse this new path and this time we will have the CPU power on the Cisco to handle it. This new deployment is a major shift in our traffic flow but customers will not experience any service interruption. -Ops Staff

Update, Wed Jun 26 02:25:28 PDT 2002: UUNet T3 Migration complete The UUNet migration has been completed successfully. We are now connecting to UUNet via a T3 at our Focal POP in San Francisco. All UUNet traffic from 300 B Street now flows to the data center at Apollo and then down to SF. This allows us to begin moving servers and equipment to Apollo over the next few weeks. -Matt, Dane, Nathan, Kelsey and Zeke

Sun Jun 23 13:26:44 PDT 2002 — Sonic.net support and SBC-ASI are working together to diagnose ongoing issues with a small number of DSL customers who are served by two DSLAMs in the Santa Rosa main central office (CO). Aaron here and Patrick at the SBC-ASI DOC are currently working to rebuild some end-user configurations in attempt to isolate the trouble, which was first observed about six days ago.

The affected customers have PVCs in the 1/.. and 39/.. ranges. You can determine your PVC by viewing your services in the member tools section of our website. Affected customers are suffering high packet loss and latency. -Dane, Aaron and staff

Update, Mon Jun 24 17:45:42 PDT 2002 – After some re-working of the circuits by ASI, and extensive testing today, the issue appears to be resolved, and performance to the affected customers has been restored. – Eli, Aaron, and staff

Sun Jun 23 00:04:30 PDT 2002 — Night Op – UUNet T3 Migration We will be shifting traffic from our UUNet T3 at 300 B Street to a new T3 circuit located at our Focal POP in San Francisco tonight at 12:30am. We will keep both circuits up, but will reconfigure BGP to announce out the T3 at Focal. This will be a seamless operation, but is a major shift in our network traffic flow.

Once complete, UUNet traffic will traverse the link between 300 B Street and Apollo Way before being routed to Focal and out the new UUNet T3. This is Phase 1 in bringing connectivity to our new NOC. Next week we will begin Phase 2 of the migration. In this phase we move equipment and servers from 300 B to Apollo. This will take place in 3 separate operations next week. -Matt, Nathan and Kelsey

Update, Sun Jun 23 02:59:41 PDT 2002: We successfully completed the turn up and BGP configuration of the new UUNet circuit, but with a minor setback. The router which terminates the T3 bridge between 300 B and Apollo as well as the T3 to Focal was unable to handle the load the new configuration put on it. Nathan determined that bridging the cross-site T3 and terminating the Focal T3 in the same router was too CPU intensive for the Cisco 7507. We have another 7507 on the way and will put the bridge and T3 on separate routers to reduce the CPU load. We returned traffic to the UUNet circuit at 300 B Street and will wait for the new equipment before making these changes permanent. The was no downtime during this operation.

Thu Jun 20 11:23:11 PDT 2002 — BroadLink is currently having technical difficulties with their head end aggregation router. They’re on site working to repair the problem now. The problem manifests as packet loss and high latency. The same trouble happened yesterday evening as well, and they’re working to implement a permanent solution. -Dane

Update: Broadlink’s network problem has been resolved.

Thu Jun 20 01:24:15 PDT 2002 — Sonic.net administration servers move. Tonight we are moving our internal administration servers into the new data center. This equipment doesn’t serve customers directly and will not cause any service interruption. We are also moving the office LAN onto the cross-site T3. This takes us off the enhanced DSL circuit we have been running on for the past two weeks. -Matt, Kelsey and Nathan

Tue Jun 18 11:22:13 PDT 2002 — More DSL maintenance: we have seen continued packet loss for a few DSL customers and are still working with ASI on a solution. At this time, SBC has optimized and restored our ATM DS3 connection and has escalated the issue to the DSL group at ASI. A small number of Santa Rosa customers may still be seeing intermittent packet loss and impaired performance; we are awaiting an ETR from ASI. -Aaron, Eli, Chris

Tue Jun 18 20:45:12 PDT 2002 — We’ve got a bridged T3 online between our B St and Apollo Way datacenters, and are preparing to move our staff and some test equipment tomorrow. Kudos to Kelsey and Nathan, who discovered that virtually every 12.x version of IOS has bugs in the IRB protocol implementation. After much troubleshooting, a version that worked was found and demonstrated, at 3am!

In addition tomorrow, we’ll be deploying a new Cisco router in San Francisco, at our new hub location there. It’ll be served by a T3 to Apollo, and will also connect to UUNet. The actual shift of UUNet traffic to this link will be done in the next few nights. We’ll post notice of this here, and it’ll be done in the middle of the night.

Our next major migration item is the move of Focal dialup numbers (XXX-9811) from B St to our San Francisco location. This is scheduled to occur at 3am on Friday morning the 21st. When this happens, connections will be dropped for customers calling these numbers, and they will need to re-dial.

We’ll keep you posted here as we work on the elements of this migration.

-Staff

Tue Jun 18 00:14:45 PDT 2002 — Our Cidera satellite news feed is offline due to a hardware failure in their server which translates the multicast satellite feed into a unicast NNTP feed to our NNTP hub. We’ve contacted their NOC and expect a replacement unit to be shipped in the morning. We anticipate that we will have degraded binary completion rates until the replacement server is online. -Kelsey and Matt