Mon Jun 24 22:02:49 PDT 2002 — Night Op – UUNet T3 Migration Wednesday morning at 1:00am we will be bringing up our new UUNet T3 at our Focal POP in San Francisco. The UUNet T3 will be backhauled to the data center at 2250 Apollo Way on a Pac Bell T3 and terminate in a new Cisco 7507.

We plan to shift all UUNet traffic to traverse this new path and this time we will have the CPU power on the Cisco to handle it. This new deployment is a major shift in our traffic flow but customers will not experience any service interruption. -Ops Staff

Update, Wed Jun 26 02:25:28 PDT 2002: UUNet T3 Migration complete The UUNet migration has been completed successfully. We are now connecting to UUNet via a T3 at our Focal POP in San Francisco. All UUNet traffic from 300 B Street now flows to the data center at Apollo and then down to SF. This allows us to begin moving servers and equipment to Apollo over the next few weeks. -Matt, Dane, Nathan, Kelsey and Zeke

Sun Jun 23 13:26:44 PDT 2002 — Sonic.net support and SBC-ASI are working together to diagnose ongoing issues with a small number of DSL customers who are served by two DSLAMs in the Santa Rosa main central office (CO). Aaron here and Patrick at the SBC-ASI DOC are currently working to rebuild some end-user configurations in attempt to isolate the trouble, which was first observed about six days ago.

The affected customers have PVCs in the 1/.. and 39/.. ranges. You can determine your PVC by viewing your services in the member tools section of our website. Affected customers are suffering high packet loss and latency. -Dane, Aaron and staff

Update, Mon Jun 24 17:45:42 PDT 2002 – After some re-working of the circuits by ASI, and extensive testing today, the issue appears to be resolved, and performance to the affected customers has been restored. – Eli, Aaron, and staff

Sun Jun 23 00:04:30 PDT 2002 — Night Op – UUNet T3 Migration We will be shifting traffic from our UUNet T3 at 300 B Street to a new T3 circuit located at our Focal POP in San Francisco tonight at 12:30am. We will keep both circuits up, but will reconfigure BGP to announce out the T3 at Focal. This will be a seamless operation, but is a major shift in our network traffic flow.

Once complete, UUNet traffic will traverse the link between 300 B Street and Apollo Way before being routed to Focal and out the new UUNet T3. This is Phase 1 in bringing connectivity to our new NOC. Next week we will begin Phase 2 of the migration. In this phase we move equipment and servers from 300 B to Apollo. This will take place in 3 separate operations next week. -Matt, Nathan and Kelsey

Update, Sun Jun 23 02:59:41 PDT 2002: We successfully completed the turn up and BGP configuration of the new UUNet circuit, but with a minor setback. The router which terminates the T3 bridge between 300 B and Apollo as well as the T3 to Focal was unable to handle the load the new configuration put on it. Nathan determined that bridging the cross-site T3 and terminating the Focal T3 in the same router was too CPU intensive for the Cisco 7507. We have another 7507 on the way and will put the bridge and T3 on separate routers to reduce the CPU load. We returned traffic to the UUNet circuit at 300 B Street and will wait for the new equipment before making these changes permanent. The was no downtime during this operation.

Thu Jun 20 11:23:11 PDT 2002 — BroadLink is currently having technical difficulties with their head end aggregation router. They’re on site working to repair the problem now. The problem manifests as packet loss and high latency. The same trouble happened yesterday evening as well, and they’re working to implement a permanent solution. -Dane

Update: Broadlink’s network problem has been resolved.

Thu Jun 20 01:24:15 PDT 2002 — Sonic.net administration servers move. Tonight we are moving our internal administration servers into the new data center. This equipment doesn’t serve customers directly and will not cause any service interruption. We are also moving the office LAN onto the cross-site T3. This takes us off the enhanced DSL circuit we have been running on for the past two weeks. -Matt, Kelsey and Nathan

Tue Jun 18 11:22:13 PDT 2002 — More DSL maintenance: we have seen continued packet loss for a few DSL customers and are still working with ASI on a solution. At this time, SBC has optimized and restored our ATM DS3 connection and has escalated the issue to the DSL group at ASI. A small number of Santa Rosa customers may still be seeing intermittent packet loss and impaired performance; we are awaiting an ETR from ASI. -Aaron, Eli, Chris

Tue Jun 18 20:45:12 PDT 2002 — We’ve got a bridged T3 online between our B St and Apollo Way datacenters, and are preparing to move our staff and some test equipment tomorrow. Kudos to Kelsey and Nathan, who discovered that virtually every 12.x version of IOS has bugs in the IRB protocol implementation. After much troubleshooting, a version that worked was found and demonstrated, at 3am!

In addition tomorrow, we’ll be deploying a new Cisco router in San Francisco, at our new hub location there. It’ll be served by a T3 to Apollo, and will also connect to UUNet. The actual shift of UUNet traffic to this link will be done in the next few nights. We’ll post notice of this here, and it’ll be done in the middle of the night.

Our next major migration item is the move of Focal dialup numbers (XXX-9811) from B St to our San Francisco location. This is scheduled to occur at 3am on Friday morning the 21st. When this happens, connections will be dropped for customers calling these numbers, and they will need to re-dial.

We’ll keep you posted here as we work on the elements of this migration.

-Staff

Tue Jun 18 00:14:45 PDT 2002 — Our Cidera satellite news feed is offline due to a hardware failure in their server which translates the multicast satellite feed into a unicast NNTP feed to our NNTP hub. We’ve contacted their NOC and expect a replacement unit to be shipped in the morning. We anticipate that we will have degraded binary completion rates until the replacement server is online. -Kelsey and Matt

Mon Jun 17 09:01:34 PDT 2002 — There was a wide spread Internet outage this morning. We are still investigating why this happened and will post an update as soon as we have more info. It does appear to have cleared up and all is working well now. -Steve

Update: The trouble affected some T1 and T3 connected customers, plus dialup customers calling our remote POP numbers. Basically, customers and sites connected to our network via mega.sonic.net were impacted because Mega lost it’s ability to peer at a BGP level with it’s upstream routers.

The problem was caused by changes to the edge router structure in preparation for our moves of the coming weeks. Mega had an incorrect upstream route that depended upon one of the two edge routers for BGP peering adjacency. What this meant was that customers downstream from Mega could not communicate with the Internet, but were able to get to local systems without any trouble.

From the perspective of our internal monitoring systems which manage paging of operations staff in case of trouble, everything looked fine. This delayed our response, as from inside the network, everything registered as normal both toward the remote sites via mega, and toward the Internet itself.

As we complete our datacenter move over the next few weeks, we’ll exercise care and planning to assure that problems like this do not affect our customers, and that we’re monitoring well in case of any unexpected trouble. To this end, Eli is working with a remote partner to deploy end to end testing of equipment in our network, so that a problem like this won’t catch us unprepared.

The MOTD is likely to be pretty busy in the next few weeks, as we’ll be posting information about all moves and network changes here in advance, though many will be transparent to most customers. -Dane, Kelsey, Nathan, Scott, Eli

Mon Jun 17 20:34:18 PDT 2002 — Possible isolated DSL maintenance. We’ve noticed some unacceptable packet loss on one of the numerous DSLAMs in the Santa Rosa area, and SBC/ASI has just informed us that they may have to take the DSLAM down for maintenance in the next few hours. This affects a small portion of our Santa Rosa DSL customers, about 100 in all. ASI expects any outages to be short. — Eli, Aaron.