Thu Jun 20 11:23:11 PDT 2002 — BroadLink is currently having technical difficulties with their head end aggregation router. They’re on site working to repair the problem now. The problem manifests as packet loss and high latency. The same trouble happened yesterday evening as well, and they’re working to implement a permanent solution. -Dane
Update: Broadlink’s network problem has been resolved.
Thu Jun 20 01:24:15 PDT 2002 — Sonic.net administration servers move. Tonight we are moving our internal administration servers into the new data center. This equipment doesn’t serve customers directly and will not cause any service interruption. We are also moving the office LAN onto the cross-site T3. This takes us off the enhanced DSL circuit we have been running on for the past two weeks. -Matt, Kelsey and Nathan
Tue Jun 18 11:22:13 PDT 2002 — More DSL maintenance: we have seen continued packet loss for a few DSL customers and are still working with ASI on a solution. At this time, SBC has optimized and restored our ATM DS3 connection and has escalated the issue to the DSL group at ASI. A small number of Santa Rosa customers may still be seeing intermittent packet loss and impaired performance; we are awaiting an ETR from ASI. -Aaron, Eli, Chris
Tue Jun 18 20:45:12 PDT 2002 — We’ve got a bridged T3 online between our B St and Apollo Way datacenters, and are preparing to move our staff and some test equipment tomorrow. Kudos to Kelsey and Nathan, who discovered that virtually every 12.x version of IOS has bugs in the IRB protocol implementation. After much troubleshooting, a version that worked was found and demonstrated, at 3am!
In addition tomorrow, we’ll be deploying a new Cisco router in San Francisco, at our new hub location there. It’ll be served by a T3 to Apollo, and will also connect to UUNet. The actual shift of UUNet traffic to this link will be done in the next few nights. We’ll post notice of this here, and it’ll be done in the middle of the night.
Our next major migration item is the move of Focal dialup numbers (XXX-9811) from B St to our San Francisco location. This is scheduled to occur at 3am on Friday morning the 21st. When this happens, connections will be dropped for customers calling these numbers, and they will need to re-dial.
We’ll keep you posted here as we work on the elements of this migration.
-Staff
Tue Jun 18 00:14:45 PDT 2002 — Our Cidera satellite news feed is offline due to a hardware failure in their server which translates the multicast satellite feed into a unicast NNTP feed to our NNTP hub. We’ve contacted their NOC and expect a replacement unit to be shipped in the morning. We anticipate that we will have degraded binary completion rates until the replacement server is online. -Kelsey and Matt
Mon Jun 17 09:01:34 PDT 2002 — There was a wide spread Internet outage this morning. We are still investigating why this happened and will post an update as soon as we have more info. It does appear to have cleared up and all is working well now. -Steve
Update: The trouble affected some T1 and T3 connected customers, plus dialup customers calling our remote POP numbers. Basically, customers and sites connected to our network via mega.sonic.net were impacted because Mega lost it’s ability to peer at a BGP level with it’s upstream routers.
The problem was caused by changes to the edge router structure in preparation for our moves of the coming weeks. Mega had an incorrect upstream route that depended upon one of the two edge routers for BGP peering adjacency. What this meant was that customers downstream from Mega could not communicate with the Internet, but were able to get to local systems without any trouble.
From the perspective of our internal monitoring systems which manage paging of operations staff in case of trouble, everything looked fine. This delayed our response, as from inside the network, everything registered as normal both toward the remote sites via mega, and toward the Internet itself.
As we complete our datacenter move over the next few weeks, we’ll exercise care and planning to assure that problems like this do not affect our customers, and that we’re monitoring well in case of any unexpected trouble. To this end, Eli is working with a remote partner to deploy end to end testing of equipment in our network, so that a problem like this won’t catch us unprepared.
The MOTD is likely to be pretty busy in the next few weeks, as we’ll be posting information about all moves and network changes here in advance, though many will be transparent to most customers. -Dane, Kelsey, Nathan, Scott, Eli
Mon Jun 17 20:34:18 PDT 2002 — Possible isolated DSL maintenance. We’ve noticed some unacceptable packet loss on one of the numerous DSLAMs in the Santa Rosa area, and SBC/ASI has just informed us that they may have to take the DSLAM down for maintenance in the next few hours. This affects a small portion of our Santa Rosa DSL customers, about 100 in all. ASI expects any outages to be short. — Eli, Aaron.
Mon Jun 17 15:46:52 PDT 2002 — SpamAssassin has been upgraded to version 2.30. The new version includes a number of improvements including new scores and tests. All users may turn SpamAssassin on by using the SpamCan membertool. For more information please see news://news.sonic.net/sonic.antispam -Kelsey
Sat Jun 15 16:41:30 PDT 2002 — News server reboot. A few minutes ago, our reader news server stopped emitting banners for its services, causing attempts to access the news service to time out. A reboot of the server solved the problem. -Scott
Wed Jun 12 14:12:25 PDT 2002 — ICMP packets delayed. A problem with our Black Diamond switch is delaying ICMP packets to parts of the Sonic.net network. This does not impact the delivery of actual data. We are currently watching to see if the problem will go away on its own, or whether or not a reboot of the switch will be necessary. -Scott and Kelsey
Update: Inspection of Netflow statistics for one of our routers revealed that it was locked in a deadly embrace of continuous ICMP packets with the Black Diamond switch. Problem solved. -Nathan, Scott, Kelsey