Author: admin

Tue Jun 18 11:22:13 PDT 2002 — More DSL maintenance: we have seen continued packet loss for a few DSL customers and are still working with ASI on a solution. At this time, SBC has optimized and restored our ATM DS3 connection and has escalated the issue to the DSL group at ASI. A small number of Santa Rosa customers may still be seeing intermittent packet loss and impaired performance; we are awaiting an ETR from ASI. -Aaron, Eli, Chris

Tue Jun 18 20:45:12 PDT 2002 — We’ve got a bridged T3 online between our B St and Apollo Way datacenters, and are preparing to move our staff and some test equipment tomorrow. Kudos to Kelsey and Nathan, who discovered that virtually every 12.x version of IOS has bugs in the IRB protocol implementation. After much troubleshooting, a version that worked was found and demonstrated, at 3am!

In addition tomorrow, we’ll be deploying a new Cisco router in San Francisco, at our new hub location there. It’ll be served by a T3 to Apollo, and will also connect to UUNet. The actual shift of UUNet traffic to this link will be done in the next few nights. We’ll post notice of this here, and it’ll be done in the middle of the night.

Our next major migration item is the move of Focal dialup numbers (XXX-9811) from B St to our San Francisco location. This is scheduled to occur at 3am on Friday morning the 21st. When this happens, connections will be dropped for customers calling these numbers, and they will need to re-dial.

We’ll keep you posted here as we work on the elements of this migration.

-Staff

Tue Jun 18 00:14:45 PDT 2002 — Our Cidera satellite news feed is offline due to a hardware failure in their server which translates the multicast satellite feed into a unicast NNTP feed to our NNTP hub. We’ve contacted their NOC and expect a replacement unit to be shipped in the morning. We anticipate that we will have degraded binary completion rates until the replacement server is online. -Kelsey and Matt

Mon Jun 17 09:01:34 PDT 2002 — There was a wide spread Internet outage this morning. We are still investigating why this happened and will post an update as soon as we have more info. It does appear to have cleared up and all is working well now. -Steve

Update: The trouble affected some T1 and T3 connected customers, plus dialup customers calling our remote POP numbers. Basically, customers and sites connected to our network via mega.sonic.net were impacted because Mega lost it’s ability to peer at a BGP level with it’s upstream routers.

The problem was caused by changes to the edge router structure in preparation for our moves of the coming weeks. Mega had an incorrect upstream route that depended upon one of the two edge routers for BGP peering adjacency. What this meant was that customers downstream from Mega could not communicate with the Internet, but were able to get to local systems without any trouble.

From the perspective of our internal monitoring systems which manage paging of operations staff in case of trouble, everything looked fine. This delayed our response, as from inside the network, everything registered as normal both toward the remote sites via mega, and toward the Internet itself.

As we complete our datacenter move over the next few weeks, we’ll exercise care and planning to assure that problems like this do not affect our customers, and that we’re monitoring well in case of any unexpected trouble. To this end, Eli is working with a remote partner to deploy end to end testing of equipment in our network, so that a problem like this won’t catch us unprepared.

The MOTD is likely to be pretty busy in the next few weeks, as we’ll be posting information about all moves and network changes here in advance, though many will be transparent to most customers. -Dane, Kelsey, Nathan, Scott, Eli

Mon Jun 17 20:34:18 PDT 2002 — Possible isolated DSL maintenance. We’ve noticed some unacceptable packet loss on one of the numerous DSLAMs in the Santa Rosa area, and SBC/ASI has just informed us that they may have to take the DSLAM down for maintenance in the next few hours. This affects a small portion of our Santa Rosa DSL customers, about 100 in all. ASI expects any outages to be short. — Eli, Aaron.

Mon Jun 17 15:46:52 PDT 2002 — SpamAssassin has been upgraded to version 2.30. The new version includes a number of improvements including new scores and tests. All users may turn SpamAssassin on by using the SpamCan membertool. For more information please see news://news.sonic.net/sonic.antispam -Kelsey

Sat Jun 15 16:41:30 PDT 2002 — News server reboot. A few minutes ago, our reader news server stopped emitting banners for its services, causing attempts to access the news service to time out. A reboot of the server solved the problem. -Scott

Wed Jun 12 14:12:25 PDT 2002 — ICMP packets delayed. A problem with our Black Diamond switch is delaying ICMP packets to parts of the Sonic.net network. This does not impact the delivery of actual data. We are currently watching to see if the problem will go away on its own, or whether or not a reboot of the switch will be necessary. -Scott and Kelsey

Update: Inspection of Netflow statistics for one of our routers revealed that it was locked in a deadly embrace of continuous ICMP packets with the Black Diamond switch. Problem solved. -Nathan, Scott, Kelsey

Tue Jun 11 09:35:27 PDT 2002 — Mail Server Trouble: Starting at about 3:00PM yesterday we observed and sharp increase in the number of POP3 sessions that were ‘lost’ by our servers. After many hours of troubleshooting last night we believed the problem had cleared itself up. However, this was incorrect- we are still losing about 5% of our POP3 connections for no apparent reason. If you experience any difficulty receiving mail please be patient and try again. We hope to have this resolved shortly -Kelsey and Nathan

UPDATE: We found a faulty network connection which we overlooked last night and have resolved the problems with dropped POP3 sessions.

Sun Jun 9 23:33:48 PDT 2002 — First diesel generator test run. In a simulated PG&E outage, the Leibert UPS and Detroit Diesel generator at our new 5000 sq. ft. datacenter facility delivered smooth and uninterrupted power. We ran on diesel for a 10 minute test interval, then switched back to utility power. The automatic startup, transfer, return to utility and shutdown of the generator were all handled smoothly by our transfer switch, UPS system and generator.

The generator is a 23.9 liter v-12 twin turbocharged Detroit Diesel, which produces 1024 horsepower and 750 kilowatts of power. We can effectively run forever, just by topping up the fuel tanks as they are used. This assures availability of your server, website or Internet access, regardless of summer black-outs or emergencies. -Dane