Month: March 2001

Wed Mar 7 18:01:23 PST 2001 — We identified the causes of the poor inbound performance and routing instability on our network and resolved them at 5:00pm. We had been aware of the performance problems since this morning and had been working hard to try to identify the source. As it turns out, there was trouble with two of our routers internal ethernet interfaces which, once identified, was easy to fix. We also ordered an additional 3Mbps for UUNet today which we expect to turn up tomorrow morning. This is in addition to the the 3Mbps that we brought online this morning and brings our UUNet T3 to approximately 24Mbps. -Kelsey, Eli, Scott and Russ.

Wed Mar 7 13:34:41 PST 2001 — We have successfully upgraded our UUNet DS3 circuit to the next access class, in response to a need for more bandwidth that we identified last week. It took longer than expected to work with UUNet to get this upgrade accomplished, but we were able to make the change today with no negative impact whatsoever on the network, and utilization stats on our circuits now show optimal performance with no bottlenecks. — Eli, Kelsey

Tue Mar 6 10:08:01 PST 2001 — Broadlink Maintenance. Broadlink will be performing some maintenance at two of their tower sites. The Barham tower which serves Rincon Valley will be down on Wednesday, 3/07 from 11:59 pm to 12:15 am. The Wmoore tower which servers Southeast Santa Rosa will be down from 12:30 am to 12:45 am. If you experience any service outages outside of this maintenance window please let us know. -Broadlink

Mon Mar 5 13:35:21 PST 2001 — Router coup. As part of our move to a redundant configuration, a simple operation turned into a serious problem for about 3 minutes. Specifically, one of the USR Total Control dialup routers took control of our internal routing protocol, even though it was configured not to. The problem was detected within about 2 minutes and corrected within about a minute. Even more specifically, two routers with OSPF priorities of “127” and “96” were couped by a router with a priority of “5”. We’re still trying to figure out how that could have happened. -Scott and Dane

Sun Mar 4 10:45:51 PST 2001 — Night Operations. Sunday morning at 12:30 am Sonic.net performed a number of maintenance upgrades designed to increase NetApp filer, news server and core switching performance as well as border router redundancy. All upgrades were preventative and further increase the redundancy and responsiveness of Sonic.net.

The news server, Typhoon, underwent a performance upgrade — unfortunately, the license key provided by the vendor did not work for our new configuration.

An upgrade of our core switch OS caused a “network freeze” that lasted about 90 seconds while the switch rebooted. The new OS has bug and performance fixes.

We commissioned into service another Cisco 7200 router as part of our ongoing efforts toward more redundant Internet connectivity. Sonic.net now uses two Cisco 7200 edge routers — “gamma” and “delta” — in addition to the Cisco 7507 that used to be our edge router, “mega”. Each edge router handles a T3 to one of the two largest Internet backbones: UUNet and Cable & Wireless. While we had reported that Internet connectivity would be lost for three minutes during this upgrade, actual impact was less than a minute.

The NetApp NFS filer underwent an OS upgrade and disk firmware upgrade, disturbing mail and web storage for about an hour.

A number of Port Masters servicing 707-522-1002 and some Oakland numbers were rebooted. There was no noticeable down time, however, dial-up connections on the 1002 equipment were terminated. -Matt, Kelsey, Scott, Steve, Russ, Scott R., and Jeff

Fri Mar 3 17:32:30 PST 2001 — Night Operations: Sunday morning at 12:30 am Sonic.net will be performing a number of maintenance upgrades designed to increase NetApp filer, news server and core switching performance as well as border router redundancy. All upgrades are preventative and will further increase the redundancy and responsiveness of Sonic.net.

The news server Typhoon will undergo a performance upgrade during which time news.sonic.net will be offline. The outage is expected to last approximately 30 minutes.

An upgrade of our core switch OS will cause a network freeze that will last about 90 seconds while the switch reboots. The new OS has bug and performance fixes.

We will be bringing up yet another Cisco 7200 router as part of our ongoing efforts toward more redundant Internet connectivity. This will leave us with two edge routers, each handling a T3 to a major Internet backbone. This change will cause loss of Internet connectivity for about three minutes.

The NetApp NFS filer will undergo an OS upgrade and disk firmware upgrade. The process should take less than 20 minutes.

A number of Port Masters servicing 707-522-1002 and some Oakland numbers will be rebooted. There will not be any noticeable down time, however, dial-up connections on the equipment will terminated. While systems are offline, some re-cabling will take place to tidy up the rapidly expanding colocation areas. -Matt, Kelsey, Scott, Steve and Russ

Fri Mar 2 12:09:03 PST 2001 — Sonic.net has completed its California rollout! New equipment is online to provide dial-up access to the remainder of California. This means that local dial-up is available almost anywhere in the state. Additional dial-up numbers have been added to the POP Finder located at www.sonic.net/cgi-bin/pops.pl

Over the next few days, Sonic.net welcomes customers to participate in the public beta testing of these numbers. Remember that Sonic.net is not responsible for long distance telephone charges. Check with your operator to make sure that any number you dial is a local one. -Matt and the Sonic.net team

Thu Mar 1 10:46:35 PST 2001 — We replaced thunder, one of the load balanced web servers with new, much faster, hardware, effectively doubling the capacity of our web server group. The primary effect of this upgrade is increased responsiveness and faster cgi execution on all of our hosted websites. We also updated and patched the latest version of sendmail, our SMTP server software, to enable connection limiting on a per peer basis. We added this feature in response to one of the DOS attacks that happened in mid February to prevent a single rogue server from consuming all of our mailservers’ resources. We’ve submitted this patch to the sendmail maintainers so other sendmail users will be able to benefit from our work. -Kelsey, Nathan & Russ