Author: admin

Sat Jan 25 11:06:01 PST 2003 — Update on the worm. Two of the five colocated customers who were disabled last night because they had been infected by the worm were brought back online at approximately 10:00AM this morning. Neither of these customers had properly secured their servers and they promptly began flooding 100mbits of outbound traffic again. Approximately 40 minutes later, Nathan and I had the ports locked down and Eli was en route to assist at the data center. While these customers were up again, reachability through our network was minimal. Technical support is experiencing long hold times and a high call volume, largely due to the affects of the worm. -Kelsey, Nathan and Eli.

Sat Jan 25 09:32:31 PST 2003 — More information on the Microsoft security problem that caused so many network issues across the Internet last night:

Internet Security Systems Security Brief January 25, 2003 Microsoft SQL Slammer Worm Propagation Synopsis: ISS X-Force has learned of a worm that is spreading via Microsoft SQL servers. The worm is responsible for large amounts of Internet traffic as well as millions of UDP/IP probes at the time of this alert’s publication. This worm attempts to exploit MS/SQL servers vulnerable to the SQL Server Resolution service buffer overflow (CVE CAN-2002-0649). Once a vulnerable computer is compromised, the worm will infect that target, randomly select a new target, and resend the exploit and propagation code to that host.

Impact:

Although the Slammer worm is not destructive to the infected host, it does generate a damaging level of network traffic when it scans for additional targets. A large amount of network traffic is created by the worm, which scans random IP addresses for vulnerable servers.

For the complete ISS X-Force Security Advisory, please visit: bvlive01.iss.net/issEn/delivery/xforce/alertdetail.jsp?oid=21824

Patching and disinfection information can be found at the URL above.

-Dane

Sat Jan 25 18:19:08 PST 2003 — Update on the worm #2. We have been filtering the spread of the worm on ingress and egress from our network for most of the day and appear to have stopped the spread of the infection inside of our network. All infected users have been contacted. Over the course of the day we have had a few brief outages related to the worm. None of them lasted for more than a few minutes.

This Event was entirely avoidable. The patches which fix the bug in the MS SQL service have been available for some time. The entire Sonic staff urges our users to keep their workstations and servers properly patched according to their vendor’s recommendations. In doing so, we can all help prevent something like this from happening again. -Kelsey

Sat Jan 25 03:04:47 PST 2003 — Night Operations Complete: Both the SMS and NetApp maintenance proceeded as planned. It’s been a wild night. -Kelsey and Nathan

Sat Jan 25 00:09:11 PST 2003 — The Internet has broken. We do not yet know the source of the DoS but we believe that it may be another worm that attacks Microsoft servers. At this time, large portions of the Internet are still unreachable. There have been widespread reports that this DoS also took the lives of many Cisco routers, including one of our 7500 border routers. Our internal network is functioning at 100% but many sites have yet to restore service and I would anticipate, may not return for some time.

A very interesting graph of this event and it’s effect on the Internet as a whole is available at the following link: average.matrix.net/ -Kelsey and Nathan

Fri Jan 24 10:47:59 PST 2003 — A configuration error (operator error) on our part caused a problem for customers that have us listed as a secondary mail exchanger for their domains. This is a very small percentage of our customers, and the symptom manifested itself as bounced mail only when the customers’ primary mail server was unavailable. _I_ apologize for the misconfiguration. — Eli

Fri Jan 24 22:29:43 PST 2003 — High latency and packet loss. We are currently experiencing high latency and packet loss in portions of our network, including part of our core. We are investigating the situation, and will hopefully have things back to normal soon. Kelsey is on-site. -Scott, Kelsey, and Nathan

Update Fri Jan 24 23:11:01 PST 2003… This turns out to be a massive worm which is causing denial of service (DoS) across the Internet. UUNet has characterized this as “the DoS of the year”. The vector is Microsoft SQL servers. So far, we have found 7 servers pumping 100 megabit/second into our core, which is the cause of the high latency and packet loss within portions of our network. We continue to work the problem. -Scott, Kelsey, and Nathan

Fri Jan 24 08:58:07 PST 2003 — Our Second POP in Santa Rosa is currently experiencing packet loss and slow performance due to some problems which are beyond our control. We’ve taken what steps we can to improve the situation here but it’s out of our hands. We anticipate that the other parties involved will have he situation repaired shortly. In the meantime, if you are experience poor performance on your dial-up connection, we suggest that you change to an alternate number for your area which can be found using the pop finder at www.sonic.net/cgi-bin/pops.pl -Kelsey and Nathan

Update: This issue has been resolved.

Fri Jan 24 18:43:54 PST 2003 — Night Operations: Saturday starting at 1:30AM (tonight) we are going to upgrade the OS and one of the interface cards on the SMS 1800. We anticipate minimal downtime of less than 5 minutes while the cards are swapped and the box reloads on the new OS. However, if we run into difficulty it could take a bit longer. While we are working on the SMS, all SBC ADSL and FRATM customers will be offline. We’ll also be changing a few configuration options on our NetApps that may cause momentary un-reachability as we reconfigure their gigabit ethernet interfaces. -Kelsey and Nathan.

Tue Jan 21 18:06:06 PST 2003 — Earlier today at approximately 2:00PM some of Broadlink’s head end gear locked up and stopped responding. They were able to restore service to customers within a few minutes. -Sonic Ops