SSL server offline.

Tue Feb 4 08:38:06 PST 2003 — SSL server offline. The server which handles webmail and SSL services is having difficulties. We are looking into the problem and will keep you updated. -Matt

Update: Tue Feb 4 09:08:25 PST 2003 — SSL server back online. All webmail and SSL services are functioning again.

SSL Woes Continue: Our new server, which is…

Tue Feb 4 20:32:30 PST 2003 — SSL Woes Continue: Our new server, which is running a completely fresh installation of linux and a different kernel, is exhibiting stability problems and is currently down. This is the second time that the new box has locked up. We hope to have service restored in about 30 minutes. -Kelsey and Nathan

Update: It’s 23:32 and we’ve swapped SSL onto yet more new hardware. It has been stable for the past few hours under EXTREME load testing. We’ll continue to keep a close eye on it. -Kelsey

The SSL server has had a disk failure.

Tue Feb 4 14:18:19 PST 2003 — The SSL server has had a disk failure. We are making repairs and restoring data from our backups. Because of the nature of the problem it may take a couple of hours to repair it. Web mail and SSL services will remain unavailable while this operation is taking place. We apologize for the inconvenience and we will update this MOTD as soon as service is restored. -Operations

Update: SSL has been restored from tape onto a new server and has been up for a little more than an hour. We believe that we’ve got the server functioning properly. If you have ssl service, please review your site to make sure it is functioning properly. If you have any questions or comments, please contact support. It took us approximately 90 minutes to restore most SSL services from complete hardware failure. In the process, SSL also got a needed upgrade, it’s now a dual XEON with 2gb of RAM. -Kelsey and Nathan

Mail Server Backlog: Our mail servers are…

Tue Jan 28 12:13:22 PST 2003 — Mail Server Backlog: Our mail servers are currently experiencing a high load and are deep in queue. No mail has or will be lost, however, it may be delayed. We have taken some stop-gap measures to try to improve performance and it appears to be helping the situation. We expect to have it cleared up shortly.

We are in the process of testing a new mail server architecture that will resolve these periodic loading issues. We’ve got six blazing fast dual Xeon servers with plenty of RAM, local RAID, dual gigabit Ethernet NICs and two new shelves for our NetApp cluster currently testing the new architecture. The new architecture will include a new locally customized message store format that should provide better end-user performance while also putting less load on the back-end. There is a thread in news://news.sonic.net/sonic.general titled ‘Upcoming changes to mail handling…’ that details the changes and how they should improve the situation. -Kelsey

Update on the worm.

Sat Jan 25 11:06:01 PST 2003 — Update on the worm. Two of the five colocated customers who were disabled last night because they had been infected by the worm were brought back online at approximately 10:00AM this morning. Neither of these customers had properly secured their servers and they promptly began flooding 100mbits of outbound traffic again. Approximately 40 minutes later, Nathan and I had the ports locked down and Eli was en route to assist at the data center. While these customers were up again, reachability through our network was minimal. Technical support is experiencing long hold times and a high call volume, largely due to the affects of the worm. -Kelsey, Nathan and Eli.

More information on the Microsoft security…

Sat Jan 25 09:32:31 PST 2003 — More information on the Microsoft security problem that caused so many network issues across the Internet last night:

Internet Security Systems Security Brief January 25, 2003 Microsoft SQL Slammer Worm Propagation Synopsis: ISS X-Force has learned of a worm that is spreading via Microsoft SQL servers. The worm is responsible for large amounts of Internet traffic as well as millions of UDP/IP probes at the time of this alert’s publication. This worm attempts to exploit MS/SQL servers vulnerable to the SQL Server Resolution service buffer overflow (CVE CAN-2002-0649). Once a vulnerable computer is compromised, the worm will infect that target, randomly select a new target, and resend the exploit and propagation code to that host.

Impact:

Although the Slammer worm is not destructive to the infected host, it does generate a damaging level of network traffic when it scans for additional targets. A large amount of network traffic is created by the worm, which scans random IP addresses for vulnerable servers.

For the complete ISS X-Force Security Advisory, please visit: bvlive01.iss.net/issEn/delivery/xforce/alertdetail.jsp?oid=21824

Patching and disinfection information can be found at the URL above.

-Dane

Update on the worm #2.

Sat Jan 25 18:19:08 PST 2003 — Update on the worm #2. We have been filtering the spread of the worm on ingress and egress from our network for most of the day and appear to have stopped the spread of the infection inside of our network. All infected users have been contacted. Over the course of the day we have had a few brief outages related to the worm. None of them lasted for more than a few minutes.

This Event was entirely avoidable. The patches which fix the bug in the MS SQL service have been available for some time. The entire Sonic staff urges our users to keep their workstations and servers properly patched according to their vendor’s recommendations. In doing so, we can all help prevent something like this from happening again. -Kelsey