Month: September 2001

We have implemented a new email anti-SPAM…

Fri Sep 28 15:08:10 PDT 2001 — We have implemented a new email anti-SPAM system and additional features to replace the rule-checking we were previously using.

The new features allow us to accept mail from our users who are accessing the net from other providers and, in the near future, to to provide SPAM blocking tailored to individual preferences.

The “POP before SMTP” (or “check before send”) feature allows sonic customers who are traveling or connecting from another ISP to send mail through Sonic.net after checking for new mail. Just check mail (at pop.sonic.net) using your Sonic.net account prior to sending mail (at mail.sonic.net).

The SPAM blocking system will allow users to select the level of SPAM blocking they desire, ranging from none to very strict blocking. Currently Sonic.net provides a moderate level blocking required to protect our customers, but individual control will give you much more flexibility.

Users who wish to opt out of the blocking entirely may do so by emailing support@sonic.net.

An interface will be provided soon to allow for user selectable levels of blocking, strict blocking, content blocking white-lists and black-lists and full opt-out. -Russ

Night Operations: Tonight starting at…

Tue Sep 25 16:35:14 PDT 2001 — Night Operations: Tonight starting at midnight we will upgrading the OS of piggy, one of our load balancing switches. This upgrade will allow us to do URL content filtering on the Alteon’s providing us with tools to combat worms like CodeRed and Nimba. The upgrade is expected to take no more than 3 minutes and customers may not notice any downtime. -Matt, Steve, Dane and Kelsey

UPDATE: The upgrade is complete. Only load balanced services such as multihomed web sites, mail and pop were affected. No customers lost connectivity at any time.

Our Domain Name Registration Tool will be…

Mon Sep 24 11:15:08 PDT 2001 — Our Domain Name Registration Tool will be taken offline for the following registry imposed maintenance window:

Date: Sunday, September 30, 2001 (GMT) Saturday, September 29, 2001 (EDT)

Time: 0100 hrs – 0500 hrs GMT 2100 hrs – 0100 hrs EDT

This maintenance window has been set by the gTLD registry operator, and will affect all ICANN accredited registrars equally.

We apologize for any inconvenience this may cause you. -Chuckc

We are currently beta testing our new email…

Fri Sep 21 15:32:17 PDT 2001 — We are currently beta testing our new email filtering software. The new software enables our customers to opt in or out of each of our anti SPAM and virus filters. It will also support customer maintained white and blacklists and stronger content based filters to block adult email. Initially customers will only be able to turn off all email filtering but we hope to have finer controls enabled soon. The new software also enables POP before SMTP relay authentication which allows customers from off of our network to send mail through our outbound mail servers. If you have any questions please post them to news:sonic.net Ultra, one of the four load balanced mail servers, is running the new software now. Unfortunately, due to a small typo in the default SPAM filtering options, we applied ORBS DNS based black-listing to all mail that was received by ultra between 1:30pm and 2:40pm. We deeply regret this error due to the controversial nature of the ORBx lists and do not ever intend to enable any of them by default. -Russ, Kelsey and Dane

Over the course of the next few days, we are…

Thu Sep 20 11:33:43 PDT 2001 — Over the course of the next few days, we are going to be pro-actively looking for customer machines that have been affected by the Nimda worm. This will take the form of several port scans per day, and we will be contacting infected customers. — Eli, ScottR

Timber and its services have been restored.

Thu Sep 20 01:03:47 PDT 2001 — Timber and its services have been restored. We were able to restore about half of the customer databases from a backup taken shortly before the RAID failed and were able to restore the rest from tape. However, updates that have been made to a database over the past few days may have been lost and may need to be reentered. The MySQL databases are now stored on the NetApps and MySQL has been upgraded to the latest stable version. This configuration has been working very well for us on our internal MySQL server. MySQL customers will receive an email detailing the events and the steps that we’re taking to prevent the same kind of failures from happening again.

We’ve also been able to restore all of the Urchin webstats from tape. The only data that was totally lost was our archived raw weblogs. We cannot currently provide access to the raw weblogs but should have them back online either later today, Thursday, or Friday, on the new log processing box which is nearly ready for deployment. -Kelsey

Timber, our public MySQL server and weblog…

Wed Sep 19 18:42:35 PDT 2001 — Timber, our public MySQL server and weblog processing box has experienced a catastrophic RAID failure. We had scheduled maintenance tonight to replace the degraded array with an NFS export on the NetApps but, the RAID decided to fail altogether before it was obsolete. We are working to bring the server back online now. Once the server is back online we will be able to assess the damage to the MySQL databases which we hope can be restored and migrated to the NetApps as planned. At this moment we are not sure how long it will take to restore public SQL services. – Kelsey, Russ, Scott and Matt

We’re having some network issues which are…

Tue Sep 18 09:15:36 PDT 2001 — We’re having some network issues which are intermittently affecting mail and web services right now. We’re working to isolate and resolve the trouble.

This appears to be related to a new worm infecting Microsoft systems; we’re rejecting thousands of attempts from our servers, which seems to be the result this potential worm.

Update: Our load balancing systems are being impacted by what appears to be the new W32.Nimda.A@mm virus/worm. For more information, see:

www.sarc.com/avcenter/venc/data/w32.nimda.a@mm.html

We’re currently attempting to find a way to deflect the tens of thousands of worm connections that we’re getting to keep our servers and load balancing systems stable.

Update: A number of infected customers systems are spreading this virus actively, and we’ve taken some customers offline. Currently, Covad customers are down, plus a few PacBell DSL customers.

Update: Covad customers as a group were down for about five minutes while we regained control of our router. Currently, we’ve got a customer or two on wireless, DSL, T1 or Covad offline due to infection of their systems. We’re working with these customers to clean up their systems.

All Sonic.net services, including mail, web, news and Internet access are normal at this time, but we are seeing some traffic load due to this worm/virus.

Update: We’re seeing ongoing intermittent performance of outgoing email servers at this time. We’re working to stabilize the systems under the heavy load. If you fail to send mail, wait a moment and try again. -Dane, Matt, Steve, Kelsey, Scott and Eli