Author: admin

Tue Aug 7 02:19:48 PDT 2001 — Night Operations Complete: We’ve completed the upgrades to both spiff, the MySQL server, and the RedBack. Both upgrades appear to have been successful. -Kelsey and Nathan

Mon Aug 6 17:58:23 PDT 2001 — Night Operations. Tonight at midnight we will begin a migration of our internal MySQL to new hardware to resolve some ongoing stability issues. The new MySQL server is configured to use our redundant NetApp filers for data storage instead of a local RAID. Our tests show that this setup performs remarkably well while providing increased redundancy. During the upgrade the member tools will be unavailable.

We will also be rebooting the RedBack SMS onto the latest General Deployment release. RedBack assures us that this release has resolved the spontaneous reboot issues that plagued the last GD release. The RedBack only takes a few minutes to reload; PacBell ADSL and Broadlink WDSL customers will be down briefly. -Kelsey, Nathan and Eli.

Sun Aug 5 18:09:00 PDT 2001 — Dial-group issues. From about 17:30 -> 18:00 some customers were receiving an ‘all circuits busy’ message when calling 522-1002 or 522-1003. I rebooted the last HiperArc card in Nas22, and all is well. — Eli

Wed Aug 1 11:22:48 PDT 2001 — Our own web servers, which are not vulnerable to Code Red, are currently logging around 40 attempted Code Red attacks per minute. Our servers host about 2500 unique IP addresses, so we could guess that each IP address is being attacked about once per hour currently. The ramp-up has been slow, but it’s getting faster and faster.

We’ve searched for vulnerable systems connected to our networks, and have informed customers when we’ve found problems. We also currently have inbound port 80 blocking (web) in place for dynamic dialup pools. This will not affect normal use of your dialup connection, but protects you from inbound worm attacks in case you’ve got a vulnerable Windows NT or 2000 system. -Dane, Eli, Steve, Russ, Scott and Kelsey

Wed Aug 1 10:05:03 PDT 2001 — A Code Red firewall filter installed on pm3 caused some people who dialed into 707-522-1001 to lose connectivity this morning. We removed the filter and the connectivity came back. Sonic.net has invested in multiple dialup numbers in each area we cover, so that there is always a number that will work for you. It is always a good idea to keep a current print out of backup numbers for your area when problems may arise. You can get a current list of dialup numbers for your area by using our pop finder tool which is located at www.sonic.net/cgi-bin/pops.pl -Steve

Mon Jul 30 12:49:44 PDT 2001 — More Code Red work. We’ll be running more scans on our networks, and also implementing some firewalling on the dial-up equipment (Not DSL or Dedicated networks) on port 80. – Eli, Kelsey, Steve, ScottR

Sat Jul 28 15:29:21 PDT 2001 — The SMS crashed again at 12:27PM today. It reloaded properly on the older, more stable code. The code that it is currently running does have some known issues but they are far less significant than the issues that we are having with the latest GD release. We will continue to push RedBack to provide better code for us and to improve their software QA. RedBack should have new code some time next week. -Kelsey

Sat Jul 28 01:05:55 PDT 2001 — The SMS crashed again, twice, in rapid succession. It didn’t boot up on the old code as I had hoped, but I’ve confirmed that it will in the event that it reloads itself again. -Kelsey

Fri Jul 27 16:24:53 PDT 2001 — We had an unexpected crash on our RedBack router. This caused some connectivity problems for DSL and Broadlink WDSL customers. We are investigating into why this has happened. -Eli, Matt, Dave, Kelsey, Steve and Russ UPDATE: RedBack reports that this is a known issue and that it should be fixed in the next stable release due out next week. In the meantime we have reconfigured the RedBack SMS to use an older version of code that we’ve already demonstrated to be more stable than the current code running on it. In the event that it fails again it will boot the stable code. -Kelsey and Matt

Fri Jul 27 15:08:16 PDT 2001 — New SPAM Filter: We implemented a new SPAM countermeasure which may result in 30% reduction in inbound SPAM. For more information about this new filter and how to enable it for your email account, please see my post in news:sonic.general entitled ‘new spam filters’. -Kelsey, Russ, Matt and Scott.