Wed Aug 15 13:44:32 PDT 2001 — Update: SBC-ASI testers are inspecting the circuit, no ETR yet. This outage affects Pacific Bell ADSL customers, and several of our T1 customers that cross connect via our ATM. -Kelsey, Nathan, Eli, Russ & Steve

Tue Aug 14 18:16:07 PDT 2001 — We will be performing upgrades to our internal MySQL server at 9:30pm tonight. Member tools will be offline for the duration of the work which should be completed within 15 minutes. -Kelsey & Nathan

Fri Aug 10 16:05:22 PDT 2001 — Marine.sonic.net is down due to a flood of outbound spam sent by a malicious user. We are working to correct the issue, until then outgoing mail delivery may be delayed. No mail will be lost. – Russ, Scott, Nathan and Steve

Fri Aug 10 00:38:07 PDT 2001 — Some minor maintenance to the Broadlink ATM circuit required approximately 15 minutes of downtime, beginning at 12:30 AM. -Kevan and Steve

Thu Aug 9 10:47:14 PDT 2001 — CodeRed filters have been added to our remote pops in Napa, Sebastopol and pops covered by the PacWest group. PacWest pops are numbers that end with 0174. -Steve

Tue Aug 7 09:46:42 PDT 2001 — BroadLink reports that the tower site in Wikiup that serves Northwest Santa Rosa had problems this AM, and that they have resolved the problem. The outage started at 8:23AM and the trouble was resolved at 9:13AM.

In other BroadLink news, John Navas has written an overview of wireless broadband technologies for TechTV. You can read it at:

www.techtv.com/print/story/0,23102,3339884,00.html

Tue Aug 7 02:19:48 PDT 2001 — Night Operations Complete: We’ve completed the upgrades to both spiff, the MySQL server, and the RedBack. Both upgrades appear to have been successful. -Kelsey and Nathan

Mon Aug 6 17:58:23 PDT 2001 — Night Operations. Tonight at midnight we will begin a migration of our internal MySQL to new hardware to resolve some ongoing stability issues. The new MySQL server is configured to use our redundant NetApp filers for data storage instead of a local RAID. Our tests show that this setup performs remarkably well while providing increased redundancy. During the upgrade the member tools will be unavailable.

We will also be rebooting the RedBack SMS onto the latest General Deployment release. RedBack assures us that this release has resolved the spontaneous reboot issues that plagued the last GD release. The RedBack only takes a few minutes to reload; PacBell ADSL and Broadlink WDSL customers will be down briefly. -Kelsey, Nathan and Eli.

Sun Aug 5 18:09:00 PDT 2001 — Dial-group issues. From about 17:30 -> 18:00 some customers were receiving an ‘all circuits busy’ message when calling 522-1002 or 522-1003. I rebooted the last HiperArc card in Nas22, and all is well. — Eli

Wed Aug 1 11:22:48 PDT 2001 — Our own web servers, which are not vulnerable to Code Red, are currently logging around 40 attempted Code Red attacks per minute. Our servers host about 2500 unique IP addresses, so we could guess that each IP address is being attacked about once per hour currently. The ramp-up has been slow, but it’s getting faster and faster.

We’ve searched for vulnerable systems connected to our networks, and have informed customers when we’ve found problems. We also currently have inbound port 80 blocking (web) in place for dynamic dialup pools. This will not affect normal use of your dialup connection, but protects you from inbound worm attacks in case you’ve got a vulnerable Windows NT or 2000 system. -Dane, Eli, Steve, Russ, Scott and Kelsey