Tue Nov 25 10:23:47 PST 2003 — SpamAssassin Changes: In order improve the overall quality and performance of our primary anti-spam filters we’ve disabled one of it’s features; The AWL, or auto-whitelist, keeps track of all addresses that mail has been received from along with statistics about that senders mail profile. This information is used to shift a given message’s score to the senders mean. This can be an effective tool in helping reduce false positives, but, as implemented currently, it has a number of flaws. The most significant flaw is run away growth which eats into users’ disk quotas; it never deletes records from the database.

One of the side effects of the ever-increasing AWL database size is increased server IO load. This morning, the IO load generated by the AWL on freezer, the NetApp filer that they are stored on, was approaching 90% of the filer’s total capacity. As freezer slows down under these high load conditions it cascades to high load on the SpamAssassin servers, mail servers and results in unfiltered mail.

We are working on a major SpamAssassin upgrade that we expect to launch around the New Year. Hopefully, if we feel that the AWL is still a valuable feature of SpamAssassin, we’ll be able to improve and re-enable it at this time. -Kelsey

Sun Nov 23 19:24:00 PST 2003 — Limited dialup service interruption. Failure of a network access server caused temporary interruption of dialup service to approximately 1% of Sonic.net dialup customers. The server has been re-booted and service has been restored. The server will be replaced as soon as possible as a precaution. – Kelsey, John and Russ

Wed Nov 19 13:05:34 PST 2003 — Brief outbound mail delivery interruption. Our outbound mail servers required a reboot. During this time customers attempting to send mail will have had their connection to the mail server refused. The outage lasted only a few minutes; normal service has been restored. – Operations

Tue Nov 18 18:08:28 PST 2003 — We are having problems with our Voice trunk provider, which is causing a number of dropped calls to our Tech Support lines. Focal is working with SBC to determine the source of the problem, and we do not have an estimated time of repair. If you have any problems reaching Support, please try your call again – We also have Online Support available, and plenty of reps standing by – www.sonic.net/livehelp — Eli, Jared, Support Staff

Mon Nov 17 10:26:47 PST 2003 — There are still some Broadlink customer outages. Broadlink service is partially restored, and the Broadlink staff are working hard to get service restored everywhere. -Operations.

Update: Mon Nov 17 12:26:02 PST 2003 — Most Broadlink issues have been resolved. There is still a small isolated network with problems. More info will be posted later. If you are still having problems with your Broadlink service, please call tech support. -Operations

Update: Mon Nov 17 13:55:08 PST 2003 — Broadlink is now showing all customers up and running. -Operations

Sun Nov 16 09:07:25 PST 2003 — Some Broadlink problems persist. Some lingering problems persist on Broadlink’s backhaul link. This has been traced to problems with the link radios’ firmware. Broadlink has been working with the vendor, with no resolution yet — in the interim, Broadlink will be replacing the backhaul link radios with those made by another vendor. They estimate the new link will be up by noon today. -Scott, and Tim from Broadlink

Update: Sun Nov 16 14:02:50 PST 2003 — Broadlink problems persist. Broadlink is currently on premises working to realign the new wireless backhaul equipment. ETR one hour. -Kavan, Matt, and Tim from Broadlink

Update: Mon Nov 17 00:48:48 PST 2003 — Broadlink problems fixed. After many hours of troubleshooting, the issues have been resolved. Many thanks to Tim, Jason, Scott and Chris for their long hours and good work. Customers having issues at this time should reboot their CPE device. If problems persist, please call technical support. -Matt, Tim, Jason and Scott W.

Fri Nov 14 14:31:14 PST 2003 — Broadlink site came up…then went back down. Broadlink had solved the problem with their backhaul link, but then lost a switch at the facility. Broadlink is working hard to restore operation of the site. -Scott, and Linda from Broadlink

Update: ETR is approximately 3:30pm. -Scott and Linda

Update: Fri Nov 14 16:43:47 PST 2003 The repair team is still on site, we will update the MOTD as soon as we know more. -Scott and Linda

Update: Fri Nov 14 20:02:44 PST 2003 The backhaul link and supporting equipment are back online. Jason and Tim have worked hard over the past 24 hours to find and fix the various problems that caused the outages. Everyone is keeping a close eye on the network and things have been stable for the past couple of hours. -Matt, Tim and Jason

Update: Sat Nov 15 18:57:33 PST 2003 Broadlink Wireless links are back up. Some network recovery issues are preventing a small number of customers from coming back on line. Broadlink is continuing to work on these individual problems. Currently we have no ETR. — Scott, Tim, Kavan, and Jason

Thu Nov 13 20:44:37 PST 2003 — Backhaul link between Sonic’s fiber facilities and one of BroadLink’s primary wireless POPs has gone down. Customers served from most of BroadLink’s network in central, west, and north east Santa Rosa are affected. J. Kane is presently working on resolving at site. Still evaluating for ETA service restoration. – Tim

Tue Nov 11 16:19:49 PST 2003 — Changes to Anti-Spam Services: We’ve discontinued two old services that used to be part of our spam countermeasures. First, users used to be able to send spam to spam@sonic.net which would remail the spam examples into the sonic.spamcan news group for review by other users and staff. As of yesterday, we are no longer supporting this; it’s hasn’t proved useful for us to continue to collect spam in this fashion. Our thanks go out to everyone that has helped us in the past by bouncing their spam to this address. Second, a reminder that no-spam@sonic.net, which used to provide a friendly auto-response explaining that our customer did not want to receive spam, no longer does so. Mail to no-spam@sonic.net bounces with a standard ‘user unknown’ DSN. -Kelsey

Wed Nov 5 10:48:55 PST 2003 — System username database propagation problem. We just corrected a problem with the propagation of username info to Sonic.net servers. This means username adds, deletes, and changes just started working — they had stopped working at about 11:30am yesterday. We apologize for the inconvenience, and will make it better: we are adding some (more) intelligence to the propagation process to notify us of this particular problem. -Scott