Fri Nov 10 14:24:55 PST 2006 — SpamAssassin Upgrades and Changes. Over the last couple of weeks we began testing a new SpamAssassin module that uses OCR (Optical Character Recognition) to scan attached images for known spam phrases resulting in a silent deployment last week. So far it, along with another 3rd party rule and score set, has proven to be very affective at catching most of the spam that was leaking through with an insignificant number of false positive matches. Also, in response to a user’s request, the ‘Drop Blacklisted Messages’ feature now also will drop messages matching Subject and ‘User in Blacklist To’ lists. -Kelsey

Wed Nov 8 03:18:10 PST 2006 — Mail Service Interruption. For one hour and thirty four minutes this morning inbound mail delivery and retrieval was unavailable do to a failure on one of our four backend message stores. The exact cause of the failure is still unknown and now that service is restored we’re continuing to investigate the situation. This is the first time we’ve ever experienced a failure like this in the 10+ years that we have been using Network Appliance Filers. At this time all mail queued in our systems is delivering to customers mail spools. No mail was lost or not accepted for delivery for any reason related to this event by our MX servers during this outage. -Kelsey

Fri Oct 27 14:13:25 PDT 2006 — Focal POP maintenance. On Tuesday, October 31 at 12:01 AM we will be performing router maintenance at our Focal POP in San Francisco. This maintenance will temporarily disrupt all dial to all our numbers ending in 9811 and 9606. Expected loss of connectivity for those numbers is 5-10 minutes. You can find alternate dial-up numbers at www.sonic.net/popf/ -Jared, Zeke, and Nathan

Update Tue Oct 31 01:59:33 PST 2006 — Focal POP maintenance completed. We have completed our scheduled maintenance at our Focal POP. Total downtime for dialup connectivity served from this POP was approximately 15 minutes beginning at 1AM. -Zeke and Jared

Tue Oct 24 09:41:06 PDT 2006 — LA ATM Maintenance. On Thursday, October 26th at 12:01am we will be performing routine maintenance on the equipment that aggregates our Los Angeles DSL customers. This operation is intended to increase performance and reliability for all our LA members. Downtime will be approximately 15 minutes. -Nathan and Kelsey

Update Wed Oct 25 18:18:16 PDT 2006 — Due to circumstances outside of our control, we are postponing this maintenance until 12:01am on Friday, October 27th. We apologize for the inconvenience. -Nathan and Kelsey

Update Thu Oct 26 19:50:05 PDT 2006 — More delays. Do to some miscoordination with another provider we have to put this maintenance off for another night. It is now scheduled for 12:01AM, Saturday. -Nathan

Update Sat Oct 28 01:50:18 PDT 2006 — This maintenance was completed successfully. -Nathan and Kelsey

Sun Oct 15 21:30:40 PDT 2006 — Core NFS Filer Maintenance. Tonight, shortly after Midnight, we are going to perform some upgrades on the NetApp filers that handle all web, ftp and home directories. This is a multi-stage upgrade that will include a few service interruptions during which all dependent services will be unavailable. These upgrades are being conducted to resolve some issues raised by the software and capacity upgrades done last Friday. -Kelsey and Nathan

Thu Oct 12 16:14:36 PDT 2006 — Core NFS Filer Maintenance. Tonight, shortly after Midnight, we are going to perform software upgrades on the NetApp filers that handle all web, ftp and home directories. This is a multi-stage operation that will include a number of short service interruptions during which all dependent services will be unavailable. -Kelsey and Nathan

Update Fri Oct 13 02:46:59 PDT 2006 — Maintenance completed. Both clustered filers are now running the latest code and have had additional capacity added. -Kelsey and Nathan

Mon Oct 9 12:37:31 PDT 2006 — Planned ATM switch maintenance. Tomorrow evening at 10PM we will be installing an additional linecard in one of our ATM switches. We do not anticipate this operation to be service-affecting, however in the event an issue does arise NOC staff will be at the ready. -Network Operations

Mon Sep 25 12:58:57 PDT 2006 — Broadlink outage. A large percentage of the customers on Broadlink’s network are currently unreachable. Broadlink is working to resolve the issue, and we are awaiting an ETR. -Eli, Support 13:46hrs – Update: The cause of the problem was traced to a core ethernet switch that is failing. The equipment is scheduled for an after-hours replacement, and all affected customers are back online.

Sat Sep 16 12:02:41 PDT 2006 — Web cluster instability. This morning our web cluster was intermittently non-responsive due to internal server issues. We have addressed the problems and web pages are being served normally now. We are investigating the root causes of the issues that arose this morning and will be taking steps to prevent them from recurring. -Jared, Dan, Nathan and Kelsey

Thu Sep 14 09:50:29 PDT 2006 — Broadlink communications tower problems have caused many customers to go offline. Broadlink is working to resolve the issue, however, there is no ETR at this time. -Kavan, Matt K., and Broadlink