Mon Sep 25 12:58:57 PDT 2006 — Broadlink outage. A large percentage of the customers on Broadlink’s network are currently unreachable. Broadlink is working to resolve the issue, and we are awaiting an ETR. -Eli, Support 13:46hrs – Update: The cause of the problem was traced to a core ethernet switch that is failing. The equipment is scheduled for an after-hours replacement, and all affected customers are back online.

Sat Sep 16 12:02:41 PDT 2006 — Web cluster instability. This morning our web cluster was intermittently non-responsive due to internal server issues. We have addressed the problems and web pages are being served normally now. We are investigating the root causes of the issues that arose this morning and will be taking steps to prevent them from recurring. -Jared, Dan, Nathan and Kelsey

Thu Sep 14 09:50:29 PDT 2006 — Broadlink communications tower problems have caused many customers to go offline. Broadlink is working to resolve the issue, however, there is no ETR at this time. -Kavan, Matt K., and Broadlink

Sun Sep 10 21:54:06 PDT 2006 — Webcluster DoS event. A brief but significant DoS attack impacted the performance of our webcluster which serves most customer sites as well as www.sonic.net. The attack is over and at this time the cluster is fully operational. We will continue to monitor closely. -Zeke and Kelsey

Tue Aug 29 09:42:31 PDT 2006 — Broadlink Communications outage. Some customers are being affected by an outage on a sector of the Barham tower. Broadlink is currently working on this issue but does not currently have an estimated time of repair.

Update: Tue Aug 29 10:29:41 PDT 2006 — The outage has been traced to a faulty antenna serving the affected sector. Broadlink is currently replacing this and expect completion by this afternoon. – Jasper C.

Update: Tue Aug 29 16:36:11 PDT 2006 — The antenna has been replaced, and service should be restored to all affected customers.

Mon Aug 21 11:53:22 PDT 2006 — Inbound Mail Server Delays. Our inbound mail complex has been experiencing some trouble over the last hour or two this morning. Although we have yet to identify the root cause we have some work arounds in place. All locally queued mail is being processed now and normal mail flows have been restored. -Kelsey, Jared and Dan

Wed Aug 16 12:47:23 PDT 2006 — DoS Attack Featuring Our Webcluster. A brief but large DoS has destabilized our web server cluster which handles most customer sites and includes www.sonic.net. While it appears the DoS attack is already over it takes a few minutes for the complex of servers to get back into order. We will continue to pay special attention until the system stabilizes. -Kelsey, Nathan, Zeke and Jared

Mon Aug 14 15:19:29 PDT 2006 — Broadlink Communications outage. In the Santa Rosa market, some Broadlink wireless customers are being affected by an outage in their network. Broadlink tells us that they’re working to resolve this issue as quickly as possible.

Update: This issue is resolved. -Dane

Fri Aug 11 15:10:48 PDT 2006 — Internal DNS Server Upgrades. We’ve upgraded the hardware on two of our internal DNS servers to better accommodate the current load they are under. These two servers handle over 10,000 DNS requests per second serving both public and private DNS based blacklists. This upgrade, combined with the hardware upgrades currently in progress on two of our other internal recursive DNS servers, will help prevent a single system failure from adversely affecting our services as a whole. -Kelsey, Augie and Dan

Thu Aug 10 18:51:18 PDT 2006 — Internal DNS Server Outage. One of the many DNS servers that we employ internally started failing early this afternoon. We finished getting it’s replacement online and things are back to normal now. The most notable affect of this failure is in delayed email flows both in and out of our network and skipped SpamAssassin checks. -Kelsey, Augie and Dan