Thu Mar 13 07:44:22 PDT 2008 — Botnet attack directed at inbound mail cluster. Since around 4am this morning we have seen a wide spread attack against our inbound mail cluster; we have taken steps to mitigate the impact, but from 4am – 6am mail coming into our servers may have been delayed. We are keeping a close eye on things and will react as appropriate while the attack lasts. -Augie and William.

Update Thu Mar 13 00:26:07 PDT 2008 — DSL Migration Complete. We have completed our migration of DSL customers. Total customer downtime was less than 10 minutes. -Jared and Dusty

Tue Mar 11 10:41:54 PDT 2008 — DSL Customer Migration. This Thursday at 12:01 AM we will be migrating our Los Angeles DSL subscribers to a new Redback. We have identified some minor hardware issues with the Redback these subscribers are currently on, and while these issues are not causing problems for our DSL subscribers, we are moving them to a new Redback to avoid impact when we replace the questionable hardware in the old Redback. -Jared and Dusty

Update Thu Mar 6 17:28:32 PST 2008 — DDoS Mitigated. The DDoS attack has been blocked at our edge and all customer connectivity problems stemming from it should be resolved. -Nathan, Jared, Matt, Dusty and Tim.

Thu Mar 6 17:04:21 PST 2008 — Inbound DDoS. We are presently experiencing an inbound DDoS. This has temporarily overloaded some of our San Francisco based transit and customers may experience spotty connectivity to some sites. We are working with our upstream providers to mitigate and resolve this issue. -Sonic NOC

Update Thu Mar 6 17:28:32 PST 2008 — DDoS Mitigated. The DDoS attack has been blocked at our edge and all customer connectivity problems stemming from it should be resolved. -Nathan, Jared, Matt, Dusty and Tim.

Update Mon Mar 3 01:16:05 PST 2008 — Router Maintenance Complete. We had run into a software bug on our Business-T and FRATM router that was impacting performance for all customers terminated on it. We upgraded to new software during the router reboot which has resolved the performance problem. All Business-T and FRATM circuits are performing at expected levels at this time. -Jared and Dusty

Sun Mar 2 21:50:13 PST 2008 — Emergency Router Maintenance. On Monday, March 3rd at 1:00 AM we will be performing a reboot on one of our customer aggregation routers. All Business-T and FRATM customers (except those in the Los Angeles area) will experience a few minutes of downtime as the router reloads. -Dusty and Jared

Update Mon Mar 3 01:16:05 PST 2008 — Router Maintenance Complete. We had run into a software bug on our Business-T and FRATM router that was impacting performance for all customers terminated on it. We upgraded to new software during the router reboot which has resolved the performance problem. All Business-T and FRATM circuits are performing at expected levels at this time. -Jared and Dusty

Fri Feb 29 09:43:03 PST 2008 — WBA Petaluma Night Op. At 2am on Sunday morning (3/2/08) all Petaluma WBA customers will be offline for approximately five minutes while we upgrade the wiring system. -Jason

Thu Feb 28 11:07:57 PST 2008 — Combating Web Server Spam. Some security changes to help combat SPAM being sent from our webcluster caused some user’s scripts to break. We have corrected the issues and everything seems to be running normal. You should now be able to sleep better at night knowing that we now have an effective way to quickly track and react to scripts that have been compromised and being used to send out SPAM on you and your websites behalf. -William, Kelsey and the SOC

Tue Feb 26 12:40:45 PST 2008 — Mail Hiccup. One of our clustered NetApp filers just paniced and, shortly after it rebooted, one of its Ethernet cards locked up. These two events lead to two brief POP/IMAP/Webmail server interruptions. Customers may have noticed timeouts or other errors while checking their mail. We’ll be observing the filer closely and once we’ve determined the cause of the failures will take appropriate action. -Kelsey, Augie, Don and William