Fri Sep 16 12:24:44 PDT 2005 — Santa Rosa, Bennett Valley DSL outage. Customers served DSL on the Bennett Valley DSLAM in Santa Rosa, are currently down. Symptoms include DSL sync, but no Internet traffic. ASI/SBC is currently working to resolve the issue, however, we have no ETR currently. -Kavan, and Technical Support

Thu Sep 15 15:16:54 PDT 2005 — Broadlink maintenance. Broadlink is working on upgrading one of their primary backhaul radio links. Service should be completed at 4pm. -Scott, and Jason from Broadlink

Tue Sep 13 23:59:59 PDT 2005 — Night Ops. Tonight at Midnight, membertools will be unavailable while we upgrade sonic.sonic.net’s hardware. The upgrade should be completed in a couple of hours. Sorry for the late notice. -Kelsey and Nathan

Update Wed Sep 14 04:07:08 PDT 2005 — The hardware upgrade and migration went as planned. All membertools and associated services have been restored. -Kelsey and Nathan

Mon Sep 12 08:21:50 PDT 2005 — Santa Rosa DSL Gateway service restored. One of our Santa Rosa DSL gateways crashed this morning at about 7:45. Downtime was approximately 30 minutes as we tried to elicit a crash dump from the unit. Service is restored. -Scott, Nathan, and Clay

Mon Sep 12 07:47:20 PDT 2005 — One DSL gateway down. One of our DSL gateways is currently inoperative. We should have it back up momentarily. -Scott, Nathan, Clay, and Kelsey

Thu Sep 8 16:17:32 PDT 2005 — One of our DSL termination boxes in San Francisco just rebooted due to a software failure. We expect to have services restored to the affected customers within 15 minutes. -Kelsey and Nathan

Thu Sep 8 15:15:02 PDT 2005 — Mail Server Upgrades. We deployed four new POP3 servers yesterday in order to maintain 2-N redundancy behind our load balancers. While our previous servers were not experiencing load related issues, these upgrades should provide improved interactive performance. -Kelsey, Nathan and Augie

Wed Sep 7 19:00:53 PDT 2005 — Night Ops. Tonight from approximately Midnight to 3:00 AM membertools will be unavailable as we work on upgrading sonic.sonic.net’s hardware. We’ll also be briefly working on the load balancers that handle mail services for customers – some normal maintenance done earlier today left them in a odd condition. We have reason to believe that a _very_ small percentage of requests for pop and smtp may be dropped at this time. We anticipate only a minute of downtime while we reboot the load balancers to clear the issue. -Kelsey and Nathan

Update Thu Sep 8 13:07:12 PDT 2005 — The work on the load balancers went as expected with only 14 seconds of downtime. The upgrade of sonic, however, was aborted due to issues with it’s replacement hardware. -Kelsey and Nathan

Tue Sep 6 12:19:28 PDT 2005 — Broadlink outage on Rosenberg Tower. Sonic.net has been informed of an outage affecting all customers whose Broadlink Wireless connections are routed through the Rosenberg Tower in downtown Santa Rosa. Customers on the Wally Moore tower, and customer connected by non-Broadlink connections should be unaffected. Broadlink technicians are working on this issue right now. – John F

Update Tue Sep 6 12:43:17 PDT 2005 — Broadlink reports a full restoral of service at 12:37pm. – John F and Broadlink

Mon Sep 5 07:43:26 PDT 2005 — Pop server downtime this morning. Between approximately 4:30am and 5:00am this morning, one of the file servers serving the mail cluster lost its beady little mind, and the expected fail-over did not take place. This affected both popmail and mail delivery. No mail was lost.

Resetting the filer’s interface cleared up the problem. We have seen this before, and will be submitting more information to the vendor regarding the failure, in hopes of a speedy resolution. -Scott and Kelsey