Fri Feb 29 09:43:03 PST 2008 — WBA Petaluma Night Op. At 2am on Sunday morning (3/2/08) all Petaluma WBA customers will be offline for approximately five minutes while we upgrade the wiring system. -Jason

Thu Feb 28 11:07:57 PST 2008 — Combating Web Server Spam. Some security changes to help combat SPAM being sent from our webcluster caused some user’s scripts to break. We have corrected the issues and everything seems to be running normal. You should now be able to sleep better at night knowing that we now have an effective way to quickly track and react to scripts that have been compromised and being used to send out SPAM on you and your websites behalf. -William, Kelsey and the SOC

Tue Feb 26 12:40:45 PST 2008 — Mail Hiccup. One of our clustered NetApp filers just paniced and, shortly after it rebooted, one of its Ethernet cards locked up. These two events lead to two brief POP/IMAP/Webmail server interruptions. Customers may have noticed timeouts or other errors while checking their mail. We’ll be observing the filer closely and once we’ve determined the cause of the failures will take appropriate action. -Kelsey, Augie, Don and William

Wed Feb 20 13:20:39 PST 2008 — WBA Petaluma Outage. At 6:45am we lost power at our Petaluma tower site; we moved a gas generator into place and ran on generated power for approximately 45 minutes before power was restored. Overall downtime was from 6:45am to 7:30am with another blip at about 8:15am when we switched back off generated power. -Jason

Tue Feb 19 12:37:46 PST 2008 — Upstream Network Issue. At 12:15 PM today, one of our upstream network providers suffered an internal network outage, which caused a lack of reachability to many Internet sites. We have routed around that upstream while they repair the problem. -Jared, Dusty, and Jasper

Wed Feb 6 00:14:25 PST 2008 — Emergency maintenance on core administrative server. One of our administrative servers that handles a lot of things behind the scenes like membertool backends, tech support tools, and … writing MOTDs, crashed earlier this evening. As it’s hardware had already been deemed suspect and we had been waiting for a maintenance window we took the opportunity to swap the drives into the spare chassis that was already racked and waiting. At this time all services have been restored and the new hardware is working perfectly. -Don and Kelsey

Thu Jan 31 14:29:24 PST 2008 — Upstream Network Problems. At 2:00pm this afternoon, one of our upstream network providers at our Los Angeles POP experienced a widespread outage. Customers in the LA area would have noticed spotty Internet connectivity. We have since rerouted traffic via a redundant link and the problem appears to have been resolved. We will continue to monitor the affected link and will return it to service when we are assured the problem has been resolved. -Dusty, Nathan and the NOC

Tue Jan 29 15:54:31 PST 2008 — Sonic.net now offers WAP enabled Webmail for your favorite mobile devices (Phone, PDA, etc.); this provides a quick and easy way to access your email when you are on the go. To try it visit m.sonic.net from any WAP2/XHTML compatible mobile device. -Augie

Mon Jan 28 21:16:52 PST 2008 — Hardware changes are going to be performed in a maintenance window from 12:01 AM – 2:00 AM (Tuesday morning, the 29th) on ssl.sonic.net. I will also be performing some changes to the replication system on custsql.sonic.net, which may cause some periods of inactivity. I will also take this opportunity to fix a simple yet important problem on the system which will result in a reboot. Customer down time will be only as required and as minimal as possible. –Don Update: Downtime for custsql was approximately 40 minutes. All other upgrades/changes went as expected. –Don

Sat Jan 26 18:24:57 PST 2008 — San Diego Packet Loss. This afternoon San Diego DSL customers began noticing 10-15% packet loss on their DSL lines. Upon inspection we found that one of the uplinks on our SMS that serves that area was taking errors. We re-routed traffic to the other uplink, thus resolving the packet loss problem. We will be performing maintenance on that SMS soon to resolve the packet loss problem permanently. -Jared and Nathan