Wed Feb 9 03:28:35 PST 2005 — Storage Network Fixed. It appears that one of the storage network switches hit a software bug and went into a state where it only selectively forwarded traffic. Reloading the switch corrected the problem. We’ll continue to investigate the cause of the fault and work with our vendor to correct it. -Kelsey

Wed Feb 9 02:20:26 PST 2005 — Storage Network Failure. At least one switch in our NFS SAN is currently off-line. The scope of this failure is currently unknown but it is not believed to have affected basic service availability. Someone is en route now to investigate and repair. -Kelsey

Tue Feb 8 15:59:43 PST 2005 — Anti Virus Filter Upgrade. We’ve updated our AV scanning engine to the latest stable release of ClamAV. For more information about ClamAV please see www.clamav.net The new version adds additional support for catching some generic virus signatures as well as improved overall performance. While all mail sent to a sonic.net hosted email account is scanned at least twice for viruses and other malware before being received by a customer, it is still a good idea to run AV software on your own computer. -Kelsey

Sun Feb 6 12:38:47 PST 2005 — Broadlink report on primary backhaul circuit. Broadlink is still experiencing problems with their primary backhaul circuit. They report that they are in the process of making adjustments to rectify the condition. Steps taken so far have not effected a resolution. They are continuing to work the problem, and will provide an update in the next two hours. Thanks to Jason and Tim for the long hours they’ve put into a resolution so far. -Scott, and Tim and Jason from Broadlink

Sun Feb 6 09:28:40 PST 2005 — Packet loss problem isolated on Broadlink network. Broadlink has isolated the problem with their backhaul, and they are working to effect a fix. ETR is about 1.5 hours. Please note that link will occasionally drop while they are effecting these repairs. -Scott and Tim with Broadlink

Sat Feb 5 12:39:33 PST 2005 — Packet loss on the Broadlink network. Severe packet loss has been detected on the Broadlink networks backhaul. Work is in progress to restore stability, we have minimized the loss to about 5% until full diagnosis and repair can be completed. -Scott, Kavan, and Tim with Broadlink

Wed Feb 2 10:30:13 PST 2005 — Authentication Server Upgrades. We migrated our four primary authentication servers yesterday to newer, faster, hardware. The new hardware is nearly twice as fast. We’re also going to be deploying a cluster of authentication servers in San Francisco in the near future to provide additional redundancy during network events. -Augie and Kelsey

Wed Feb 2 17:16:54 PST 2005 — Testing of the new mailman system. We are moving the motd-list to our new Mailman server, and this represents the first post to the new system. No problems are expected, and this should be a seamless upgrade. -Kevan

Sat Jan 29 10:14:28 PST 2005 — User Database Issue. Between 8:54 and 9:13 this morning it appears the system that creates and manages the unix passwd databases encountered an unexplained error and pushed a truncated file to our servers. The next automated update corrected the situation. During this period mail to many sonic customers was returned with ‘User Unknown’ errors. Similarly, many users would have received errors when retrieving their mail, trying to access FTP, and in specific cases, some users’ websites may have returned errors. -Kelsey

Thu Jan 27 11:22:26 PST 2005 — New DSL Reverse DNS Tool. DSL customers with static IP addresses can now set their own reverse DNS records with the new member tool. The only requirement imposed on the RDNS records is that the forward already matches the requested reverse record. All manually configured and currently valid records were preserved. sonic.sonic.net/members/dsldns/ -Kelsey and Chris B.