Wed Jun 4 21:16:10 PDT 2003 — Routing issues at Sebastopol POP. The last portmaster in the dial group that servers Sebastopol had an apparent misconfiguration that was just uncovered. It has been fixed and now customers using this box are fully functional. -Kelsey, Nathan and Russ

Tue Jun 3 17:41:15 PDT 2003 — Broadlink head-end router reboot. The Broadlink router which connects their subscribers to the Sonic network rebooted ten minutes ago. At this time, all subscribers should be up and functional. -John and Nathan

Tue Jun 3 15:58:35 PDT 2003 — News server lockup. Our news server, news.sonic.net, locked up a few minutes ago. We are working to restore service, but in the mean time, altnews.sonic.net and supernews.sonic.net remain available. -Kelsey and Nathan

Update: The server hardware for news.sonic.net appears to be having some difficultly: the box has rebooted itself a few times. We are investigation options we have to replace the hardware. At this time it is back online, no wait, it’s down. -Kelsey and Nathan

Update: News.sonic.net is back online, one of it’s CPU had a failed cooling fan, it has been replaced. -Kelsey, Kevan, Zeke and Nathan

Tue Jun 3 00:30:36 PDT 2003 — Name server issue. We experienced an issue with one of our primary name servers where an old configuration directive caused the server to shutdown. The problem has been fixed. Sonic.net has multiple name servers and customers should not have noticed the issue. -Matt and Kelsey

Mon Jun 2 16:52:13 PDT 2003 — Sundry Updates to DNS, Authentication and SpamAssassin. We were able to track down the sources of the periodic problems with our DNS servers and made some changes that appear to have resolved the problems. The DNS server failures were also causing cascading failures on our mail and authentication servers. We’ve taken additional measures to insulate these services from failures of our core DNS servers. We’ll still be going ahead with the DNS server upgrades and replacement.

We also did some profiling and performance tuning on our SpamAssassin cluster and were able to drive the average time to inspect a message down to half a second from between 2 to 6 seconds. This marked increase in performance should result in snappier email delivery and less load on our core mail servers. -Kelsey and Nathan

Mon Jun 2 16:45:06 PDT 2003 — Delayed Mail Server Migration: We’ve delayed the migration of ‘mail.sonic.net’ onto the new mail servers until we can resolve some issues surrounding the support of POPB4SMTP authentication. Our logs had indicated that not a single user was requiring POPB4STMP auth to send mail, upon reexamination, we turned up several hundred users. At this time we are unsure if we will enable support for POPB4SMTP auth or if we will assist these customers in transitioning to SMTPAUTH. -Kelsey, Nathan, John and Russ.

Fri May 30 09:57:32 PDT 2003 — Name server issues. We’ve had occasional issues with our nameservers, which can cause slow queries as the queries fail over to redundant nameservers. We think that doubling the 1GB of memory to 2GB will greatly improve the situation, and toward that end we have three 2GB servers that will shortly become our new nameserver cluster. -Scott, Kelsey, and Operations

Tue May 27 10:08:52 PDT 2003 — Mail Server Migration: On Monday the 2nd of June we will be migrating ‘mail.sonic.net’ to a new cluster of servers. This is the first stage in the complete re-architecture of our mail server infrastructure. There are a number of significant changes on the new servers which may affect some customers. We have replaced POPB4SMTP authentication with SMTPAUTH and enabled STARTTLS. SMTPAUTH and STARTTLS are well defined standards that allow users to securely authenticate and relay mail off our servers while they are not directly connected to our network. Some users, connecting from remote networks may need to make a few minor configuration changes to their email clients. If you have any questions please contact Technical Support or post them to news://news.sonic.net/sonic.net -Kelsey

Tue May 27 13:17:19 PDT 2003 — Our free DSL equipment (with rebate!) offer has been extended by 24 hours, to this Thursday the 29th at Midnight. We are trying to hit a 5% growth target (or SBC doesn’t fund it, leaving us absorbing the costs), so if you were thinking of switching to DSL, now would be a great time, at least from our perspective. =) -Dane

Mon May 25 10:30:17 PDT 2003 — NetApp filer lockup. One of our NetApp filers locked up this evening causing the redundant filer to perform an automated take-over. After investigation, no cause for the failure could be determined. We rebooted the troubled filer and transfered control back to the unit and both filer are functioning normally now. Sonic.net has invested in a redundant, fail-safe storage system which is designed to handle such events. The failure of a filer is transparent to customers as when one filer experiences problems which prevent data from being served, the other file takes over control of the downed filers disks and serves its data seamlessly. Our Network Appliance F740 network file system cluster is the basis of Sonic.net’s storage solution. All user data resides on the two NetApp filers, and they’re configured in a completely redundant configuration. The drives are dual-channel fiber arbitrated loops, and are served by redundant processor heads. The units are RAID level four, WAFL filesystems, and include redundant power and cabling to all disks and the network itself at Gigabit speeds. -Matt and Kelsey