Mail server upgrade: The mail server…

Sat Aug 23 09:57:09 PDT 2003 — Mail server upgrade: The mail server conversion will start at 10:30 am PDT today. This migration is expected to cause some delays in delivery of messages. If you experience this, please wait and try again in a short time. Tech support can not expedite delivery of email to your mail box during the migration.

For more information, please see:

www.sonic.net/whatsnew/mailservers/

If your email client reports an actual error when downloading your messages, please call tech support. We may be experiencing higher than usual call volumes, so please be patient. -Operations and Tech Support

Update 11:11:24 PDT 2003 — We have migrated to the new POP servers and new email is being delivered on the new servers. We have corrected one small problem and are working on another one involving mixed case logins. We expect to resolve that shortly.

Update 12:18:57 PDT 2003 — So far everything looks good. We are beginning the migration of old email spools to the new system now. -Operations

Update 14:08:50 PDT 2003 — 98% of mailboxes converted. We are converting the mailboxes in smallest to largest size order. Only the customers with the largest mailboxes are left to do. -Operations

Mailbox conversion done.

Sat Aug 23 15:22:12 PDT 2003 — Mailbox conversion done. The regular mailboxes have all been converted to the new format. We are now converting folders from webmail. We estimate this will take about three hours. -Kevan, Kelsey and John

Altnews service unavailable.

Thu Aug 21 23:05:39 PDT 2003 — Altnews service unavailable. The company that provides the news service reachable as altnews.sonic.net has been down and unavailable all day today. They had a planned outage to move some equipment and apparently ran into problems. The current status message says they expect to be back by 2 am Friday PDT. -John

Update: We got our last error message from Altnews at 9:21 am PDT, Friday, Aug. 22. Their server appears to be up and responding now. -John

Mail Server Upgrades: The new MX servers are…

Mon Aug 18 11:32:49 PDT 2003 — Mail Server Upgrades: The new MX servers are handling a share of inbound email handling now. So far everything appears to be going well. If the new servers still appear to be working properly in a few hours we will direct all inbound email flow to pass through them. -Kelsey, Nathan and Russ

Mail issues from the shell server: A long…

Mon Aug 18 22:39:50 PDT 2003 — Mail issues from the shell server: A long lived configuration error in sendmail was uncovered today when the new MX servers where brought online. This only affected mail sent from bolt to unqualified addressees and possibly to ‘user@sonic.net’ as well. This mail might have intermittently been bounced by the new servers. We’ve corrected the error and bolt is now properly addressing and routing email sent to sonic.net. -Kelsey

Sebastopol POP down.

Mon Aug 18 15:50:44 PDT 2003 — Sebastopol POP down. The router at our Sebastopol facility is having problems. We are on our way out with a replacement, but until that time, dialup will not work through 707-823-8812. Please use our POP finder at www.sonic.net/cgi-bin/pops.pl to find an alternate. -Zeke, John, Nathan and Matt

Update: The router’s services have been restored but we are still going to replace the router around 5:00PM today. Users accessing the Sebastopol POP may notice a brief interruption as the replacement is swapped in.

Update: The Sebastopol POP router has been replaced.

Night Operations: Tonight at 01:00 AM we are…

Mon Aug 18 15:31:53 PDT 2003 — Night Operations: Tonight at 01:00 AM we are going to be performing maintenance on the SMS 1800 in Santa Rosa which terminates most SBC DSL customers. We do not expect any interruption in services to occur. We will also be taking shell.sonic.net offline to add a second ethernet interface in preparation for the mail server migration on Saturday. -Kelsey and Nathan

Mail Server Upgrades: The new MX servers are…

Mon Aug 18 14:43:55 PDT 2003 — Mail Server Upgrades: The new MX servers are now handling all of our inbound email flow. We will be continuing to keep a very close eye on the new servers behavior. If you notice any unusual behavior please contact technical support. -Kelsey

Major Mail Server Upgrades: Over the past…

Sat Aug 16 13:44:14 PDT 2003 — Major Mail Server Upgrades: Over the past year we have been working hard to design and implement an entirely new mail server architecture. The new architecture is built around load balanced single task servers allowing for high availability and easy on-line expansion of capacity. We have been silently preparing and upgrading our existing servers and network to support the new systems. This includes the deployment of 6 new clustered NFS servers for mail spool storage, a new gigabit Ethernet SAN comprised of 4 switches, four new dedicated internal DNS servers, four new authentication and accounting servers, a new dedicated SpamAssassin user database server, and a pair of Alteon AD3 server load balancing switches in an active-standby automatic fail over configuration. And, of course, the nine new mail servers themselves. (Three of which are already deployed as ‘mail.sonic.net’ and ‘custmx.sonic.net’.) The final stages of deployment are upon us. Monday, August 18th, we are going to begin transitioning the flow of inbound email to our new MX handling servers. For now, these new MX handlers will direct mail to the existing MX handlers for local delivery. There will be no negative customer impact during this migration. Saturday, August 23rd, we are going to swing local delivery and pop to the new delivery and pop servers. This change itself should only take a few minutes to complete; most customers will probably not notice that anything has changed. However, there will be some important and significant changes that take place at this time. The most significant changes are: 1) Mail left on the servers will be redelivered into the new message store format. If your mail client is configured to leave mail on the server, all of these messages with be re-downloaded by your mail program which will create duplicates of all these messages in your inbox. Additionally it should be noted that it may take as long as 12 hours to completely redeliver all email to the new spools. We recommend that users stop leaving mail on our servers immediately and then re-enable it once the migration has finished if they wish. No messages left on our servers will be lost. 2) The new POP3 server supports STARTTLS as well as the currently support alternate port SSL encryption. Users using email clients with broken STARTTLS implementations or with broken proxy servers may need to disable STARTTLS in their email client, upgrade their software or disable their proxy. This is not expected to affect a significant number of customers. 3) The only ‘Officially’ supported shell mail clients will be Mutt and Pine, the only shell clients which have support for Maildir format message stores. We are recommending that users who are not already doing so use Mutt. It should be noted that there are a number of potential solutions for users to continue to use whichever email client that they choose but they will be doing so without official support from Sonic.net staff.

Due to the significance of these changes we will be sending out two emails detailing the changes and expected issues to be encountered to our customers. The first will go to all users. The second will specifically deal with reading email on the shell and will only be sent to users who have enabled shell access.

If you have any questions or comments please direct them to news://news.sonic.net/sonic.net -Kelsey, Nathan, Russ, Kevan, Scott and John from Operations. Eli, Chris, Kavan, Randy and John from Support.

Intermittent busies on 522-1003.

Thu Aug 14 19:34:44 PDT 2003 — Intermittent busies on 522-1003. We are working to identify the problem. Please note that we have alternate dial-up numbers available at www.sonic.net/cgi-bin/pops.pl -Matt and John

Update Thu Aug 14 20:17:58 PDT 2003 — Rebooting a router card has cleared the busy signals. We will be replacing this card in the next few days. -Matt, Russ and John