Month: August 2003

Mon Aug 18 15:31:53 PDT 2003 — Night Operations: Tonight at 01:00 AM we are going to be performing maintenance on the SMS 1800 in Santa Rosa which terminates most SBC DSL customers. We do not expect any interruption in services to occur. We will also be taking shell.sonic.net offline to add a second ethernet interface in preparation for the mail server migration on Saturday. -Kelsey and Nathan

Mon Aug 18 14:43:55 PDT 2003 — Mail Server Upgrades: The new MX servers are now handling all of our inbound email flow. We will be continuing to keep a very close eye on the new servers behavior. If you notice any unusual behavior please contact technical support. -Kelsey

Sat Aug 16 13:44:14 PDT 2003 — Major Mail Server Upgrades: Over the past year we have been working hard to design and implement an entirely new mail server architecture. The new architecture is built around load balanced single task servers allowing for high availability and easy on-line expansion of capacity. We have been silently preparing and upgrading our existing servers and network to support the new systems. This includes the deployment of 6 new clustered NFS servers for mail spool storage, a new gigabit Ethernet SAN comprised of 4 switches, four new dedicated internal DNS servers, four new authentication and accounting servers, a new dedicated SpamAssassin user database server, and a pair of Alteon AD3 server load balancing switches in an active-standby automatic fail over configuration. And, of course, the nine new mail servers themselves. (Three of which are already deployed as ‘mail.sonic.net’ and ‘custmx.sonic.net’.) The final stages of deployment are upon us. Monday, August 18th, we are going to begin transitioning the flow of inbound email to our new MX handling servers. For now, these new MX handlers will direct mail to the existing MX handlers for local delivery. There will be no negative customer impact during this migration. Saturday, August 23rd, we are going to swing local delivery and pop to the new delivery and pop servers. This change itself should only take a few minutes to complete; most customers will probably not notice that anything has changed. However, there will be some important and significant changes that take place at this time. The most significant changes are: 1) Mail left on the servers will be redelivered into the new message store format. If your mail client is configured to leave mail on the server, all of these messages with be re-downloaded by your mail program which will create duplicates of all these messages in your inbox. Additionally it should be noted that it may take as long as 12 hours to completely redeliver all email to the new spools. We recommend that users stop leaving mail on our servers immediately and then re-enable it once the migration has finished if they wish. No messages left on our servers will be lost. 2) The new POP3 server supports STARTTLS as well as the currently support alternate port SSL encryption. Users using email clients with broken STARTTLS implementations or with broken proxy servers may need to disable STARTTLS in their email client, upgrade their software or disable their proxy. This is not expected to affect a significant number of customers. 3) The only ‘Officially’ supported shell mail clients will be Mutt and Pine, the only shell clients which have support for Maildir format message stores. We are recommending that users who are not already doing so use Mutt. It should be noted that there are a number of potential solutions for users to continue to use whichever email client that they choose but they will be doing so without official support from Sonic.net staff.

Due to the significance of these changes we will be sending out two emails detailing the changes and expected issues to be encountered to our customers. The first will go to all users. The second will specifically deal with reading email on the shell and will only be sent to users who have enabled shell access.

If you have any questions or comments please direct them to news://news.sonic.net/sonic.net -Kelsey, Nathan, Russ, Kevan, Scott and John from Operations. Eli, Chris, Kavan, Randy and John from Support.

Thu Aug 14 19:34:44 PDT 2003 — Intermittent busies on 522-1003. We are working to identify the problem. Please note that we have alternate dial-up numbers available at www.sonic.net/cgi-bin/pops.pl -Matt and John

Update Thu Aug 14 20:17:58 PDT 2003 — Rebooting a router card has cleared the busy signals. We will be replacing this card in the next few days. -Matt, Russ and John

Thu Aug 14 15:50:43 PDT 2003 — Northeastern United States power outage. Cities affected include New York; Boston, Massachusetts; Cleveland, Ohio; Detroit, Michigan; Toronto, Ontario; and Ottawa, Ontario. The power outage occurred at around 4:20 p.m. Eastern.

On the Internet, we have seen some small East Coast web and mail servers become unresponsive due to this event. Most large websites and services are hosted in colocation facilities with power backup, so this hasn’t caused a very significant impact online.

In the real world, in New York city, cellular providers are offline, people are trapped in subways and elevators, and they’re having a heat wave. Streets and bridges are packed with people, walking home from work. It’s a mess. -Dane

Tue Aug 12 12:37:13 PDT 2003 — Sonic.net is portscanning our IP ranges for open port 4444, which may be an indication that a system is infected with the Microsoft Blaster worm. FYI, in case your firewall logs note this portscan. -Dane, Nathan and Eli

Tue Aug 12 17:31:05 PDT 2003 — Intermittent busies: A failure in one of the 8 load-balanced access routers at our primary dialup POP caused intermittent busy signals on some of our access numbers. The router has been rebooted and is now working properly. -Russ

Mon Aug 11 13:58:30 PDT 2003 — Blaster worm. A new Internet worm, called “blaster”, is currently spreading throughout the Internet. Information about this worm can be found at:

www.sarc.com/avcenter/venc/data/w32.blaster.worm.html

The worm is designed to exploit a known DCOM RPC vulnerability using TCP port 135. This is a vulnerability affecting computers running Microsoft Windows. Information about the vulnerability can be found at:

www.microsoft.com/technet/treeview/?url=/technet/security/bulletin/MS03-026.asp The recommended action is for all users of all versions of MS Windows to Update their software as soon as possible. We have had reports that users of Windows XP may experience difficulty in dialing up due to the traffic being generated by this worm’s activity; turning on the Internet Connection Firewall provided with XP should make it possible to dial up.

While Sonic.net is not blocking port 135 at this time, we may find it necessary to do so in the future. If you use applications that require traffic over TCP port 135 you might wish to explore switching to an alternate port. -Chris, Scott, Nathan and Support

Fri Aug 8 20:28:01 PDT 2003 — Sebastopol POP Maintenance: Tomorrow morning at 8:00AM our Sebastopol POP, which serves (707) 823-8812, will go offline while its UPS is serviced. We hope the outage will be brief but, the worst case is that it may take up to an hour before the POP will be back online. Customers are encouraged to use the pop finder to find another local access number in their area if needed. The pop finder is available at www.sonic.net/cgi-bin/pops.pl -Network Operations

Thu Aug 7 15:18:53 PDT 2003 — Broadlink emergency maintenance. They are currently doing work on one of their tower sites, and this has created some brief interruptions. -Dane