Today we fixed a problem on the Customer Web Cluster that would have effected mail flow from Customer scripts on our Web Cluster that were using the local Sendmail program to send e-mail. The problem began on Sunday and would have appeared as an intermittent failure to Customer scripts as only two of the servers in our Web Cluster displayed the problem.
We will be crafting more detailed monitoring so that we can resolve this problem much faster in the future.
–Augie and William.
Starting this morning, we observed a transient problem with one of our upstream providers. Impact would have been inability to connect to certain websites. We have taken action to correct this issue. -Nathan, Tim and the NOC.
SpamAssasin may have stopped filtering mail for some users this afternoon. No mail was lost during this period. Service has restored to normal at this time and mail is being filtered for all users again. We apologize for any inconvenience this may have caused. -William
This morning one of our DHCP servers in our Los Angeles PoP suffered a disk failure that prevented it from storing leases that customers obtained. This should not have affected customers connectivity to the Internet. We have swapped over to spare hardware that we had on site to resolve the issue. A small minority of customers may have experienced a brief hiccup while we made the transition while most customers should not have even noticed the transition. -William and Jared
Tonight, shortly after midnight, we will replace the failing disk shelf in one of our Network Appliance filers responsible for the POP, IMAP and Webmail service interruptions earlier today. Replacing the shelf is not expected to take more than 30 minutes. During the maintenance users may not be able to check their email. However, new mail will be queued for delivery on our MX cluster and all outbound email will continue to flow unaffected. -Kelsey
Uptdate – The faulty shelf has been replaced and all services have been fully restored. Total downtime for POP, IMAP and Webmail was less than 20 minutes. -Kelsey
Early this afternoon, we experienced a failure in one of our Netapps which caused some content to be unavailable, and a few key systems to present timeout errors. Downtime is estimated to be between 5 and 7 minutes. We have identified the problem and restored all services. –William, Augie, and Don
At 12:55PM this afternoon, we experienced an outage of our main web cluster. Customer downtime was under 5 minutes. All services have been restored. –Don
Update: Upon investigation, we have taken steps to increase the robustness of this cluster to ensure this particular problem does not occur again. We apologize for the inconvenience.
Sonic.net System Operations has upgraded our IMAP server software to a version we’ve had in testing. Besides a noticeable speed improvement over the previous version, we have made message Tagging and client message status available!
–Don, Kelsey and the SOC.
Technical Support will be closed between 11am and 6pm today as Sonic.net celebrates its fourteenth birthday. The NOC Hotline will still be available for our co-located and high capacity customers. Any voice-mail messages left while we’re out will be returned promptly when we return. We apologize for the short notice.
This Friday at 12:01 AM we will be inserting an interface card into one of our ATM switches at our San Francisco POP. This new card will allow us to continue to expand our ATM infrastructure to support DSL growth. Card insertion is not expected to cause any interruption of service to customers served by this ATM switch.
Update: This maintenance has been completed with zero customer impact.
-Jared and Nathan