Tonight, beginning at 10PM PST, we will be applying updates to multiple systems. Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.
The systems include, among others:
Update: Maintenance complete
Dane Jasper here, to write a final “message of the day” (MOTD) to honor the retirement of our shell server platform which will be turned off on June 30th when it’s operating system goes end of life.
Since starting Sonic in 1994, the use of Linux has been a key part of our operation, and providing remote text-based shell access to a Linux host has been included in our offerings. Shell access was a very useful capability in the early days of the internet, when home systems were connected at slow speeds, while a centrally hosted shell server would have very high speed access to the internet. And shell access provided a centralized gateway to much of the early culture of the internet, via text-based access to the Usenet discussion forums and internet relay chat (IRC) channels, email and more. Today the internet has largely moved beyond these text-only applications, and Sonic members have too. Today just one hundred die-hard members continue to log in and access our shell service platform. As a result of the evolution of the internet away from shell applications and services, plus the workload involved in keeping a secure shell platform online, we have made the decision to finally shut down our Linux shell platform.
As an alternative, we would suggest now that high speed access is widely available at home, consider installing Linux on a small computer in your home! This was how Scott and I got started with Linux at Sonic, back in the early days of “Sonoma InterConnect” (SonIC, get it?).
There are also lots of remote cloud-based shell solutions available now too. Here are some starting points if you’d like to establish a shell account or set up a Linux box yourself:
https://www.linuxtrainingacademy.com/get-access-shell-account-learn-linux
https://aruljohn.com/freeshell
Please post questions or comments at https://forums.sonic.net/viewforum.php?f=13
For those needing to continue to maintain procmail recipes you willl need to use ftp://ftp.sonic.net/.
To our Linux shell server platform: So long, and thanks for all the fish.
-Dane, Scott, Kelsey and everyone at Sonic.
Wednesday, May 1st, starting at 11:59PM, we will be upgrading network equipment serving all fiber customers in San Leandro. There is no expected downtime or service interruptions. The maintenance window will be 2 hours.
-Network Engineering
UPDATE(11:25am) The issue with Sonic voice services has been resolved. Thank you for your patience.
Tonight, beginning at 10PM PST, we will be applying updates to multiple systems. Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.
The systems include, among others:
Tonight at 10pm we will be upgrading the operating system on the Sonic customer FTP server ftp.sonic.net. We expect the service downtime to be brief and no longer than 1 hour in total.
Update: This maintenance has been completed successfully.
Update (12:55AM): This maintenance has been completed.
Tonight, starting at 11:59 PM, we will be conducting maintenance on core networking equipment in the LA area. While we anticipate no downtime for end users, there might be occasional routing instability as traffic is rerouted from the affected equipment. The maintenance window for this operation is estimated to be 4 hours.
UPDATE: Maintenance complete.
Tonight, beginning at 10PM PST, we will be applying updates to multiple systems. Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.
The systems include, among others:
At 11am our back-end IMAP/POP3 cluster entered a critical state which lead to an interruption in those services, as well as other services that rely on our mail infrastructure. The initial cause of the failure was a routine maintenance procedure that involved dropping traffic to a portion of the cluster. While the remaining cluster should have been able to run temporarily with a smaller group, that quickly turned out to not be the case. The remaining servers began to fail intermittently as they tried to shift traffic to account for the sudden increase in load. This would have caused noticeable mail client issues, and it also led to service availability interruptions on both our webmail and our voicemail platforms.
As soon as the problem was detected, we acted by aborting the maintenance. This was followed by additional resources being added to the cluster to prevent further disruptions. As of 11:34am service was fully restored. We do not expect loss of email to have resulted from this. As always we will look into improving our metrics and analytics to improve our response time.
Yesterday afternoon at approximately 4:45PM we completed the turn up and migration of services to a new set of geographically redundant proxy servers used by our internal mail infrastructure to communicate between the various clusters that support email services. Unfortunately one of the new proxies was left out of an ACL which prevented that proxy from being able to connect to the mail forwarding cluster. This resulted in about 50% of forwarded mail (typically configured in the “Forwarding” section of Sonic Member Tools) being rejected. These rejected messages would have resulted in non-deliverable messages being returned to the original sender instead of being delivered as intended.
We apologize for any issues this may have caused and will review our systems for any opportunities to improve monitoring to ensure that if something like this happens again it is immediately detected and resolved.