Flexlink/Fusion Outage – San Francisco

Update (7:34AM): Service has been restored.

During routine maintenance a piece of equipment starting having issues. We are working to restore service and will update as things progress.

– Network Engineering

Systems Maintenance

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

The systems include, among others:

  • Customer shell services
  • Various public facing applications
  • Some back-end mail systems

Update: Maintenance complete

Shell Server Turn Down

Dane Jasper here, to write a final “message of the day” (MOTD) to honor the retirement of our shell server platform which will be turned off on June 30th when it’s operating system goes end of life.

Since starting Sonic in 1994, the use of Linux has been a key part of our operation, and providing remote text-based shell access to a Linux host has been included in our offerings.  Shell access was a very useful capability in the early days of the internet, when home systems were connected at slow speeds, while a centrally hosted shell server would have very high speed access to the internet.  And shell access provided a centralized gateway to much of the early culture of the internet, via text-based access to the Usenet discussion forums and internet relay chat (IRC) channels, email and more.  Today the internet has largely moved beyond these text-only applications, and Sonic members have too.  Today just one hundred die-hard members continue to log in and access our shell service platform.  As a result of the evolution of the internet away from shell applications and services, plus the workload involved in keeping a secure shell platform online, we have made the decision to finally shut down our Linux shell platform.

As an alternative, we would suggest now that high speed access is widely available at home, consider installing Linux on a small computer in your home!  This was how Scott and I got started with Linux at Sonic, back in the early days of “Sonoma InterConnect” (SonIC, get it?).

There are also lots of remote cloud-based shell solutions available now too.  Here are some starting points if you’d like to establish a shell account or set up a Linux box yourself:
https://www.linuxtrainingacademy.com/get-access-shell-account-learn-linux
https://aruljohn.com/freeshell

Please post questions or comments at https://forums.sonic.net/viewforum.php?f=13

For those needing to continue to maintain procmail recipes you willl need to use ftp://ftp.sonic.net/.

To our Linux shell server platform: So long, and thanks for all the fish.

-Dane, Scott, Kelsey and everyone at Sonic.

System Maintenance

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

The systems include, among others:

  • IPv6 tunneling service
  • VPN servers
  • Various public facing applications

Sonic FTP Server Maintenance

Tonight at 10pm we will be upgrading the operating system on the Sonic customer FTP server ftp.sonic.net. We expect the service downtime to be brief and no longer than 1 hour in total.

Update: This maintenance has been completed successfully.

LA Area Network Maintenance

Update (12:55AM): This maintenance has been completed.

Tonight, starting at 11:59 PM, we will be conducting maintenance on core networking equipment in the LA area. While we anticipate no downtime for end users, there might be occasional routing instability as traffic is rerouted from the affected equipment. The maintenance window for this operation is estimated to be 4 hours.

System Maintenance

UPDATE: Maintenance complete.

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

The systems include, among others:

  • VPN servers
  • Various public facing applications

 

 

Cascading IMAP/POP3 failures this morning

At 11am our back-end IMAP/POP3 cluster entered a critical state which lead to an interruption in those services, as well as other services that rely on our mail infrastructure. The initial cause of the failure was a routine maintenance procedure that involved dropping traffic to a portion of the cluster. While the remaining cluster should have been able to run temporarily with a smaller group, that quickly turned out to not be the case. The remaining servers began to fail intermittently as they tried to shift traffic to account for the sudden increase in load. This would have caused noticeable mail client issues, and it also led to service availability interruptions on both our webmail and our voicemail platforms.

As soon as the problem was detected, we acted by aborting the maintenance. This was followed by additional resources being added to the cluster to prevent further disruptions. As of 11:34am service was fully restored. We do not expect loss of email to have resulted from this. As always we will look into improving our metrics and analytics to improve our response time.