Early this morning a library used by many of our backend rpc services was updated as part of our nightly automatic upgrade process. This particular update caused an issue that required services which depended upon it to be restarted in order for them to function correctly under some specific circumstances. Unfortunately most of those specific circumstances are not tested for by our monitoring suite and so we didn’t discover the issue until this morning.
During this period we were unable to process some payments and the Fusion Voicemail system wasn’t able to process inbound calls correctly. All services have been restored now and we’re continuing to dig into the underlying cause of the bug and what can be done to prevent it from happening again.
-Kelsey and the rest of System Operations