Internal Log Server Issues

One of our central syslog servers had a series of raid failures that lead to filesystem corruption late last night.  Normally, this would only be an issue for our backend systems but at approximately 8:30AM this morning local logging on our MX servers, which forward logs to that  syslog server, blocked.  This caused inbound mail handling to stop along with other services including the redis sentinel cluster that is used to store configuration information.  That, in turn, caused the membertools to have issues since they depend upon some of the configuration stored in redis.  The MX servers were all back up and running by 9:15AM.  We’re continuing to work on restoring the failed syslog server but there is no customer impact at this time.  – Sonic Ops

2 comments for “Internal Log Server Issues

Leave a Reply

Your email address will not be published. Required fields are marked *