Month: June 2015

Airport Business Park Fiber Brief Outage

Update(3:35am):Our transit provider has completed maintenance

Update(10:26am): We have confirmation that tonight, in the window of 12am to 4am, our transit provider will be replacing the piece of gear that feeds the ASE connection.  Their management team has guaranteed us that this will happen.  We have made some changes to try to minimize any temporary outage you may experience to the best of our capabilities at this time given what visibility/control we have on the transit link.

 

Update(8:54am): Our Transit provider was unable complete they work they had planned for.  We are escalating aggressively to get this issue resolved.

Update(1:53pm): Our transit provider is going to perform maintenance tonight between the time of 12pm and 4am.  The expected actual downtime is minimal as they are replacing an optic.

Update(11:05AM): ASE customers are also experiencing issues during this time. We are still working towards resolution with our transit provider.

This morning at 9:16 am we experienced a brief momentary loss of connection serving the Airport Business Park.  We are working with our link transit provider to look into the matter.

-Brandon

 

The connection appears to going up and down still.  We are currently working with out Transit provider for a solution.

-Brandon

 

System Maintenance

UPDATE: Work complete.

 

Tonight starting at 11:59pm System Operations will be running updates that will effect the following services:

  • POP3/IMAP
  • Webmail
  • Membertools
  • SMTP
  • FTP
  • IPv6 tunnels
  • customer websites

All of the interuptions should be momentary.

Grant and SOC

Fiber Outage – ASE/Airpot Fiber

Service has been restored

-Brandon

 

We have an outage affecting customers being served off of ASE and the Airport Business park.  We are currently looking into the issue and will update soon.

-Brandon

 

 

Fusion/FlexLink Outage – Danville

This evening, June 23 starting at 9:23pm, circuits serving a subset of Danville Fusion/FlexLink subscribers went down. We are working to restore service as quickly as possible.

Update(2:10am): One of the affected circuits has been repaired. Service has been restored for all affected users.

-Tomoc

IMAP/POP3 outage

Due to the ongonig logging issues we have seen today, there was a brief outage of our IMAP/POP3 services. As of right now both are up and running. Total down time was 25 minutes.

SOC

Internal Log Server Issues

One of our central syslog servers had a series of raid failures that lead to filesystem corruption late last night.  Normally, this would only be an issue for our backend systems but at approximately 8:30AM this morning local logging on our MX servers, which forward logs to that  syslog server, blocked.  This caused inbound mail handling to stop along with other services including the redis sentinel cluster that is used to store configuration information.  That, in turn, caused the membertools to have issues since they depend upon some of the configuration stored in redis.  The MX servers were all back up and running by 9:15AM.  We’re continuing to work on restoring the failed syslog server but there is no customer impact at this time.  – Sonic Ops