Amazon Traffic Outage

This evening, November 18, starting around 5:00pm, Amazon began blackholing traffic towards the Sonic network. We attempted to route through a different upstream provider, but it appears the routing issue was too deep into the Amazon network. We believe they have fixed the routing issue, and all traffic has been restored as of 5:25pm.

Update: This issue appears to have re-surfaced. We are reaching out to Amazon to determine the cause of the outage and do everything we can to ensure it does not happen again.

-Tomoc and the NOC

Fusion/FlexLink Intrusive Maintenance – San Francisco

Update(3:20AM): This maintenance is now complete.

Beginning tonight at midnight, we will be performing maintenance on equipment serving a small portion of Fusion and FlexLink customers in the San Francisco area. This operation can cause downtime for up to a few hours in some cases.

– Robbie and Michael

Core Router Maintenance

Update(3:30am): This maintenance is now complete.

Beginning tomorrow morning at 3:00AM, I will be performing maintenance on a core router in the San Jose area. There should be no downtime as a result of this operation, however, users may notice traffic and routing changes.

– Robbie

Server Upgrades

Update: Maintenance complete.

Tonight at 10pm, Operations will be upgrading one of the servers responsible for handling account and web-based services. During this time, Member Tools and Sign-ups will be unavailable. The expected downtime is 30 minutes.

— Joe @ System Operations Team

Scheduled System Maintenance

Update: Maintenance complete.

Tonight at 11pm we will be performing updates to several systems; Impacted customer facing services will include mail and member tools.  Any interruptions to service should be brief and the maintenance should be completed within 1 hour. Thank you.

Fusion/FlexLink Maintenance – San Francisco

Tonight at midnight a very small subset of Fusion and FlexLink Ethernet customers located in San Francisco will experience about 10 minutes of downtime due to emergency system maintenance needed to restore some network services.

 

12:14AM update: This maintenance is now complete

 

-Michael

Fusion/FlexLink Maintenance – San Francisco

Update( 4:30AM): This maintenance is now complete.

Beginning tonight at midnight, we will be performing hardware and transport upgrades for a small portion of customers in San Francisco. Affected users will experience some routing changes and periods of downtime while the upgrades take place. This type of maintenance typically takes a few hours.

– Robbie and Michael

Fusion/FlexLink Intrusive Maintenance – San Francisco

Update ( 1:00AM ): This maintenance is now complete.

Beginning tonight at midnight, we will be performing hardware and transport upgrades for a small portion of customers in San Francisco. Affected users will experience some routing changes and periods of downtime while the upgrades take place.

– Robbie and Michael

Non-Intrusive Voice Maintenance

Tonight starting at midnight, we will be performing maintenance updates to our Phone Switch.

There should be no issues placing or receiving calls during this time, however it is possible that you could experience a call drop, or call quality spikes during this time.

Estimated maintenance time is ~2 hours

Thank you – Brandon

 

Update – 2:04am – 10/27/2015

Maintenance is going longer than expected. Estimated completion time by 3 am.

Update – 2:26am – 10/27/2015

Maintenance is complete.