System Maintenance

UPDATE: Maintenance complete

 

Starting at 11:59pm this evening, SOC will be running updates on multiple customer facing systems. Downtime should be limited, and affected services of particular note include:

  • Open VPN
  • Email
  • Customer Websites
  • Customer Databases
  • Sonic.com
  • Member Tools
  • Webmail

This post will be updated when the maintenance is complete.

-SOC

Updating firmware for Pace 5268ac

We will be updating the firmware to our Pace 5268ac modems over the next few weeks. This will happen Monday through Thursday nights between 12 am and 4 am.  There will be 5 to 10 minutes of down time as the modem reboots after the new firmware is installed. We will update this post when we have finished the upgrades, approximately 3 weeks from now.

 

Thank You

Sonic

Moss Beach – Fusion Outage

Update (4:15PM): It appears service has been restored as of 3:55PM, but we are still awaiting confirmation from our vendor stating all work to be officially complete.

Update (12:10PM): This outage is believed to be part of a large scale outage in the area. A significant fiber cut is likely the cause.

We are currently investigating an outage situation in the Moss Beach area. We will update this message when we have further information.

  • Network Engineering

San Francisco – Sonic Business Fiber Intrusive Maintenance

Tonight on 12/14/2017 at 11:59PM, we will be performing maintenance on a subset of Sonic Business Fiber customers in the San Francisco area. Expected customer downtime is less than 5 minutes per customer.  The maintenance window will be 1 hour.

Update(1:04am): All maintenance is now complete

-Network Engineering

 

Support Closing Early for Holiday Party

Sonic Support will be closing early tomorrow, 12/15/2017, at 5pm for our annual Staff Holiday Party.

Our 24×7 network operations team will remain on call for our Enterprise, dedicated circuit and colocated customers.

Support will return Saturday 12/16 to our normal working hours, 8am to 10pm.

-Dan & Support

Fusion Voicemail Message Delays

Voicemail messages on our fusion voices platform experienced a delay in delivery today between 9am and 2pm. This was due to a problem that arose during a hardware and software upgrade on the servers which host the service. We have rolled back our upgrades for the time being while we diagnose the problem further. All delayed messages have been delivered and the service is currently functioning normally.

Update: Further voicemail service interruptions have been narrowed down to issues with a text to speech module in the voicemail platform. Service should be stable.

Update: Unfortunately, a second issue with the new platform caused some voicemails to be queued and not delivered again over the weekend.  All of the queued messages have been delivered now.  While we investigate and resolve the underlying issue we’re also adding additional monitoring to ensure that should it happen again it doesn’t go unnoticed.  We’re sorry for any trouble this may have caused, prompt delivery of voicemail message is obviously very important.

-SOC

San Francisco Emergency Fiber Maintenance

Tonight at 11:59PM, we will be performing emergency maintenance on equipment serving a subset of San Francisco Fusion Fiber customers. Expected downtime is less than 1 hour. This maintenance window is 5 hours.

Update(2:47am): All maintenance is now complete

– Network Engineering

Phone Switch Maintenance

We will be doing maintenance on our phone switch tonight at midnight. This work should not be service impacting.

 

~ Network Engineering

 

Update: 12:30am Maintenance is complete.

Core Router Maintenance

Update (3:40AM): Maintenance complete.

Beginning at 11:00PM on Monday December 11 we will be performing maintenance on some core routing infrastructure. This equipment primarly services legacy DSL customers in Northern California. There is no expected downtime from this operation, however customers may notice some routing changes during the maintenance window.

– Network Engineering

System Maintenance

Update: Maintenance complete.

 

Starting at 11:59pm this evening, SOC will be running updates on multiple customer facing systems. The following services may experience brief outages during the maintenance window:

  • Webmail
  • Customer Databases
  • Faxline
  • Member Tools
  • Forums
  • Sonic.com
  • Wiki
  • IPv6 Tunnel service
  • POP3/SMTP/IMAP

This post will be updated when the maintenance is complete.

-SOC