Fusion Voicemail Message Delays

Voicemail messages on our fusion voices platform experienced a delay in delivery today between 9am and 2pm. This was due to a problem that arose during a hardware and software upgrade on the servers which host the service. We have rolled back our upgrades for the time being while we diagnose the problem further. All delayed messages have been delivered and the service is currently functioning normally.

Update: Further voicemail service interruptions have been narrowed down to issues with a text to speech module in the voicemail platform. Service should be stable.

Update: Unfortunately, a second issue with the new platform caused some voicemails to be queued and not delivered again over the weekend.  All of the queued messages have been delivered now.  While we investigate and resolve the underlying issue we’re also adding additional monitoring to ensure that should it happen again it doesn’t go unnoticed.  We’re sorry for any trouble this may have caused, prompt delivery of voicemail message is obviously very important.

-SOC

San Francisco Emergency Fiber Maintenance

Tonight at 11:59PM, we will be performing emergency maintenance on equipment serving a subset of San Francisco Fusion Fiber customers. Expected downtime is less than 1 hour. This maintenance window is 5 hours.

Update(2:47am): All maintenance is now complete

– Network Engineering

Phone Switch Maintenance

We will be doing maintenance on our phone switch tonight at midnight. This work should not be service impacting.

 

~ Network Engineering

 

Update: 12:30am Maintenance is complete.

Core Router Maintenance

Update (3:40AM): Maintenance complete.

Beginning at 11:00PM on Monday December 11 we will be performing maintenance on some core routing infrastructure. This equipment primarly services legacy DSL customers in Northern California. There is no expected downtime from this operation, however customers may notice some routing changes during the maintenance window.

– Network Engineering

System Maintenance

Update: Maintenance complete.

 

Starting at 11:59pm this evening, SOC will be running updates on multiple customer facing systems. The following services may experience brief outages during the maintenance window:

  • Webmail
  • Customer Databases
  • Faxline
  • Member Tools
  • Forums
  • Sonic.com
  • Wiki
  • IPv6 Tunnel service
  • POP3/SMTP/IMAP

This post will be updated when the maintenance is complete.

-SOC

Shell Server Maintenance

Beginning at midnight tonight, the server backing shell.sonic.net will be taken offline for a network upgrade. Downtime should be no more than a few minutes.

–System Operations

Core Router Maintenance

Beginning tonight at midnight we will be performing maintenance on core routing equipment that primarily serves legacy DSL customers. We do not expect any downtime as a result of this operation. Customers may notice brief routing interruptions and changes during the maintenance window.

  • Network Engineering

 

 

 

Brentwood – Fusion Fiber Intrusive Maintenance

Next Week, 12/4/2017 at midnight we will be upgrading equipment serving Fusion Fiber customers in the Brentwood area.  Expected customer downtime is around 30 minutes while the equipment upgrades and reboots.  The maintenance window will be 3 hours.

-Network Engineering

 

Update(1:11am) This maintenance is now complete

San Francisco – Fusion Fiber Intrusive Maintenance

Next week, 12/5/2017 through 12/7/2017 starting at 11:59pm, Sonic will be upgrading equipment serving Fusion Fiber customers throughout the San Francisco area.  Expected customer downtime is 30 minutes while the equipment reboots.  The maintenance window will be 5 hours for each night.

-Network Engineering

Fusion Fiber Outage – San Francisco

Update 6:09pm

At this point we believe we have restored service and contacted all affected customers. If you are still experiencing issues with your fiber service please contact our support team.

 

Update 11:22am

We have remotely restored service to most of the customers affected by this outage. If you believe you are still affected, please try rebooting the white ONT box. Support will be reaching out to assist with known affected customers shortly.

-Network Operations

 

We are currently investigating a Residential Fiber outage that affects a sub-set of users in San Francisco. We are actively working to resolve this outage and restore services as soon as possible.

 

-Network Operations