Month: May 2018

Palo Alto Fusion Service Issues

As of around 8:30PM this evening, we were alerted of an impacted piece of hardware serving fusion customers in Palo Alto.  After some time troubleshooting, we identified the issue but even with vendor assistance were unable to remotely perform a repair.  A technician is presently en route to replace the failed equipment.  Update will be provided on completion of service restoration.

Update(6:06am): A subset of customers continues to be affected after physical equipment replacement. We are continuing to work with our vendor to find the root cause and fully restore service.

Update(8:24am): All services have been restored and we are working to protect ourselves from a failure of this nature in the future.

-Network Engineering

Voice Issue

A subset of our customers are experiencing call quality issues affecting outbound dialing, and some inbound toll free calls. Our engineering team is actively working the issue.
There is no ETR for repair. The next update will be for any meaningful progress reported.

 

Update 2:21 PM PST – We have isolated the issue to a specific carrier and to mitigate the issue all voice traffic is being routed around to a different path. Once repairs are made we will relay another update.

San Francisco Flexlink Maintenance

Update: 2:27am – This maintenance is now complete.

Tonight, starting at 11:59PM, we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in San Francisco. This window is scheduled for 2 hours. During this time, circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.

– Network Operations

System Maintenance

UPDATE: Complete

 

Tonight at 11:59pm we will be performing upgrades to various systems. Any interruption to services will be brief, and we expect to complete the maintenance within 1 hour.

-Sonic System Operations

San Francisco Flexlink Maintenance

Update: 3:21am This maintenance is now complete.

Update: This window has been extended for 1 hour.

Tonight, 5/10/2018, starting at 11:59pm we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in San Francisco. This window is scheduled for 2 hours. During this time circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.

– Network Operations

Multiple system interruptions

Starting at 8am today, SOC became aware of an issue caused by automated system updates that resulted i n outages of several customer facing services. Credit card payment, account signup, fusion voicemail, and other voice service applications where affected during this outage. As of 10am today the issue has been resolved, though SOC is still looking to prevent further issues.

 

-SOC

San Francisco Flexlink Maintenance

Update: 3:27am This maintenance is now complete

Update: This window has been extended for 1 hour.

Tonight 5/8/2018 starting at 12:00am we will be performing maintenance on equipment that serves a subset of our Flexlink Ethernet customers based in San Francisco. This window is scheduled for 2 hours. During this time circuits may experience a brief interruption of service. This post will be updated once the maintenance is complete.

Phone Switch Maintenance

Tonight starting at 11:59 PM we are moving a component in our phone switch to a new power circuit for increased redundancy. We do not expect any service impact.

 

~ Network Engineering

 

Update: 12:40 am Maintenance is completed.

Santa Rosa Datacenter UPS Preventative Maintenance

Our Santa Rosa Datacenter UPS’ will be undergoing their annual preventative maintenance on Thursday May 10th, starting at 10:00 AM.  This is a fully scripted event with our vendor and should not result in any service interruptions.

Update:  This maintenance was postponed and rescheduled to June 6th starting at 10AM.

– Sonic Facilities and Systems

Small Fiber Outage in San Francisco

We are currently experiencing a small outage for a few of our Fusion Fiber customers in San Francisco. Our Engineers are working on the problem but we have no estimated time of repair as of yet.

 

*Update*

Outage has been cleared. If you are still having issues please contact Support.