Network outage San Francisco

We are currently experiencing an outage in the San Francisco area. At this time we are working to isolate and address the cause of the issue to restore services as quickly as possible. We do not have an ETR at this time but will provide updates as they develop.

-Sean

(Update 12:06pm) Attempts to restore service remotely to the central office which went down have been unsuccessful. We currently have a tech in route to investigate. ETA is 1 hour, we believe we should have services restores shortly after.

(Update 1:53PM) Our technician was able to isolate the issue to two pieces of gear that had failed in the central office. One piece of equipment has been restored and a subset of customers served out of this data center should be back online. The second piece of gear will need to be replaced. There is currently no ETR on the replacement of the failed equipment at this time.

-Sean

(Update 3:08pm) The second piece of gear has been replaced. Our technician is the process of upgrading code and deploying configuration.

-Mike T.

(Update 5:00pm) The equipment is all fully online. We are scrubbing through the customer configuration to make sure all configuration is back in place. If you are still experiencing an outage, please contact your support line.

-Mike T.

(Update 5:41pm) All services restored and back online. For lingering issues please report immediately to NOC.

-Mike T.

Sonic Core Voice Maintenance

Tonight, 5/3/2016 at 11:59pm we will be upgrading a piece of equipment that is part of our core voice infrastructure.  There should be no impact to calls that are all ready in progress, but there may be a brief delay when completing a newly dialed call during the maintenance window.  Expected completion of the maintenance is 2 hours.

-Brandon

Update – Maintenance is being postponed until tomorrow night, 5/4/2016

-Brandon

Update – Maintenance will be pushed out one more day to 5/5/2016 at 11:59pm

-Brandon

Update – Maintenance is completed. 5/6/2016 at 12:15am

Brian

Fusion Fiber Maintenance – San Francisco

Tonight, 5/3/2016 starting at 11:59 we will be preforming maintenance on a small subset of Fusion Fiber customers in the San Francisco Area.  Expected downtime is 10 minutes.

-Brandon

Update: 12:01am – Maintenance is now complete

-Brandon

San Francisco Fusion Fiber Service Interruptions

Early this morning, we updated firmware on equipment serving some San Francisco Fusion Fiber subscribers. This new firmware has proven unstable and has caused intermittent service interruptions throughout the day. We have been working with our vendor to resolve the issue as quickly as possible and ensure it does not continue in the future.

 

-Tomoc and Brandon

 

Update (10:00pm)  We have isolated and fixed the issue, service should now be restored/stable.

-Brandon

Sebastopol Fusion Fiber Maintenance

Tonight, April 27, starting at 11:59pm, we will be performing software updates on equipment serving Fusion Fiber customers in Sebastopol.  Estimated time of completion is 3am.

Update(12:29am): Maintenance complete

-Tomoc