Category: Uncategorized

Tonight, May 16, starting at 11:59pm, we will be performing non-intrusive maintenance on core routing equipment serving our telecom network in northern California.

Update(1:16am): Maintenance complete

-Tomoc

At Midnight, tonight, we’re going to replace the motherboard in one of the clustered NFS mail store appliances with an on-site spare.  Unfortunately, due to the failures observed, the cluster partner is currently unable to take over duties and so this will be a service impacting event.  While we are replacing the motherboard POP, IMAP and Webmail services will be unavailable.  Outbound email will not be affected, all inbound mail will be queued for delivery.  Total downtime should between one to two hours.

-Kelsey, William and Joe.

Update:  Mon May 16 01:50:21 PDT 2016, the motherboard has been replaced and all systems appear to be operating normally.

We are currently investigating a potential issue with routing VOIP traffic through our network. At this time our engineers are working to isolate the cause of the issue. There is no ETR at this time. We will update this post as soon as we have more details.

-Sean

(Update 3:40) All effected services are now restored. If you are still experiencing any issues with outbound calls please contact support.

Tonight, May 12, starting at 11:59pm, we will be performing non-intrusive maintenance on core routing equipment serving our telecom network in northern California.

Update: Maintenance postponed until further notice

-Tomoc

Update: This maintenance is complete.

Tonight, Monday 2016 05 09, at 10pm we will be performing system maintenance on equipment in our data center at 2260 Apollo Way in Santa Rosa as well as in a remote data center in San Jose.  Part of this maintenance will require us to take the ovpn server offline, likely for less than 15 minutes.  The maintenance should be completed within 1 hour.

• Dexter K – System Operations

Update (12:20AM): This maintenance is now complete.

Beginning tonight at midnight, we will be performing maintenance on equipment serving Fusion and FlexLink customers in San Ramon, Livermore, and Moraga. Expected downtime is about 15 minutes.

-Michael

Update (12:17AM): This maintenance is now complete.

Beginning tonight at midnight, we will be performing maintenance on equipment serving Fusion and FlexLink customers in Danville. Expected downtime is about 20 minutes.

-Michael

We are currently experiencing an outage in the San Francisco area. At this time we are working to isolate and address the cause of the issue to restore services as quickly as possible. We do not have an ETR at this time but will provide updates as they develop.

-Sean

(Update 12:06pm) Attempts to restore service remotely to the central office which went down have been unsuccessful. We currently have a tech in route to investigate. ETA is 1 hour, we believe we should have services restores shortly after.

(Update 1:53PM) Our technician was able to isolate the issue to two pieces of gear that had failed in the central office. One piece of equipment has been restored and a subset of customers served out of this data center should be back online. The second piece of gear will need to be replaced. There is currently no ETR on the replacement of the failed equipment at this time.

-Sean

(Update 3:08pm) The second piece of gear has been replaced. Our technician is the process of upgrading code and deploying configuration.

-Mike T.

(Update 5:00pm) The equipment is all fully online. We are scrubbing through the customer configuration to make sure all configuration is back in place. If you are still experiencing an outage, please contact your support line.

-Mike T.

(Update 5:41pm) All services restored and back online. For lingering issues please report immediately to NOC.

-Mike T.

Tonight, 5/3/2016 starting at 11:59 we will be preforming maintenance on a small subset of Fusion Fiber customers in the San Francisco Area.  Expected downtime is 10 minutes.

-Brandon

Update: 12:01am – Maintenance is now complete

-Brandon

Early this morning, we updated firmware on equipment serving some San Francisco Fusion Fiber subscribers. This new firmware has proven unstable and has caused intermittent service interruptions throughout the day. We have been working with our vendor to resolve the issue as quickly as possible and ensure it does not continue in the future.

-Tomoc and Brandon

Update (10:00pm)  We have isolated and fixed the issue, service should now be restored/stable.

-Brandon