Server Maintenance

UPDATE: work is complete.

Server maintenance tonight starting at 11:59pm. The following services may experience interruptions:

  • Mail
  • Customer hosted sites
  • IPv6 Tunnel services

The maintenance window is one hour, but interruptions to individual services will be brief.

 

-SOC

Los Angeles Fusion/FlexLink Outage

This morning, March 14, at 12:00am, fiber circuits serving a hub site in Los Angeles went down. We are working to restore service as quickly as possible and will provide updates as they become available. Affected areas include Burbank, Canoga Park, North Hollywood, Northridge, Reseda, Sherman Oaks, and Sunnyvale.

Update(1:46am): We have routed traffic through an intermediary office until the affected fiber circuits are fixed. Service should be restored at this time.

Update(11:12am): Fiber circuits are fully restored as of 5:15am.

-Tomoc and Tim

Sonic Telecom Outage

Tonight, March 13, at 9:39pm, a core router in the northern California telecom network stopped responding. We are working to restore service and determine why traffic is not routing around.

Update(11:36pm): We have isolated and routed around the responsible equipment. The equipment in question was silently black-holing a large portion of traffic flowing through it. Customers network wide may have experienced degraded or interrupted service. All affected services appear to be restored at this point and we will be working with our hardware vendor to ensure this does not happen again.

-NOC

Fusion/FlexLink Outage in Sunnyvale

Today, March 13, at 11:00am, circuits serving the Sunnyvale central office went down. We are working to restore service as fast as possible, and will provide updates as they are received.

Update(2:33pm): AT&T has confirmed there is a fiber cut in Sunnyvale, no ETR as of yet.

Update(4:55pm): AT&T has a construction crew on site, and has given an initial ETR of 14 hours from now (7am).

Update(11:16am): Service restored as of 2:48am.

-Tomoc

Fusion Intrusive Maintenance

3/13/15 – 1:55AM PDT – This maintenance is now complete.

Tonight at 11:59PM PDT we will be performing a network-wide software upgrade of roughly 150 cards serving Fusion customers throughout our network. The expected outage from these cards is roughly 5-10 minutes as the cards reboot with the new software.

-Tim J.

Fusion/FlexLink Intrusive Maintenance

Update(2:48AM): This maintenance is now complete.

Beginning tonight at midnight we will be performing maintenance on equipment that serves a portion of Fusion and FlexLink customers in Oakland. Downtime could be as long as one hour for this operation.

– Robbie and Tim

Santa Rosa Datacenter UPS Maintenance

On March 16th, starting at noon, one of vendors will begin working to replace all of the batteries in one of the three UPSes in Santa Rosa.  This particular UPS is configured in such a way that we will be able to replace the three active battery chains individually without affecting the runtime or sacrificing internal redundancy of the UPS.  As such, this planned and fully scripted maintenance is expected to have no impact on any services.  -Kelsey and Russ

Fusion Maintenance – Santa Rosa, Petaluma, Sebastopol, Cotati, Rohnert Park and Healdsburg

This has been rescheduled for Monday night 3/9/15 at 11:59PM PDT.

Tomorrow night (Saturday 3/7/2015) starting at 11:59PM we will be performing intrusive maintenance on cards serving a small portion of Fusion customers in Santa Rosa, Petaluma, Sebastopol, Cotati, Rohnert Park and Healdsburg. This maintenance should take ~10-15 minutes per card.

Update: All maintenance is now complete. Affected subscribers may need to power cycle their modem and/or router.

-Tim