UPDATE: work is complete.

Server maintenance tonight starting at 11:59pm. The following services may experience interruptions:

• Mail
• Customer hosted sites
• IPv6 Tunnel services

The maintenance window is one hour, but interruptions to individual services will be brief.

-SOC

Customer support including Billing and Sales will be closed today for an extra half hour for our weekly meeting.

In order to provide better coverage during our busiest hours, we will be changing our telephone support hours to 8am-10pm, seven days a week.

We look forward to serving you!

-Eli Caul

This morning, March 14, at 12:00am, fiber circuits serving a hub site in Los Angeles went down. We are working to restore service as quickly as possible and will provide updates as they become available. Affected areas include Burbank, Canoga Park, North Hollywood, Northridge, Reseda, Sherman Oaks, and Sunnyvale.

Update(1:46am): We have routed traffic through an intermediary office until the affected fiber circuits are fixed. Service should be restored at this time.

Update(11:12am): Fiber circuits are fully restored as of 5:15am.

-Tomoc and Tim

Tonight, March 13, at 9:39pm, a core router in the northern California telecom network stopped responding. We are working to restore service and determine why traffic is not routing around.

Update(11:36pm): We have isolated and routed around the responsible equipment. The equipment in question was silently black-holing a large portion of traffic flowing through it. Customers network wide may have experienced degraded or interrupted service. All affected services appear to be restored at this point and we will be working with our hardware vendor to ensure this does not happen again.

-NOC

Today, March 13, at 11:00am, circuits serving the Sunnyvale central office went down. We are working to restore service as fast as possible, and will provide updates as they are received.

Update(2:33pm): AT&T has confirmed there is a fiber cut in Sunnyvale, no ETR as of yet.

Update(4:55pm): AT&T has a construction crew on site, and has given an initial ETR of 14 hours from now (7am).

Update(11:16am): Service restored as of 2:48am.

-Tomoc

3/13/15 – 1:55AM PDT – This maintenance is now complete.

Tonight at 11:59PM PDT we will be performing a network-wide software upgrade of roughly 150 cards serving Fusion customers throughout our network. The expected outage from these cards is roughly 5-10 minutes as the cards reboot with the new software.

-Tim J.

On March 16th, starting at noon, one of vendors will begin working to replace all of the batteries in one of the three UPSes in Santa Rosa.  This particular UPS is configured in such a way that we will be able to replace the three active battery chains individually without affecting the runtime or sacrificing internal redundancy of the UPS.  As such, this planned and fully scripted maintenance is expected to have no impact on any services.  -Kelsey and Russ