Month: March 2015

Tonight, March 13, at 9:39pm, a core router in the northern California telecom network stopped responding. We are working to restore service and determine why traffic is not routing around.

Update(11:36pm): We have isolated and routed around the responsible equipment. The equipment in question was silently black-holing a large portion of traffic flowing through it. Customers network wide may have experienced degraded or interrupted service. All affected services appear to be restored at this point and we will be working with our hardware vendor to ensure this does not happen again.

-NOC

Today, March 13, at 11:00am, circuits serving the Sunnyvale central office went down. We are working to restore service as fast as possible, and will provide updates as they are received.

Update(2:33pm): AT&T has confirmed there is a fiber cut in Sunnyvale, no ETR as of yet.

Update(4:55pm): AT&T has a construction crew on site, and has given an initial ETR of 14 hours from now (7am).

Update(11:16am): Service restored as of 2:48am.

-Tomoc

3/13/15 – 1:55AM PDT – This maintenance is now complete.

Tonight at 11:59PM PDT we will be performing a network-wide software upgrade of roughly 150 cards serving Fusion customers throughout our network. The expected outage from these cards is roughly 5-10 minutes as the cards reboot with the new software.

-Tim J.

On March 16th, starting at noon, one of vendors will begin working to replace all of the batteries in one of the three UPSes in Santa Rosa.  This particular UPS is configured in such a way that we will be able to replace the three active battery chains individually without affecting the runtime or sacrificing internal redundancy of the UPS.  As such, this planned and fully scripted maintenance is expected to have no impact on any services.  -Kelsey and Russ

This has been rescheduled for Monday night 3/9/15 at 11:59PM PDT.

Tomorrow night (Saturday 3/7/2015) starting at 11:59PM we will be performing intrusive maintenance on cards serving a small portion of Fusion customers in Santa Rosa, Petaluma, Sebastopol, Cotati, Rohnert Park and Healdsburg. This maintenance should take ~10-15 minutes per card.

Update: All maintenance is now complete. Affected subscribers may need to power cycle their modem and/or router.

-Tim

3/6/15 – 2:15 AM PST – This maintenance is now complete.

Tonight, March 5, starting at 11:59pm, we will be performing software updates on equipment serving a subset of customers in Danville, Livermore, and Pleasanton. Expected downtime is 10 minutes.

-Tomoc and Tim

3/6/15 – 1:22AM PST – This maintenance is now complete.

Tonight, March 5, starting at 11:59pm, we will be performing a maintenance reload on core networking equipment in northern California. This equipment is fully redundant, and no downtime is expected.

-Tomoc and Tim

3/6/15 – 1:08AM – This maintenance is now complete.

Beginning tonight at 11:59PM PST we will be performing maintenance on equipment that serves a portion of Fusion and FlexLink customers in San Francisco. Downtime could be as long as one hour for this operation.

– Robbie and Tim

Today, starting at 9:00am, a port on equipment serving a small subset of Legacy DSL subscribers starting taking a large number of errors. We are working to get the hardware replaced as soon as possible.

Update: A workaround has been put in place to restore service. Affected subscribers should now be online, however there may be a brief (~2 minute) window of downtime when the permanent fix is implemented.

Update: Service restored.

-Tomoc