Emergency Router Maintenace

Today, September 17, at 5:22PM, a blade in one of our core routers stopped passing traffic. All traffic immediately switched to redundant links, however Fusion/FlexLink voice customers may have experienced a dropped call.

– Tomoc, Tim, Robbie and Nathan

Update: We will be performing emergency maintenance on one of our core routers tonight, September 17th, at midnight. We do not expect any customer impact as a result of this operation.

Update 2: We have replaced the suspected failing blade. There was no impact to customer voice calls.

Web Cluster instability.

UPDATE: We believe to have service restored to normal at this time. We are continuing to evaluate service and equipment.

We are currently seeing instability in our web cluster, we are investigating and hope to have the problem resolved shortly.

–SOC

Legacy DSL Maintenance

Tonight, September 13th, starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Update: The maintenance has been completed and all affected customers are back online.

Various system updates tonight

Tonight, 9/12/20012, starting at 11:00 PM, we will be pushing software and OS updates out to several system. These updates and associated reboots could cause intermittent issues using various member tools and other internal systems.  No connectivity or voice services will be affected.

– William and the SOC

Fusion and FlexLink Long Distance Issue

We have identified an issue affecting a number of long distance and toll-free phone calls through our network this morning. Users would have noticed delayed ringing or busy signals on affected calls. We have routed around the affected trunks and calls should be completing normally now.

-Brian and the NOC

Brief webmail outage

This morning at approximately 10:00 AM a small number of customers may have experienced issues when connecting to our webmail cluster. One of the servers in our webmail cluster started experiencing issues which caused connections to be slower then normal and may have led to browser timeouts. Since connections to the problematic server were slow and not completely failing, our load balancer thought the server was still running correctly and didn’t pull it out of rotation for live traffic. Coincidentally the server in question was scheduled to be replaced today. We are also going to review the configuration of our load balancer so we can prevent problems like this from happening in the future.  -William and the SOC

 

Update:

A small number of customers may have experienced trouble reaching our webmail services from approximately 2:30pm to 3:30pm. This was due to a configuration mismatch between old and new servers. System Operations has performed the necessary fixes to ensure better stability and capacity for the future.

— Joe and the SOC

Legacy DSL Maintenance

Tonight, September 10th, we will be performing maintenance on equipment serving legacy dsl customers in northern California. This maintenance window will be from 11:00pm – 2:00am. Expected downtime for affected customers is less than 15 minutes.

– Robbie and Tomoc

 

Update: Service has been restored to all affected customers as of 1:38AM.

Legacy DSL Outage

At 10:00pm, a port on one of our ATM aggregation switches failed, taking a small subset of our legacy DSL customers in the bay area offline. We are working on rerouting this traffic and bringing these customers back online.

-Tim and Tomoc

Update: Service has been restored to affected customers as of 10:32PM.

Legacy DSL Maintenance

Tonight, August 27th starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Router Maintenance

At 11:59pm on Saturday, August 25, we will be performing maintenance on equipment serving Fusion and Flexlink customers in the San Leandro area. Estimated down time is less than 20 minutes.

-Tomoc