Month: September 2012

Legacy DSL Outage

At 8:57PM a port on one of our ATM aggregation switches failed, taking a small subset of Bay Area legacy DSL customers offline. We are working to re-route these customers to a working port as soon as possible.

-Tim

Update: All customers have been re-routed and are back online.

Legacy DSL Maintenance

Tonight, September 27th, starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento and Bay areas. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Update: This maintenance window is being pushed to a future date

Server Infrastructure Upgrades

Tonight, September 26, 2012, starting at 11:59pm, we will be performing upgrades on some of the networking equipment in our Santa Rosa data center. These upgrades should not cause down time for any Sonic.net connections or services.

-Tomoc

Update: Maintenance completed successfully

Legacy DSL Maintenance

Tonight, September 18th, starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Emergency Router Maintenace

Today, September 17, at 5:22PM, a blade in one of our core routers stopped passing traffic. All traffic immediately switched to redundant links, however Fusion/FlexLink voice customers may have experienced a dropped call.

– Tomoc, Tim, Robbie and Nathan

Update: We will be performing emergency maintenance on one of our core routers tonight, September 17th, at midnight. We do not expect any customer impact as a result of this operation.

Update 2: We have replaced the suspected failing blade. There was no impact to customer voice calls.

Web Cluster instability.

UPDATE: We believe to have service restored to normal at this time. We are continuing to evaluate service and equipment.

We are currently seeing instability in our web cluster, we are investigating and hope to have the problem resolved shortly.

–SOC

Legacy DSL Maintenance

Tonight, September 13th, starting at 11:59PM, we will be performing maintenance on equipment serving legacy dsl customers in the Sacramento area. Expected downtime is less than 15 minutes.

– Robbie and Tomoc

Update: The maintenance has been completed and all affected customers are back online.

Various system updates tonight

Tonight, 9/12/20012, starting at 11:00 PM, we will be pushing software and OS updates out to several system. These updates and associated reboots could cause intermittent issues using various member tools and other internal systems.  No connectivity or voice services will be affected.

– William and the SOC

Fusion and FlexLink Long Distance Issue

We have identified an issue affecting a number of long distance and toll-free phone calls through our network this morning. Users would have noticed delayed ringing or busy signals on affected calls. We have routed around the affected trunks and calls should be completing normally now.

-Brian and the NOC

Brief webmail outage

This morning at approximately 10:00 AM a small number of customers may have experienced issues when connecting to our webmail cluster. One of the servers in our webmail cluster started experiencing issues which caused connections to be slower then normal and may have led to browser timeouts. Since connections to the problematic server were slow and not completely failing, our load balancer thought the server was still running correctly and didn’t pull it out of rotation for live traffic. Coincidentally the server in question was scheduled to be replaced today. We are also going to review the configuration of our load balancer so we can prevent problems like this from happening in the future.  -William and the SOC

 

Update:

A small number of customers may have experienced trouble reaching our webmail services from approximately 2:30pm to 3:30pm. This was due to a configuration mismatch between old and new servers. System Operations has performed the necessary fixes to ensure better stability and capacity for the future.

— Joe and the SOC