Month: August 2011

Brief FlexLink and Fusion outage

This afternoon, at approximately 11:59AM, a network misconfiguration caused a brief outage affecting all FlexLink and Fusion customers in the Pacific Heights area of San Francisco. The cause was isolated and fixed, and service was restored by approximately 12:10PM. We are investigating the cause of the misconfiguration further so that we can prevent it from occurring again in the future.

-Tim and Jared

Non-disruptive NetApp Filer Maintenance.

Tonight ( 29, August ) at 11:00 PM we will be performing a set of software upgrades across our NetApp Filers.

These Filers store customers’ e-mail, web files, and other data.

There will be no impact on customers ability to access their information during this maintenance, and we expect it to take no more than an hour to complete.

–Augie

Update: Maintenance completed at 12:30 AM Tuesday morning. Not all of the software updates were applied during the maintenance window, another maintenance window is scheduled for tonight ( 30, August ) at 11:00 PM to complete the software updates.

Update: 31, August The final round of software upgrades will take place tonight at 11:00 PM.

Update: 1, September Maintenance continues tonight at 11:00 PM as last night’s upgrade did not proceed as planned, tonight will hopefully be the final round of upgrades, again there will be no customer impact during the maintenance. –Augie

FINAL UPDATE: 2, September 00:01 Maintenance complete all NetApp filers have been updated.

Non-Intrusive ATM Switch Maintenance

Tonight, August 25 at 11:00 PM we will be performing minor hardware maintenance on one of our ATM switches that serves LATA1 (SF Bay Area) to increase capacity. This operation should not cause any impact to in-service customers, and is expected to take 5-10 minutes.

-Jared

Update: The maintenance has been completed. The work was not started until after 12 PM.

Fusion Voicemail Message Waiting Issues

Starting Aug 17th at 3:47PM one of the servers in the voicemail cluster stopped sending notices to the voice switch to update the message waiting indicator status.  This lead to the MWI (the stutter dial-tone and light that flashes on your phone to let you know you have new voicemail) being stuck either on or off for the affected users.  The lapse in monitoring that let this go unnoticed has been corrected.  -Kelsey

NFS File Locking Bug and Mail Delivery

A small number of users with “functional but incorrect” custom procmail rules may have experienced issues with their inbound mail delivery starting yesterday afternoon after the firewalls on the mail servers that handle message delivery were updated to be more secure.  While the new firewall configuration was technically correct, it exposed a previously unnoticed bug with the version of software running on one of our NetApp filer clusters and prevented one of the mail servers from obtaining file locks correctly.  An explanation of the incorrect procmail syntax is in this forum post.  -Kelsey

Legacy DSL Subscriber Rebalancing

Tonight, Thursday August 18 at 12:01 AM we will be moving some DSL subscribers from a congested DSL aggregation router to a new router to relieve the congestion. Affected customers will lose connectivity for 5-10 minutes.

-Jared

Update: The subscriber move is completed and all affected customers should be back online.

Non-Intrusive ATM Switch Maintenance

Tonight, August 17 at 12:01 AM we will be inserting new cards into one of our ATM switches that serves LATA1 (SF Bay Area) to increase capacity. This operation should not cause any impact to in-service customers, and is expected to take 5-10 minutes.

-Jared, Nathan and Matt

Fusion Voice Feature Updates

All Fusion Voice customers can now enable Call Forwarding directly from their phones as well as in the updated Voice Options membertool.  In addition to being able to control call forwarding, the new Voice Options membertool can also manage Voicemail settings, disable Call Waiting, enable Caller ID Blocking, and enable international long distance dialing.  Please note, while planned, *72 and *73 do not work at this time. 707-583-7702 and 707-583-7703 can be used in their place.  For more information check out the forums.  -Kelsey and Chris

Brief Backbone Service Interruption

At approximately 10:50 AM this morning, some routine router configuration caused an unexpected loss of connectivity for some of our POPs surrounding the Santa Rosa area. Service was restored at 10:55 AM. We are monitoring the situation, and looking into why innocuous configuration would cause this outage.

-Jared