This afternoon, at approximately 11:59AM, a network misconfiguration caused a brief outage affecting all FlexLink and Fusion customers in the Pacific Heights area of San Francisco. The cause was isolated and fixed, and service was restored by approximately 12:10PM. We are investigating the cause of the misconfiguration further so that we can prevent it from occurring again in the future.
-Tim and Jared
Tonight ( 29, August ) at 11:00 PM we will be performing a set of software upgrades across our NetApp Filers.
These Filers store customers’ e-mail, web files, and other data.
There will be no impact on customers ability to access their information during this maintenance, and we expect it to take no more than an hour to complete.
–Augie
Update: Maintenance completed at 12:30 AM Tuesday morning. Not all of the software updates were applied during the maintenance window, another maintenance window is scheduled for tonight ( 30, August ) at 11:00 PM to complete the software updates.
Update: 31, August The final round of software upgrades will take place tonight at 11:00 PM.
Update: 1, September Maintenance continues tonight at 11:00 PM as last night’s upgrade did not proceed as planned, tonight will hopefully be the final round of upgrades, again there will be no customer impact during the maintenance. –Augie
FINAL UPDATE: 2, September 00:01 Maintenance complete all NetApp filers have been updated.
Tonight, August 25 at 11:00 PM we will be performing minor hardware maintenance on one of our ATM switches that serves LATA1 (SF Bay Area) to increase capacity. This operation should not cause any impact to in-service customers, and is expected to take 5-10 minutes.
-Jared
Update: The maintenance has been completed. The work was not started until after 12 PM.
Starting Aug 17th at 3:47PM one of the servers in the voicemail cluster stopped sending notices to the voice switch to update the message waiting indicator status. This lead to the MWI (the stutter dial-tone and light that flashes on your phone to let you know you have new voicemail) being stuck either on or off for the affected users. The lapse in monitoring that let this go unnoticed has been corrected. -Kelsey
A small number of users with “functional but incorrect” custom procmail rules may have experienced issues with their inbound mail delivery starting yesterday afternoon after the firewalls on the mail servers that handle message delivery were updated to be more secure. While the new firewall configuration was technically correct, it exposed a previously unnoticed bug with the version of software running on one of our NetApp filer clusters and prevented one of the mail servers from obtaining file locks correctly. An explanation of the incorrect procmail syntax is in this forum post. -Kelsey
Tonight, Thursday August 18 at 12:01 AM we will be moving some DSL subscribers from a congested DSL aggregation router to a new router to relieve the congestion. Affected customers will lose connectivity for 5-10 minutes.
-Jared
Update: The subscriber move is completed and all affected customers should be back online.
Tonight, August 17 at 12:01 AM we will be inserting new cards into one of our ATM switches that serves LATA1 (SF Bay Area) to increase capacity. This operation should not cause any impact to in-service customers, and is expected to take 5-10 minutes.
-Jared, Nathan and Matt
All Fusion Voice customers can now enable Call Forwarding directly from their phones as well as in the updated Voice Options membertool. In addition to being able to control call forwarding, the new Voice Options membertool can also manage Voicemail settings, disable Call Waiting, enable Caller ID Blocking, and enable international long distance dialing. Please note, while planned, *72 and *73 do not work at this time. 707-583-7702 and 707-583-7703 can be used in their place. For more information check out the forums. -Kelsey and Chris
At approximately 10:50 AM this morning, some routine router configuration caused an unexpected loss of connectivity for some of our POPs surrounding the Santa Rosa area. Service was restored at 10:55 AM. We are monitoring the situation, and looking into why innocuous configuration would cause this outage.
-Jared