We are currently performing emergency maintenance on equipment serving Legacy DSL customers in the San Diego area. We expect 5 to 10 minutes of downtime while the work is performed.
-Tim and Jared
Update: The intrusive maintenance has been completed, and all affected customers should be back online. The trouble found was a failing ATM port in our ATM core that was causing degraded performance for all customers in the San Diego area.
Due to a recent database upgrade the nightly graymail reports were not being delivered to users. We have corrected the issue and are now processing and delivering reports to users. We apologize for any inconvenience this may have caused. -William
Tonight, September 7 at 12:01 AM we will be performing several non-intrusive network maintenance tasks on our Telecom backbone. The work should be completed by 1AM, and no impact is expected to customer traffic.
-Jared and Nathan
The primary server in the NNRP cluster, news.sonic.net, lost power briefly and was rebooted. It’s currently checking its filesystems and should be back in service shortly but news.sonic.net is unavailable in the meantime.
This afternoon, at approximately 11:59AM, a network misconfiguration caused a brief outage affecting all FlexLink and Fusion customers in the Pacific Heights area of San Francisco. The cause was isolated and fixed, and service was restored by approximately 12:10PM. We are investigating the cause of the misconfiguration further so that we can prevent it from occurring again in the future.
-Tim and Jared
Tonight ( 29, August ) at 11:00 PM we will be performing a set of software upgrades across our NetApp Filers.
These Filers store customers’ e-mail, web files, and other data.
There will be no impact on customers ability to access their information during this maintenance, and we expect it to take no more than an hour to complete.
–Augie
Update: Maintenance completed at 12:30 AM Tuesday morning. Not all of the software updates were applied during the maintenance window, another maintenance window is scheduled for tonight ( 30, August ) at 11:00 PM to complete the software updates.
Update: 31, August The final round of software upgrades will take place tonight at 11:00 PM.
Update: 1, September Maintenance continues tonight at 11:00 PM as last night’s upgrade did not proceed as planned, tonight will hopefully be the final round of upgrades, again there will be no customer impact during the maintenance. –Augie
FINAL UPDATE: 2, September 00:01 Maintenance complete all NetApp filers have been updated.
Tonight, August 25 at 11:00 PM we will be performing minor hardware maintenance on one of our ATM switches that serves LATA1 (SF Bay Area) to increase capacity. This operation should not cause any impact to in-service customers, and is expected to take 5-10 minutes.
-Jared
Update: The maintenance has been completed. The work was not started until after 12 PM.
Starting Aug 17th at 3:47PM one of the servers in the voicemail cluster stopped sending notices to the voice switch to update the message waiting indicator status. This lead to the MWI (the stutter dial-tone and light that flashes on your phone to let you know you have new voicemail) being stuck either on or off for the affected users. The lapse in monitoring that let this go unnoticed has been corrected. -Kelsey
A small number of users with “functional but incorrect” custom procmail rules may have experienced issues with their inbound mail delivery starting yesterday afternoon after the firewalls on the mail servers that handle message delivery were updated to be more secure. While the new firewall configuration was technically correct, it exposed a previously unnoticed bug with the version of software running on one of our NetApp filer clusters and prevented one of the mail servers from obtaining file locks correctly. An explanation of the incorrect procmail syntax is in this forum post. -Kelsey
Tonight, Thursday August 18 at 12:01 AM we will be moving some DSL subscribers from a congested DSL aggregation router to a new router to relieve the congestion. Affected customers will lose connectivity for 5-10 minutes.
-Jared
Update: The subscriber move is completed and all affected customers should be back online.