Fusion Fiber Voice Issues – San Francisco Mission District

The bulk of voice service was restored at 5:15 PM when an emergency repair was engineered by our team.  It was determined additional manual intervention was necessary to restore voice for a small subset of customers, which we expect to be complete by 7:00PM.

 

At approximately 3:15 today, our support began getting reports of Fusion Fiber customers in the Mission reporting voice services down.  We are working to restore connectivity as quickly as possible and will update as more details become available.

-Network Engineering

System Maintenance

Update: Maintenance complete.

This evening at 11:59pm we will be performing updates to several systems including custom hosting servers, openVPN, webmail, etc.  Individual system downtime will be brief and we expect to complete the maintenance within 2 hours.

Fusion Fiber Maintenance – Richmond District San Francisco

Update: Maintenance was completed at 3:30AM 10/12/2018.

 

Thursday (10/11/18) night at 11:59PM we will performing intrusive maintenance on a subset of Fusion Fiber customers located in  San Francisco’s Richmond District.  Approximate expected customer downtime is 30 minutes. An update to this MOTD will be provided when all customer connectivity has been restored.

-Network Engineering

 

Santa Rosa Business Fiber Outage

Update 9:30am – Services should be restored at this time. If you are still having any issues with your service please contact our Network Operation team.

We are currently investigating a reachability issue to certain pieces of networking equipment which serves many of our Santa Rosa Northpoint Business fiber customers as well as some customers with transport into our colocation facility. There is currently no ETR but an update will be posted as soon as we have more information from our engineering team.

System Maintenance

Update: Maintenance is complete.

This evening at 11:59pm we will be performing maintenance on multiple customer facing systems, including customer databases, webmail, and the openVPN server.  Individual system downtime should be brief, and we expect to complete the maintenance within 2 hours.

System Maintenance

Update: Maintenance complete.

This evening at 11:59pm the OpenVPN service at ovpn.sonic.net will be briefly interrupted for maintenance. Downtime is expected to last only a few minutes, and afterwards users may need to manually reconnect.

Small outage in San Francisco

As of 1:30pm 9/24 the issue should be clear. If you are still having trouble please give our Support folks a call.

 

9am 9/24

We are having trouble with some of our FUSION DSL equipment in San Francisco. Our engineering team is working hard on repairs. We apologize for any inconvenience.

Legacy DSL Maintenance

Update: Maintenance complete.

Tomorrow night, September 23rd starting at 11:59PM, we will be performing intrusive maintenance on equipment serving legacy DSL customers. Our maintenance window will be for one hour until 1:00AM, but actual expected customer downtime is only expected to be 10-15 minutes.

-Network Engineering

IMAP Server Issues post Update/Rollback

Update 22:56:37 PDT: Everything looks good and all services appear to be functioning.

Update 21:37:03 PDT: The final cohort has been enabled.  We’ll continue to watch the cluster for next several hours to ensure that everything is working as intended.  It will still take some time before we’ve regained expected performance.

Update 19:17:35 PDT: Approximately 5/6 of user’s access to mail spools has been restored.  We’re still on target for having services fully restored in a few hours.

Update 17:39:57 PDT: Approximately 2/3 of user’s access to mail spools has been restored and while users can expect slow performance, the cluster and storage pools appear to be stable and chugging along.  At this point, we expect to have services fully restored later tonight.  Please note that no email was lost during this outage and our inbound servers were able to deliver mail without issue.

Update 16:19:00 PDT: The status of our email system is continuing to improve – we will post an update as our systems return to normal operation.

Update 13:00:56 PDT: In order to allow the cluster to recover we’ve taken steps to reduce the total number of user who can log in at once.  This appears to be having positive results and is allowing the storage pools to catch up with the load.  We still don’t have an ETA but the situation is improving.

Update: We believe the underlying issues have been been resolved.  However, it will take some time for IMAP and POP services to return to normal.  The fix has resulted in substantially increased IO load on our mail spool storage as all mailbox indexes have to be rebuilt.  This rebuild process can take a long time to complete, especially for large mailboxes, and it will be some time before the process is complete for all active mailboxes.

Update: We’re continuing to restore services for effected users.  Please note this may also affect some Fusion DSL user’s voicemail services.

Update: mailboxes for many users have been fixed and are working properly.  We’re continuing to repair remaining effected users.

Unfortunately, the mail server upgrade and rollback has caused issues for our IMAP cluster and it looks like most IMAP sessions are failing.  We’re working to restore services ASAP and will have more information forthcoming.  -Kelsey and Grant.