Category: Uncategorized

Update 22:56:37 PDT: Everything looks good and all services appear to be functioning.

Update 21:37:03 PDT: The final cohort has been enabled.  We’ll continue to watch the cluster for next several hours to ensure that everything is working as intended.  It will still take some time before we’ve regained expected performance.

Update 19:17:35 PDT: Approximately 5/6 of user’s access to mail spools has been restored.  We’re still on target for having services fully restored in a few hours.

Update 17:39:57 PDT: Approximately 2/3 of user’s access to mail spools has been restored and while users can expect slow performance, the cluster and storage pools appear to be stable and chugging along.  At this point, we expect to have services fully restored later tonight.  Please note that no email was lost during this outage and our inbound servers were able to deliver mail without issue.

Update 16:19:00 PDT: The status of our email system is continuing to improve – we will post an update as our systems return to normal operation.

Update 13:00:56 PDT: In order to allow the cluster to recover we’ve taken steps to reduce the total number of user who can log in at once.  This appears to be having positive results and is allowing the storage pools to catch up with the load.  We still don’t have an ETA but the situation is improving.

Update: We believe the underlying issues have been been resolved.  However, it will take some time for IMAP and POP services to return to normal.  The fix has resulted in substantially increased IO load on our mail spool storage as all mailbox indexes have to be rebuilt.  This rebuild process can take a long time to complete, especially for large mailboxes, and it will be some time before the process is complete for all active mailboxes.

Update: We’re continuing to restore services for effected users.  Please note this may also affect some Fusion DSL user’s voicemail services.

Update: mailboxes for many users have been fixed and are working properly.  We’re continuing to repair remaining effected users.

Unfortunately, the mail server upgrade and rollback has caused issues for our IMAP cluster and it looks like most IMAP sessions are failing.  We’re working to restore services ASAP and will have more information forthcoming.  -Kelsey and Grant.

UPDATE: This upgrade has been postponed

Tonight at 11:59pm we will be upgrading our POP3 and IMAP servers to a new software release. There may be brief service interruptions while the updates are rolled out.

-SOC

Tonight, starting at 11:59 PM, we will be performing intrusive maintenance affecting a subset of Fusion DSL and FlexLink customers in San Francisco and the South Bay. Expected down time is approximately 30 minutes. The maintenance window is 6 hours.

Update3: Maintenance is done, but some equipment in San Francisco and Santa Cruz failed. We’re dispatching out to get the equipment replaced.

Update2: This maintenance will be done tonight(9/23/18) starting at 11:59pm. The maintenance window is 6 hours.

Update: This maintenance has been cancelled

– Network Engineering

Tonight, starting at 11:59 PM, we will be performing intrusive maintenance affecting a subset of Fusion DSL and FlexLink customers in the North Bay and East Bay. Expected down time is approximately 30 minutes. The maintenance window is 6 hours.

Update 5:59 AM: This maintenance is now complete.

– Network Engineering

Tonight, starting at 11:59 PM, we will be performing intrusive maintenance affecting a subset of Fusion and FlexLink customers in the East Bay. Expected down time is approximately 30 minutes. The maintenance window is 6 hours.

Update 5:55 AM: This maintenance is now complete.

– Network Engineering

We have received reports that the Sonic Speedtest site is throwing an error message. Ops is investigating and will update this post when the problem has been resolved.

Update: We have isolated the issue to an outage with an upstream provider and are working with them to resolve that outage. A final update will be provided once the issues are resolved.

Update: The issue was resolved overnight by our upstream provider. We apologize for any inconvenience.

Tonight, starting at 11:59 PM, we will be performing intrusive maintenance affecting a subset of Fusion and FlexLink customers in the North Bay and East Bay. Expected down time is approximately 30 minutes. The maintenance window is 6 hours.

Update 6.03 AM: This maintenance is now complete.

– Network Engineering

We are currently experiencing an issue affecting fiber customers located within the Jasper building in San Francisco. We have dispatched a technician to investigate on-site. We will update this message when we have more information.

Update (15:00): This issue has been resolved and service to customers has been restored. There was a power outage at the Jasper building. As soon as that outage was cleared Sonic services were restored.

– Andrew & Support

Tonight, starting at 11:59 PM, we will be performing intrusive maintenance affecting a subset of Fusion and FlexLink customers in the East Bay. Expected down time is approximately 30 minutes. The maintenance window will be from midnight to 4 AM.

Update 4:20 AM: The maintenance is now complete. A subset of customers in Albany and Oakland are down due to unforeseen issue’s. We are working to restore connectivity as quickly as possible.

– Network Engineering

Sonic’s SMS support system is currently undergoing an unplanned outage. We are investigating the problem and will update here with updates.

Update: SMS support services have been restored. We have identified the root cause and are taking steps to avoid such occurrences in the future.