Update 22:56:37 PDT: Everything looks good and all services appear to be functioning.
Update 21:37:03 PDT: The final cohort has been enabled. We’ll continue to watch the cluster for next several hours to ensure that everything is working as intended. It will still take some time before we’ve regained expected performance.
Update 19:17:35 PDT: Approximately 5/6 of user’s access to mail spools has been restored. We’re still on target for having services fully restored in a few hours.
Update 17:39:57 PDT: Approximately 2/3 of user’s access to mail spools has been restored and while users can expect slow performance, the cluster and storage pools appear to be stable and chugging along. At this point, we expect to have services fully restored later tonight. Please note that no email was lost during this outage and our inbound servers were able to deliver mail without issue.
Update 16:19:00 PDT: The status of our email system is continuing to improve – we will post an update as our systems return to normal operation.
Update 13:00:56 PDT: In order to allow the cluster to recover we’ve taken steps to reduce the total number of user who can log in at once. This appears to be having positive results and is allowing the storage pools to catch up with the load. We still don’t have an ETA but the situation is improving.
Update: We believe the underlying issues have been been resolved. However, it will take some time for IMAP and POP services to return to normal. The fix has resulted in substantially increased IO load on our mail spool storage as all mailbox indexes have to be rebuilt. This rebuild process can take a long time to complete, especially for large mailboxes, and it will be some time before the process is complete for all active mailboxes.
Update: We’re continuing to restore services for effected users. Please note this may also affect some Fusion DSL user’s voicemail services.
Update: mailboxes for many users have been fixed and are working properly. We’re continuing to repair remaining effected users.
Unfortunately, the mail server upgrade and rollback has caused issues for our IMAP cluster and it looks like most IMAP sessions are failing. We’re working to restore services ASAP and will have more information forthcoming. -Kelsey and Grant.
It’s ok folks. Put down your devices, disconnect, and take a breather. Take the day off. You deserve it. Sh*t happens in tech from time to time. It’s going to be ok. 99.9% uptime is still pretty good.
None here either in AZ. When is it coming back?
Would it help if we got y’all working on this mega-problem some pizza? Because I’m thinking we could totally crowdfund for some pizza.
Hi, It’s now 5 p.m. How’s it going?
time 1716, still no access to webmail or POP
ETA update please as to when will access be restored, and why was there apparently no hot fallover plan to keep a critical service such as email up and running? Time-critical issues are being adversely impacted.
Have been a very happy customer – and recommending sonic – for decades, but at moment this is a patience tester. Zero reason a critical communications/infrastructure system having a software update should not have had a fallback mirror. Does not inspire confidence.
In my case, you may have contributed to the death of a 14-year civil rights case now in US Supreme Court.. Thanks.
@Dane: ever since the rebranding/expansion, there has been a noticeable change in sonic QoS. Is your team committed to provision of services such as email and VPN, or more toward branding/marketing/expansion purely for access?
You are marketing heavily moving into neighborhoods such as mine in Berkeley. It could be helpful to understand your thoughts regarding the current situation, and what its handling portends for the future.
chuck
Please……
I’ve had no email since 9:20 this morning. This is beginning to get trying. My Inbox has 4 messages, not several hundred. And we are in the midst of a bathroom remodel. Not good. My wife is planning a trip in a week. Not good. Any word at all would be helpful.
Please explain how my husband, whose Sonic email is on the same account, received in the same manner and at the same physical and IP address as mine is coming into his Apple POP account on his computer, but mine is not accessible on my laptop or iPhone (in addition, of course, to your webmail server). Puzzling. Thanks.
I too have a been a long time happy customer — until today! I have about a dozen projects, each with multiple emails that were deliberately left open in my multiple browsers and tabs (some projects had more than twenty received and sent emails open from the last six months) so that I could refer to them as notes of my conversations — many were chain conversations that I am required to respond to (but not until weeks later!)
Nonetheless, today’s email interruption has caused most of the open email tabs to lose their contents! I now need to go back through hundreds of emails to TRY to recover my position. My memory is not good enough to possibly come close.
I will now be APOLOGIZING to my client and colleagues for not completing many required tasks in the next weeks and months(?) due to this loss of email. service! This is NOT ACCEPTABLE!!
Not to mention the disruption of most of what I was trying to accomplish today because I could not read or respond to my email. I am expecting a response. Or I will find another ISP.
While my desktop continues with the problem, my android phone app receives with no problem.
Northbay customer — no incoming or outgoing emails for 9 hours! This urgent — my work.
Like others, I’ve been without email since mid-morning. Was in the midst of finalizing arrangement for several freelance assignments, and now those balls are very much dropped. The oddest thing is that in my Mac Mail, my sonic inbox has been duplicated, and now I have two. What’s that about?
time 1826
webmail login page appears but login with correct credentials fails, with error message “Login failed.”
SquirrelMail login with correct credentials also fails.
ETA update please as to when will access be restored
@Dane: ?
Wow. If you guys actually offered static IPs and reverse DNS with the fiber offering I wouldn’t have to rely on your overloaded servers. Muchas gracias.
time 1942
webmail login page appears but login with correct credentials fails, with error message “Login failed.â€
SquirrelMail login with correct credentials also fails.
POP access fails with error message “temporarily unavailable.”
ETA update please as to when will access be restored
@Dane: ?
Update, please. It’s been 2 hours since you reported that 5/6ths of your users’ accounts have been restored. I’m one of the unlucky ones and becoming concerned. Additional reassurance would help me sleep better tonight. Everyone else in my family now has email access, but I am the one who manages all family and work management, and one of our family members is very ill. Thanks.
Updates coming soon Linda, in the meantime your account has been shifted into the active pool.
Almost 12 hours and no email. Can you switch me to the active spool please?
Thank you! Happy to be able to continue to recommend Sonic to my library’s patrons as AWESOME! Get a good night’s sleep. I know I now will.
You’re welcome Linda, I appreciate the support. We’re on close to our 16 hour already and will definitely get a good nights sleep.
For various reasons, I sometimes keep messages in my IMAP folders that are marked as “deleted” without expunging them (as one can do with Pine/Alpine, for example). One consequence of this transition and rollback is that the deleted messages in my several IMAP folders were all expunged when I did not want them to be.