Update 22:56:37 PDT: Everything looks good and all services appear to be functioning.
Update 21:37:03 PDT: The final cohort has been enabled. We’ll continue to watch the cluster for next several hours to ensure that everything is working as intended. It will still take some time before we’ve regained expected performance.
Update 19:17:35 PDT: Approximately 5/6 of user’s access to mail spools has been restored. We’re still on target for having services fully restored in a few hours.
Update 17:39:57 PDT: Approximately 2/3 of user’s access to mail spools has been restored and while users can expect slow performance, the cluster and storage pools appear to be stable and chugging along. At this point, we expect to have services fully restored later tonight. Please note that no email was lost during this outage and our inbound servers were able to deliver mail without issue.
Update 16:19:00 PDT: The status of our email system is continuing to improve – we will post an update as our systems return to normal operation.
Update 13:00:56 PDT: In order to allow the cluster to recover we’ve taken steps to reduce the total number of user who can log in at once. This appears to be having positive results and is allowing the storage pools to catch up with the load. We still don’t have an ETA but the situation is improving.
Update: We believe the underlying issues have been been resolved. However, it will take some time for IMAP and POP services to return to normal. The fix has resulted in substantially increased IO load on our mail spool storage as all mailbox indexes have to be rebuilt. This rebuild process can take a long time to complete, especially for large mailboxes, and it will be some time before the process is complete for all active mailboxes.
Update: We’re continuing to restore services for effected users. Please note this may also affect some Fusion DSL user’s voicemail services.
Update: mailboxes for many users have been fixed and are working properly. We’re continuing to repair remaining effected users.
Unfortunately, the mail server upgrade and rollback has caused issues for our IMAP cluster and it looks like most IMAP sessions are failing. We’re working to restore services ASAP and will have more information forthcoming. -Kelsey and Grant.
As of 10:40AM 9/21, I am unable to access my email. Santa Clara.
As of 10:50 AM I cannot use my Email! How long will it be down? I use it for work and do not have it set up in Thunderbird or anything for that Email address which is different than my home, which is also with you guys.
note page with problem description is much appreciated.
Is it possible to have our new mail forwarded to a non-sonic address while we wait for the main mailbox indexes to be rebuilt?
as of 11:04 am 9/21 no email for us in Sunnyvale CA :^(
Haven’t been able to access email since 830am
Could you say this sentence? The effected are affected by inappropriate appropriateness?
What the ETR on this?
Thanks,
Patrick O’Brien
Seems my mail program invalidated it’s entire mailbox cache and wants to re-download every message now (11:10am 9/21), and is receiving them about one email a minute (with thousands left to go). So I took my account offline for now, but I really need to get it back ASAP. Hoping another notification can be sent when things are really back to normal.
11:20 AM, September 21, 2018, still unable to access email. Very frustrating.
I am unable to access e-mail Hidden Valley Lake
No email for us in Fort Wayne Indiana. I use it for work also. Couldn’t the upgrade be done on a weekend evening?
When you say: “This rebuild process can take a long time to complete, especially for large mailboxes, and it will be some time before the process is complete for all active mailboxes”, can you give us some definition for your terminology “long time” and “some time”? Are you expressing those terms as in hours? Or days? or What?
We understand that something like this can happen, but we REALLY need better info in order to deal with the issues your problem is causing us.
This is the first hiccup in many years for us. Thank you for your efforts in restoring service quickly, and your continued updates here.
Any chance of a vague ETA? “Today” / “By Monday” / etc?
Slowly repopulating my inbox at 11:25 in San Carlos.
Not very well this morning. :/
Michael, I’m sorry that I can’t provide a better ETR at this time but we should be talking about hours, not days.
Will, I’m sorry, that’s not the behavior I’d expect to see in this case and our server stats don’t show it to be a widespread issue. Everyone in the ops team is working on this and we’ll update the MOTD when we have more information.
Timestamps on your updates would be great so we knew when the latest update was and whether the most up to date was on top or bottom.
Hey, I’m just happy to find that this is not some virus that has destroyed my outlook .ost file, and that is fixable. I’ve been using Sonic’s email since 2001, and this is a first… That is a very good thing, keeping this system smooth for a couple decades (at least 17 years in my accounting, anyway). I’ve been on major corporate exchange servers with more issues than this. You guys rock.
Best of luck for a speedy recovery of normally great services. Thanks for the hard work.
For some reason Apple Mail started deleting my Sonic mail locally rather than simply realizing that IMAP was down. It must’ve rec’d erroneous information that all my mail was deleted. Very annoying. This will put more pressure on the servers when others like me have their clients re-download everything. Not a normal hiccup. Can’t think of a way to stop the client from doing this. Any suggestions from anyone to avoid this in the future?
Will the auto response function in “member tools” allow us to notify others who are trying to reach us that your system is temporarily down?
LOVE YOUR EXCELLENT RELIABLE SERVICE, CUSTOMER SERVICE REPS, REASONABLE RATES, AND YOUR TOTAL OPERATION!!!! This is the first hiccup I’ve experience in all my years with SONIC. As can be expected, you have your experts on top of the problem. Best of luck for a speedy fix and recovery. You guys rock!!!
12:22 PM PST and I’m still not able to access my email at all. Is that still expected at this time? The update sounds as if I should be able to connect, even if the download process is slow.
It is 12:25. How about an update?
It is now noon. Unable to access email since 8 a.m. 9/21
NOT able to access my webmail or POP3 mailboxes from Salem. OR
12:28 and no email in Berkeley.
12:31pm, in Rohnert Park webmail is still saying:
Sonic
We’ll be back as soon as possible.
Sorry for the inconvenience but we’re performing some maintenance at the moment.
— Sonic.com Operations
A couple of hours ago, ALL my email vanished from iPhone and computer. I THOUGHT I’D BEEN HACKED!
Just now, email is slowly beginning to re-load — starting at MARCH 2016, when I first signed up with Sonic.
It’s gonna be a long day…
Since you send out text Authorization Codes – can you send a text when email is back up? I am under deadline and what comes in today is vital for production. I am getting calls … did you get my email? My column? My article, etc. If I can know right away, it would help me get on track – thank you. Your box about notifying by email is not going to work right now. perhaps a box Notify me by TEXT would help. And offer a box for that cell # since I don’t want to include it in my COMMENT.
Somehow when you say things like the issue is not widespread or most users are unaffected that doesn’t make those of us who cannot access email at all feel any better. I hope for the sake of us who cannot access email you’re taking this issue seriously even if from your perspective we are few in number. In my opinion, this is a long time to wait for a fix.
My emails are missing and when I sent an email to someone it arrived truncated. This was from Apple Mail which is really having hard time. Well, I’ve been with sonic for 20 years and this is the first time you really messed up so I guess that’s good but right now I’m a little worried.
Sonic customer for over 20 years. This is incredibly rare. Thank you for your excellent service!
This is frustrating. How much longer do we have to wait for this issue to be resolved?
I’ve been in my office since 8AM and is still not fixed.
Couldn’t this rebuild process take place on the weekend to not affect as many customers who rely on you for work?
I’m wondering what IMAP has to do with either SquirrelMail or WebMail.
It’s understandable that all mailboxes are not available.
However, we consider access to current and urgent email must be avaiable.
Please explain why a simpl backup emergency email system cannot be created.
Thanks for your attention. Jeffrey from Honokowai, Maui, HI
Thank you Kelsey and Team!
IMAP services seem to be fully restored for our users (US & UK). Webmail is still not available to any of us. Some are back to normal service levels while others (e.g. with large inboxes) are slogging through what appears to be a very long process or resynchronize their Outlook clients.
Note that some are asking why Sonic chose to perform maintenance during normal business hours … is there an approved Sonic message we could share? (Reply to this message is obviously of *much lower priority* than getting WEBMAIL back up — it would be nice to have a reply later on.)
It would be helpful to have the time posted to the “Updates” from Sonic.
My outlook still will not work. My emails are back in the account even some new ones however when it asks me log back in it just keeps popping back up asking over and over again. Can we get another update please.
Email still down at 1:03 pm 9/21/2018 in Oakland – how did this happen? Why wasn’t this foreseen, with no workaround plan?
I feel strangely responsible for this 🙂
Good luck Kelsey.
I have done my part by _not_ ticking the “notify me by email” boxes on this post.
James,
Yes, that would be an option. Auto-response was not effected by this. Just remember to disable the auto-response when service is restored.
Noted. Thanks for the input.
WTF Nobody does an update during the week. All companies do it on the weekends. Those of us who need internet for work are screwed by this MAJOR screwup. Next time do it on Saturday night. What was management thinking?
Charlie,
We don’t expect any missing emails once service is restored.
I count on you and just keep checking your progress.
You are so dependable, that this is just a glitch and I hope it is repaired soon.
thanks.
Be patient everyone.
Sonic is one of the few ISP/email/comm. services that actively updates its customers when something is not working. In fact, whenever possible Sonic’s team tries to notify us before, during and after events. ATT, Verizon, Comcast, etc. representatives rarely know *IF* something is wrong (let alone “what”), assuming you can even get in contact with them.
No technology has 100% uptime, sadly.
OUTLOOK notes: I can see emails in my inbox on my phone, but not in Outlook (2016). All my subfolders are visible, just not the inbox. Trying to sync that folder results in an error. Now trying to force Outlook to rebuild the *.ost file, but that takes a long time to even see if it works. (Sorry if this is putting added load on the servers; I started it before I saw this page.)
Kristi: I, too, was getting the repeated login requests. I was able to get that to stop by retyping the password, rather than relying on the remembered password.
For the record, I’m in the camp of this is an exceedingly rare occurrence, and the Sonic team seems to be handling it well (certainly better than I’d expect from certain other companies), so I’m still happy overall.
When I try to log in I get:
ERROR
Unknown user or password incorrect.
Is that part of the “limiting users”?